Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs: align README to reality + add public ROADMAP (agent memory, local embeddings, anomaly detection)

xerj-orgd53f2eb146ee
diff --git a/.gitignore b/.gitignore
index fd3d744b7..42372c8a7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -95,3 +95,4 @@ landing/resources/xerj-sales-kit.zip
# Re-include the public architecture doc (Markdown is ignored by default above).
!docs/ARCHITECTURE.md
+!ROADMAP.md
diff --git a/README.md b/README.md
index 19c87cf7e..dc8d0acca 100644
--- a/README.md
+++ b/README.md
@@ -135,6 +135,8 @@ XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compati
`match_all`, `match_none`, `match`, `match_phrase`, `match_phrase_prefix`, `multi_match`, `term`, `terms`, `range`, `prefix`, `wildcard`, `exists`, `ids`, `bool`, `fuzzy`, `regexp`, `query_string`, `simple_query_string`, `constant_score`, `boosting`, `dis_max`, `geo_distance`, `knn`, `semantic`, `hybrid`.
+> **Vector & semantic search notes.** `knn` (dense-vector HNSW) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. The `semantic` query resolves query text to a vector at search time via an **external, OpenAI-compatible `/v1/embeddings` endpoint** that you configure (`embedding.default_endpoint` / `embedding.default_model`) — XERJ ships the proxy, not a built-in embedding model. Built-in/local embeddings, auto-embed-on-ingest, and an agent-memory API are on the [roadmap](#roadmap).
+
**Supported aggregations**
`terms`, `stats`, `avg`, `sum`, `min`, `max`, `value_count`, `cardinality`, `range`, `histogram`, `date_histogram`, `percentiles`, `filter`, `missing`, `composite`.
@@ -165,7 +167,7 @@ XERJ is a Cargo workspace under [`engine/`](./engine). The crates:
| `xerj-fts` | Full-text search: BM25 scoring, analyzer registry, postings lists. |
| `xerj-vector` | Dense-vector HNSW index for kNN / semantic search. |
| `xerj-logs` | Columnar log ingestion and retention. |
-| `xerj-ai` | Text chunking, embedding proxy, and memory store. |
+| `xerj-ai` | Text chunking and an embedding proxy (external OpenAI-compatible API) that powers the `semantic` query. Also houses an experimental memory store not yet exposed over the REST API — see the [roadmap](#roadmap). |
| `xerj-compress` | Block compression codecs (LZ4, Zstd). |
| `xerj-common` | Shared types: `Config`, `Schema`, `FieldType`, `XerjError`. |
| `xerj-cluster` | Embedded Raft consensus for cluster metadata (no external Raft dependency). |
@@ -221,6 +223,18 @@ If a test expects a response and XERJ returns something different, that's a bug
---
+## Roadmap
+
+XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search today. Several AI-adjacent capabilities are **planned but not yet implemented** — they are called out here (and in [`ROADMAP.md`](./ROADMAP.md)) so the README stays honest about what is and isn't built:
+
+- **Built-in / local embeddings & auto-embed on ingest.** Today, `semantic` search proxies to an external OpenAI-compatible embeddings endpoint you configure; there is no bundled embedding model, and the `semantic_text` field type is a mapping placeholder that does not yet auto-embed on ingest.
+- **Agent-memory API.** An internal memory store exists in `xerj-ai` but is not yet wired to a REST endpoint (store/recall/forget over the wire).
+- **Anomaly detection / ML jobs.** The Elasticsearch `_ml` / `_cat/ml/*` surface is present as compatibility stubs only; there is no detection or forecasting engine behind it yet.
+
+See [ROADMAP.md](./ROADMAP.md) for the full list and status.
+
+---
+
## Contributing
Contributions are welcome. Please read [CONTRIBUTING.md](./CONTRIBUTING.md) for how to build, test, and structure changes. A few conventions:
diff --git a/ROADMAP.md b/ROADMAP.md
new file mode 100644
index 000000000..28c37db42
--- /dev/null
+++ b/ROADMAP.md
@@ -0,0 +1,58 @@
+# XERJ Roadmap
+
+This roadmap tracks capabilities that are **planned but not yet fully implemented**, so the project's public claims stay honest about what ships today versus what is coming. Status is verified against the actual code and by real API requests, not aspirational.
+
+Last reviewed: 2026-07-06 (against `v1.0.0-rc.1`).
+
+## Shipping today (for context)
+
+These are implemented and exercised by the test suite / benchmarks:
+
+- Elasticsearch REST wire compatibility (1,326 / 1,329 ES-YAML conformance cases).
+- Full-text search (BM25) and all 38 supported query types.
+- The standard aggregation suite.
+- **Dense-vector kNN** over HNSW (`knn` query and top-level `knn`), at recall parity with Elasticsearch.
+- **Hybrid search** — BM25 + kNN combined in a single request (`{"hybrid": {"query": …, "knn": …}}`).
+- Columnar log analytics, bulk / scroll / delete-by-query, aliases, index templates.
+
+## Planned / in progress
+
+### 1. Built-in & local embeddings, auto-embed on ingest
+
+**Status: partial — external proxy only.**
+
+Today the `semantic` query resolves query text to a vector at search time by proxying to an **external, OpenAI-compatible `/v1/embeddings` endpoint** that the operator configures (`embedding.default_endpoint`, `embedding.default_model`). Without that configuration, `semantic` returns a clear error. There is **no bundled/local embedding model**, and the `semantic_text` field type is currently a mapping placeholder — it does not yet auto-embed documents on ingest (indexing into a `semantic_text` field is not wired up).
+
+Planned:
+- Auto-embed on ingest for `semantic_text` fields (text in → vector stored, transparently).
+- An optional bundled/local embedding model so semantic search works with zero external dependencies.
+- An ingest-time embedding pipeline with the existing text chunker (overlapping chunks → per-chunk vectors).
+
+### 2. Agent-memory API
+
+**Status: internal module only, not exposed over REST.**
+
+`xerj-ai` contains a memory-store module (entry model + recall), but it is **not wired to any HTTP endpoint** — there is no `store` / `recall` / `forget` REST API today (all memory paths return `404`).
+
+Planned:
+- A REST API to store, semantically recall, and expire agent-memory entries per namespace.
+- Recall backed by the existing vector index (kNN over stored memories) plus recency/metadata filters.
+
+### 3. Anomaly detection & ML jobs
+
+**Status: compatibility stubs only.**
+
+The Elasticsearch machine-learning surface is present for wire compatibility (`_cat/ml/anomaly_detectors`, `_cat/ml/datafeeds`, `_cat/ml/trained_models` return empty), but there is **no detection or forecasting engine** behind it — creating an anomaly job (`PUT /_ml/anomaly_detectors/{id}`) is not implemented.
+
+Planned:
+- Time-series anomaly detection (baseline + deviation scoring) over indexed metrics/logs.
+- Optional forecasting for capacity/write-load signals.
+
+### 4. Other tracked items
+
+- **Distributed clustering maturity** — embedded Raft handles cluster metadata today; multi-node sharding/replication hardening is ongoing.
+- **Broader aggregation coverage** — geo/IP/nested/join/span families are partially covered; see the conformance suite for the current surface.
+
+---
+
+Found something claimed but not working? That is a bug in our docs or our code — please [open an issue](https://github.com/xerj-org/xerj/issues). We would rather ship an honest roadmap than an overstated feature list.