AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs+test: ROADMAP was the file that got the sampling claim wrong
xerj-orga420d58b95bc
diff --git a/ROADMAP.md b/ROADMAP.mdindex 52a518461..5cc157a65 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -9,9 +9,10 @@ Last reviewed: 2026-07-12 (against `v1.0.0-rc.3` + RC4 wave-4 `main` HEAD, live-These are implemented and exercised by real API requests / the test suite / benchmarks:- Elasticsearch REST wire compatibility (1,365 / 1,368 ES-YAML conformance cases).-- Full-text search (BM25) and **50 query types**. The list is no longer maintained by hand here: it is generated from `xerj_query::parser::SUPPORTED_QUERY_TYPES`, printed in full in [engine/README.md](./engine/README.md#query-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned to `parse_query`'s dispatch table by `parser::tests::dispatch_table_matches_capability_manifest`. Two further keys — `has_child` and `has_parent` — are recognised and **rejected with a 400**, and are listed as such in the same places (issue #211).- **Honest caveat, unchanged:** those 50 are the *dispatch* surface — every one of them parses, plans and executes, which is not a claim that all 50 are semantically faithful to ES. The known divergences are enumerated under *Partial / in progress* below, and the ES-YAML conformance suite is the measured answer.-- **Aggregations: 62 types**, likewise generated from `xerj_engine::aggs::SUPPORTED_AGG_TYPES` and printed in [engine/README.md](./engine/README.md#aggregation-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt). This includes the full **pipeline family** (`avg_bucket`, `sum_bucket`, `min_bucket`, `max_bucket`, `stats_bucket`, `extended_stats_bucket`, `percentiles_bucket`, `derivative`, `cumulative_sum`, `serial_diff`, `moving_avg`, `moving_fn`, `bucket_script`, `bucket_selector`, `bucket_sort`), which was absent from every published list until issue #211 — a reviewer reading those lists concluded XERJ had no pipeline aggregations and nearly filed a roadmap item to build them a second time. All aggregations are exact (no HLL, no sampling). `weighted_avg` is **not** among the 62 — see *Aggregations that are stubbed or silently degrade*.+- Full-text search (BM25) and **<!-- generated:query-type-count -->50<!-- /generated:query-type-count --> query types**. Neither the list nor the number is maintained by hand here: the list is generated from `xerj_query::parser::SUPPORTED_QUERY_TYPES`, printed in full in [engine/README.md](./engine/README.md#query-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned to `parse_query`'s dispatch table by `parser::tests::dispatch_table_matches_capability_manifest`; the number above sits in a machine-checked region pinned to that constant's length by `docs_capability_lists::published_capability_counts_match_the_constants`. A further <!-- generated:rejected-query-type-count -->2<!-- /generated:rejected-query-type-count --> keys — `has_child` and `has_parent` — are recognised and **rejected with a 400**, and are listed as such in the same places (issue #211).+ **Honest caveat, unchanged:** that count is the *dispatch* surface — every name on it parses, plans and executes, which is not a claim that every one is semantically faithful to ES. The known divergences are enumerated under *Partial / in progress* below, and the ES-YAML conformance suite is the measured answer.+- **Aggregations: <!-- generated:agg-type-count -->62<!-- /generated:agg-type-count --> types**, likewise generated from `xerj_engine::aggs::SUPPORTED_AGG_TYPES`, printed in [engine/README.md](./engine/README.md#aggregation-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned by the same count test. This includes the full **pipeline family** — `bucket_script`, `derivative`, `moving_fn` and the rest, enumerated in those machine-checked lists rather than copied here — which was absent from every published list until issue #211 — a reviewer reading those lists concluded XERJ had no pipeline aggregations and nearly filed a roadmap item to build them a second time. `weighted_avg` is **not** in `SUPPORTED_AGG_TYPES` — see *Aggregations that are stubbed or silently degrade*.+ **Exactness, precisely.** No probabilistic sketch sits in the metric path: `cardinality` is a true distinct count rather than an HLL estimate, and `terms` `doc_count` is precise. Two deliberate exceptions, stated the same way in `engine/README.md` and `llms-full.txt`: (1) the **sampling family** is a sample by definition — `run_sampler` sorts the matched documents by `_score` and keeps the first `shard_size` (default **200**), so every sub-aggregation under `sampler`, `random_sampler` or `diversified_sampler` is computed over that slice rather than the whole match set, `diversified_sampler` additionally caps documents per `field` value, and `random_sampler` shares the `sampler` implementation and **ignores ES's `probability`** (an accepted-and-ignored input, #204); (2) `percentiles` with the `hdr` option returns HdrHistogram-quantized values, deliberately, so ES's own outputs reproduce — the default `tdigest` path sorts every value and interpolates instead.- **Dense-vector kNN** (`knn` query and ES 8.x top-level `knn`): unfiltered kNN on a full-precision cosine field (≥1,024 docs) is served by a **persisted HNSW graph with exact rescoring** — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937); filtered/nested kNN, non-cosine similarity, SQ8 fields, and small indexes run the exact brute-force scan (cosine mapped to `(1+cos)/2`). See "Landed since rc-2" below.- **Hybrid search** — BM25 + kNN combined in a single request via the `hybrid` **query type**: `{"query":{"hybrid":{"queries":[{"query":…,"weight":…}, …],"fusion":"rrf|linear|learned"}}}`. RRF-fused union verified live. (See *Partial* for the ES-native top-level `query`+`knn` path, which does **not** fuse.)- **Columnar storage** — the ZBS2 columnar block (per-column codec) with exactly **9 domain-aware encodings** (`BitsetEnum`, `DeltaTimestamp`, `PackedIp`, `UrlTemplate`, `Varint`, `Dictionary`, `RawString`, `Bitpacked`, `FixedPrecision`), ZSTD/LZ4 codecs, and SQ8 vector quantization — all real and wired into the segment write path.diff --git a/engine/README.md b/engine/README.mdindex 57a22920a..d8fff10f1 100644--- a/engine/README.md+++ b/engine/README.md@@ -205,14 +205,20 @@ Complete and machine-checked the same way, frommetric path — `cardinality` is a true distinct count, not an HLL estimate, and`terms` `doc_count` is precise.-The **sampling family is the deliberate exception**, and it is a sample by-definition: `run_sampler` sorts the matched documents by `_score` and keeps the-first `shard_size` (default 200), so every sub-aggregation under `sampler` or-`random_sampler` is computed over that slice rather than the whole match set.-`diversified_sampler` truncates the same way and additionally caps how many-documents may share a `field` value (`max_docs_per_value`, default 1).-`random_sampler` shares the `sampler` implementation and does not read ES's-`probability`.+There are two deliberate exceptions. The first is the **sampling family**,+which is a sample by definition: `run_sampler` sorts the matched documents by+`_score` and keeps the first `shard_size` (default 200), so every+sub-aggregation under `sampler` or `random_sampler` is computed over that slice+rather than the whole match set. `diversified_sampler` truncates the same way+and additionally caps how many documents may share a `field` value+(`max_docs_per_value`, default 1). `random_sampler` shares the `sampler`+implementation and does not read ES's `probability`.++The second is `percentiles` with the **`hdr`** option, which returns+HdrHistogram-quantized values on purpose, so that ES's own outputs reproduce.+The default `tdigest` path is not a t-digest at all: it sorts every matched+value and interpolates linearly between the neighbours, reading the whole value+set (O(N) memory) instead of a sketch.<!-- generated:agg-types -->Metric — `avg`, `sum`, `min`, `max`, `stats`, `extended_stats`, `value_count`,diff --git a/engine/crates/xerj-engine/tests/docs_capability_lists.rs b/engine/crates/xerj-engine/tests/docs_capability_lists.rsindex 49e67a7a2..1b57dedb4 100644--- a/engine/crates/xerj-engine/tests/docs_capability_lists.rs+++ b/engine/crates/xerj-engine/tests/docs_capability_lists.rs@@ -622,10 +622,18 @@ fn every_query_clause_in_a_published_sample_is_a_real_query_type() {/// Unlike the list regions, a file may carry the same count section more than/// once (the pricing page prints the figure in a plan bullet and again in the/// comparison table); every occurrence is checked.+///+/// `ROADMAP.md` is markdown rather than HTML, and was originally left out —+/// which re-created the defect inside its own fix: it published `50 query+/// types` and `62 aggregation types` as hand-typed literals, so adding query+/// type 51 would have left it silently saying 50 with every test green.+/// HTML comments are invisible in rendered markdown, so the same marker works+/// there unchanged.const COUNT_DOCS: &[&str] = &["landing/docs/migration-from-es.html","landing/pricing/index.html","landing/demo/index.html",+ "ROADMAP.md",];/// The count sections and where each number comes from.@@ -684,7 +692,7 @@ fn published_capability_counts_match_the_constants() {}assert!(- seen >= 5,+ seen >= 9,"only {seen} marked counts found across {COUNT_DOCS:?} — a page dropped its \markers and its number is now unchecked");diff --git a/engine/crates/xerj-query/src/parser.rs b/engine/crates/xerj-query/src/parser.rsindex e1ca52974..b9541be47 100644--- a/engine/crates/xerj-query/src/parser.rs+++ b/engine/crates/xerj-query/src/parser.rs@@ -5641,8 +5641,11 @@ mod tests {}/// Behavioural half: the manifest's two halves must actually behave the- /// way they are labelled. A supported type must not answer- /// `unknown query type`, and a rejected type must refuse.+ /// way they are labelled — in **both** directions. A supported type must+ /// neither answer `unknown query type` nor refuse with a `not supported`+ /// message, and a rejected type must refuse with exactly that message.+ /// Checking only the first of those let a recognised-and-refused type be+ /// relabelled supported with the whole manifest suite green.#[test]fn manifest_labels_match_observed_behaviour() {// `{ "<type>": {} }` — the minimal body that reaches the dispatch arm.@@ -5656,11 +5659,29 @@ mod tests {for name in SUPPORTED_QUERY_TYPES {let err = parse_query(&bare(name)).err();- if let Some(QueryError::Parse(ParseError::UnknownQueryType(t))) = err {+ if let Some(QueryError::Parse(ParseError::UnknownQueryType(t))) = &err {panic!("`{name}` is listed as supported but parse_query rejects it as unknown: {t}");}+ // The unknown-query-type check alone does not prove the *supported*+ // label: a type that is recognised and then refused with a 400 —+ // exactly what `has_child` does — never produces+ // `UnknownQueryType`, so moving it into SUPPORTED_QUERY_TYPES used+ // to pass here and then oblige the docs guard to publish it as a+ // capability. That is the #211 defect in the direction this test+ // exists to catch, so the refusal wording is checked too: the+ // phrase REJECTED_QUERY_TYPES is required to carry is the phrase+ // SUPPORTED_QUERY_TYPES is forbidden to carry.+ if let Some(e) = &err {+ let msg = e.to_string();+ assert!(+ !msg.contains("not supported"),+ "`{name}` is listed as supported but refuses with a not-supported \+ message: {msg}\n A recognised-and-refused type belongs in \+ REJECTED_QUERY_TYPES, or the docs will advertise a query that 400s."+ );+ }}for name in REJECTED_QUERY_TYPES {let err = parse_query(&bare(name))diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 22f630e41..a3032a47a 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -483,12 +483,14 @@ track_total_hits, and (via dedicated endpoints) scroll and point-in-time.Aggregations (complete; no probabilistic sketch in the metric path — cardinalityis a true distinct count, not an HLL estimate, and terms doc_count is precise).-The Sampling family below is the deliberate exception: sampler and+Two deliberate exceptions. (1) The Sampling family below: sampler andrandom_sampler keep only the top `shard_size` matched docs by _score (default200) and diversified_sampler additionally caps docs per field value, sosub-aggregations under any of the three run over that sample, not the wholematch set — and random_sampler shares the sampler implementation and ignores-ES's `probability`:+ES's `probability`. (2) percentiles with the `hdr` option returns+HdrHistogram-quantized values, deliberately, to reproduce ES's own outputs; the+default `tdigest` path sorts every value and interpolates instead:<!-- generated:agg-types -->Metric `avg`, `sum`, `min`, `max`, `stats`, `extended_stats`, `value_count`,