Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs+test: ROADMAP was the file that got the sampling claim wrong

xerj-orga420d58b95bc
diff --git a/ROADMAP.md b/ROADMAP.md
index 52a518461..5cc157a65 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -9,9 +9,10 @@ Last reviewed: 2026-07-12 (against `v1.0.0-rc.3` + RC4 wave-4 `main` HEAD, live-
These are implemented and exercised by real API requests / the test suite / benchmarks:
- Elasticsearch REST wire compatibility (1,365 / 1,368 ES-YAML conformance cases).
-- Full-text search (BM25) and **50 query types**. The list is no longer maintained by hand here: it is generated from `xerj_query::parser::SUPPORTED_QUERY_TYPES`, printed in full in [engine/README.md](./engine/README.md#query-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned to `parse_query`'s dispatch table by `parser::tests::dispatch_table_matches_capability_manifest`. Two further keys — `has_child` and `has_parent` — are recognised and **rejected with a 400**, and are listed as such in the same places (issue #211).
- **Honest caveat, unchanged:** those 50 are the *dispatch* surface — every one of them parses, plans and executes, which is not a claim that all 50 are semantically faithful to ES. The known divergences are enumerated under *Partial / in progress* below, and the ES-YAML conformance suite is the measured answer.
-- **Aggregations: 62 types**, likewise generated from `xerj_engine::aggs::SUPPORTED_AGG_TYPES` and printed in [engine/README.md](./engine/README.md#aggregation-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt). This includes the full **pipeline family** (`avg_bucket`, `sum_bucket`, `min_bucket`, `max_bucket`, `stats_bucket`, `extended_stats_bucket`, `percentiles_bucket`, `derivative`, `cumulative_sum`, `serial_diff`, `moving_avg`, `moving_fn`, `bucket_script`, `bucket_selector`, `bucket_sort`), which was absent from every published list until issue #211 — a reviewer reading those lists concluded XERJ had no pipeline aggregations and nearly filed a roadmap item to build them a second time. All aggregations are exact (no HLL, no sampling). `weighted_avg` is **not** among the 62 — see *Aggregations that are stubbed or silently degrade*.
+- Full-text search (BM25) and **<!-- generated:query-type-count -->50<!-- /generated:query-type-count --> query types**. Neither the list nor the number is maintained by hand here: the list is generated from `xerj_query::parser::SUPPORTED_QUERY_TYPES`, printed in full in [engine/README.md](./engine/README.md#query-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned to `parse_query`'s dispatch table by `parser::tests::dispatch_table_matches_capability_manifest`; the number above sits in a machine-checked region pinned to that constant's length by `docs_capability_lists::published_capability_counts_match_the_constants`. A further <!-- generated:rejected-query-type-count -->2<!-- /generated:rejected-query-type-count --> keys — `has_child` and `has_parent` — are recognised and **rejected with a 400**, and are listed as such in the same places (issue #211).
+ **Honest caveat, unchanged:** that count is the *dispatch* surface — every name on it parses, plans and executes, which is not a claim that every one is semantically faithful to ES. The known divergences are enumerated under *Partial / in progress* below, and the ES-YAML conformance suite is the measured answer.
+- **Aggregations: <!-- generated:agg-type-count -->62<!-- /generated:agg-type-count --> types**, likewise generated from `xerj_engine::aggs::SUPPORTED_AGG_TYPES`, printed in [engine/README.md](./engine/README.md#aggregation-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned by the same count test. This includes the full **pipeline family** — `bucket_script`, `derivative`, `moving_fn` and the rest, enumerated in those machine-checked lists rather than copied here — which was absent from every published list until issue #211 — a reviewer reading those lists concluded XERJ had no pipeline aggregations and nearly filed a roadmap item to build them a second time. `weighted_avg` is **not** in `SUPPORTED_AGG_TYPES` — see *Aggregations that are stubbed or silently degrade*.
+ **Exactness, precisely.** No probabilistic sketch sits in the metric path: `cardinality` is a true distinct count rather than an HLL estimate, and `terms` `doc_count` is precise. Two deliberate exceptions, stated the same way in `engine/README.md` and `llms-full.txt`: (1) the **sampling family** is a sample by definition — `run_sampler` sorts the matched documents by `_score` and keeps the first `shard_size` (default **200**), so every sub-aggregation under `sampler`, `random_sampler` or `diversified_sampler` is computed over that slice rather than the whole match set, `diversified_sampler` additionally caps documents per `field` value, and `random_sampler` shares the `sampler` implementation and **ignores ES's `probability`** (an accepted-and-ignored input, #204); (2) `percentiles` with the `hdr` option returns HdrHistogram-quantized values, deliberately, so ES's own outputs reproduce — the default `tdigest` path sorts every value and interpolates instead.
- **Dense-vector kNN** (`knn` query and ES 8.x top-level `knn`): unfiltered kNN on a full-precision cosine field (≥1,024 docs) is served by a **persisted HNSW graph with exact rescoring** — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937); filtered/nested kNN, non-cosine similarity, SQ8 fields, and small indexes run the exact brute-force scan (cosine mapped to `(1+cos)/2`). See "Landed since rc-2" below.
- **Hybrid search** — BM25 + kNN combined in a single request via the `hybrid` **query type**: `{"query":{"hybrid":{"queries":[{"query":…,"weight":…}, …],"fusion":"rrf|linear|learned"}}}`. RRF-fused union verified live. (See *Partial* for the ES-native top-level `query`+`knn` path, which does **not** fuse.)
- **Columnar storage** — the ZBS2 columnar block (per-column codec) with exactly **9 domain-aware encodings** (`BitsetEnum`, `DeltaTimestamp`, `PackedIp`, `UrlTemplate`, `Varint`, `Dictionary`, `RawString`, `Bitpacked`, `FixedPrecision`), ZSTD/LZ4 codecs, and SQ8 vector quantization — all real and wired into the segment write path.
diff --git a/engine/README.md b/engine/README.md
index 57a22920a..d8fff10f1 100644
--- a/engine/README.md
+++ b/engine/README.md
@@ -205,14 +205,20 @@ Complete and machine-checked the same way, from
metric path — `cardinality` is a true distinct count, not an HLL estimate, and
`terms` `doc_count` is precise.
-The **sampling family is the deliberate exception**, and it is a sample by
-definition: `run_sampler` sorts the matched documents by `_score` and keeps the
-first `shard_size` (default 200), so every sub-aggregation under `sampler` or
-`random_sampler` is computed over that slice rather than the whole match set.
-`diversified_sampler` truncates the same way and additionally caps how many
-documents may share a `field` value (`max_docs_per_value`, default 1).
-`random_sampler` shares the `sampler` implementation and does not read ES's
-`probability`.
+There are two deliberate exceptions. The first is the **sampling family**,
+which is a sample by definition: `run_sampler` sorts the matched documents by
+`_score` and keeps the first `shard_size` (default 200), so every
+sub-aggregation under `sampler` or `random_sampler` is computed over that slice
+rather than the whole match set. `diversified_sampler` truncates the same way
+and additionally caps how many documents may share a `field` value
+(`max_docs_per_value`, default 1). `random_sampler` shares the `sampler`
+implementation and does not read ES's `probability`.
+
+The second is `percentiles` with the **`hdr`** option, which returns
+HdrHistogram-quantized values on purpose, so that ES's own outputs reproduce.
+The default `tdigest` path is not a t-digest at all: it sorts every matched
+value and interpolates linearly between the neighbours, reading the whole value
+set (O(N) memory) instead of a sketch.
<!-- generated:agg-types -->
Metric — `avg`, `sum`, `min`, `max`, `stats`, `extended_stats`, `value_count`,
diff --git a/engine/crates/xerj-engine/tests/docs_capability_lists.rs b/engine/crates/xerj-engine/tests/docs_capability_lists.rs
index 49e67a7a2..1b57dedb4 100644
--- a/engine/crates/xerj-engine/tests/docs_capability_lists.rs
+++ b/engine/crates/xerj-engine/tests/docs_capability_lists.rs
@@ -622,10 +622,18 @@ fn every_query_clause_in_a_published_sample_is_a_real_query_type() {
/// Unlike the list regions, a file may carry the same count section more than
/// once (the pricing page prints the figure in a plan bullet and again in the
/// comparison table); every occurrence is checked.
+///
+/// `ROADMAP.md` is markdown rather than HTML, and was originally left out —
+/// which re-created the defect inside its own fix: it published `50 query
+/// types` and `62 aggregation types` as hand-typed literals, so adding query
+/// type 51 would have left it silently saying 50 with every test green.
+/// HTML comments are invisible in rendered markdown, so the same marker works
+/// there unchanged.
const COUNT_DOCS: &[&str] = &[
"landing/docs/migration-from-es.html",
"landing/pricing/index.html",
"landing/demo/index.html",
+ "ROADMAP.md",
];
/// The count sections and where each number comes from.
@@ -684,7 +692,7 @@ fn published_capability_counts_match_the_constants() {
}
assert!(
- seen >= 5,
+ seen >= 9,
"only {seen} marked counts found across {COUNT_DOCS:?} — a page dropped its \
markers and its number is now unchecked"
);
diff --git a/engine/crates/xerj-query/src/parser.rs b/engine/crates/xerj-query/src/parser.rs
index e1ca52974..b9541be47 100644
--- a/engine/crates/xerj-query/src/parser.rs
+++ b/engine/crates/xerj-query/src/parser.rs
@@ -5641,8 +5641,11 @@ mod tests {
}
/// Behavioural half: the manifest's two halves must actually behave the
- /// way they are labelled. A supported type must not answer
- /// `unknown query type`, and a rejected type must refuse.
+ /// way they are labelled — in **both** directions. A supported type must
+ /// neither answer `unknown query type` nor refuse with a `not supported`
+ /// message, and a rejected type must refuse with exactly that message.
+ /// Checking only the first of those let a recognised-and-refused type be
+ /// relabelled supported with the whole manifest suite green.
#[test]
fn manifest_labels_match_observed_behaviour() {
// `{ "<type>": {} }` — the minimal body that reaches the dispatch arm.
@@ -5656,11 +5659,29 @@ mod tests {
for name in SUPPORTED_QUERY_TYPES {
let err = parse_query(&bare(name)).err();
- if let Some(QueryError::Parse(ParseError::UnknownQueryType(t))) = err {
+ if let Some(QueryError::Parse(ParseError::UnknownQueryType(t))) = &err {
panic!(
"`{name}` is listed as supported but parse_query rejects it as unknown: {t}"
);
}
+ // The unknown-query-type check alone does not prove the *supported*
+ // label: a type that is recognised and then refused with a 400 —
+ // exactly what `has_child` does — never produces
+ // `UnknownQueryType`, so moving it into SUPPORTED_QUERY_TYPES used
+ // to pass here and then oblige the docs guard to publish it as a
+ // capability. That is the #211 defect in the direction this test
+ // exists to catch, so the refusal wording is checked too: the
+ // phrase REJECTED_QUERY_TYPES is required to carry is the phrase
+ // SUPPORTED_QUERY_TYPES is forbidden to carry.
+ if let Some(e) = &err {
+ let msg = e.to_string();
+ assert!(
+ !msg.contains("not supported"),
+ "`{name}` is listed as supported but refuses with a not-supported \
+ message: {msg}\n A recognised-and-refused type belongs in \
+ REJECTED_QUERY_TYPES, or the docs will advertise a query that 400s."
+ );
+ }
}
for name in REJECTED_QUERY_TYPES {
let err = parse_query(&bare(name))
diff --git a/landing/llms-full.txt b/landing/llms-full.txt
index 22f630e41..a3032a47a 100644
--- a/landing/llms-full.txt
+++ b/landing/llms-full.txt
@@ -483,12 +483,14 @@ track_total_hits, and (via dedicated endpoints) scroll and point-in-time.
Aggregations (complete; no probabilistic sketch in the metric path — cardinality
is a true distinct count, not an HLL estimate, and terms doc_count is precise).
-The Sampling family below is the deliberate exception: sampler and
+Two deliberate exceptions. (1) The Sampling family below: sampler and
random_sampler keep only the top `shard_size` matched docs by _score (default
200) and diversified_sampler additionally caps docs per field value, so
sub-aggregations under any of the three run over that sample, not the whole
match set — and random_sampler shares the sampler implementation and ignores
-ES's `probability`:
+ES's `probability`. (2) percentiles with the `hdr` option returns
+HdrHistogram-quantized values, deliberately, to reproduce ES's own outputs; the
+default `tdigest` path sorts every value and interpolates instead:
<!-- generated:agg-types -->
Metric `avg`, `sum`, `min`, `max`, `stats`, `extended_stats`, `value_count`,