Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(roadmap): rewrite forward-looking against rc.15 + main, replacing the rc.3-era review

xerj-orgee29361bccce
diff --git a/ROADMAP.md b/ROADMAP.md
index 5cc157a65..b6cc75f35 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -2,148 +2,82 @@
This roadmap tracks capabilities that are **planned but not yet fully implemented**, so the project's public claims stay honest about what ships today versus what is coming. Status is verified against the actual code and by real API requests to the release binary, not aspirational.
-Last reviewed: 2026-07-12 (against `v1.0.0-rc.3` + RC4 wave-4 `main` HEAD, live-verified against `engine/target/release/xerj`).
+Last reviewed: 2026-08-11 (against `v1.0.0-rc.15` and `main`). Statuses trace to issues, merged PRs, the CHANGELOG, and the conformance suite; items carried forward from the previous review (2026-07-12) without fresh live verification are marked as such.
+
+## Follow the roadmap
+
+- **This file** is authoritative. If any other surface disagrees with it, this file wins — and that disagreement is a bug worth an issue.
+- **[Milestones](https://github.com/xerj-org/xerj/milestones)** — the release-by-release view. Every open issue is triaged onto a milestone; the next RC's milestone is the short-term roadmap.
+- **[Project board](https://github.com/users/xerj-org/projects/1)** — live status of every open item.
+- **[Pinned issue #298](https://github.com/xerj-org/xerj/issues/298)** — the standing pointer, including how releases are cut and how to influence priorities.
## Shipping today (for context)
These are implemented and exercised by real API requests / the test suite / benchmarks:
-- Elasticsearch REST wire compatibility (1,365 / 1,368 ES-YAML conformance cases).
+- Elasticsearch REST wire compatibility (1,365 / 1,368 ES-YAML conformance cases; the gate on every change is **0 failed**, and the case count grows as cases are added — read the current number off CI, not off this file).
- Full-text search (BM25) and **<!-- generated:query-type-count -->50<!-- /generated:query-type-count --> query types**. Neither the list nor the number is maintained by hand here: the list is generated from `xerj_query::parser::SUPPORTED_QUERY_TYPES`, printed in full in [engine/README.md](./engine/README.md#query-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned to `parse_query`'s dispatch table by `parser::tests::dispatch_table_matches_capability_manifest`; the number above sits in a machine-checked region pinned to that constant's length by `docs_capability_lists::published_capability_counts_match_the_constants`. A further <!-- generated:rejected-query-type-count -->2<!-- /generated:rejected-query-type-count --> keys — `has_child` and `has_parent` — are recognised and **rejected with a 400**, and are listed as such in the same places (issue #211).
- **Honest caveat, unchanged:** that count is the *dispatch* surface — every name on it parses, plans and executes, which is not a claim that every one is semantically faithful to ES. The known divergences are enumerated under *Partial / in progress* below, and the ES-YAML conformance suite is the measured answer.
-- **Aggregations: <!-- generated:agg-type-count -->62<!-- /generated:agg-type-count --> types**, likewise generated from `xerj_engine::aggs::SUPPORTED_AGG_TYPES`, printed in [engine/README.md](./engine/README.md#aggregation-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned by the same count test. This includes the full **pipeline family** — `bucket_script`, `derivative`, `moving_fn` and the rest, enumerated in those machine-checked lists rather than copied here — which was absent from every published list until issue #211 — a reviewer reading those lists concluded XERJ had no pipeline aggregations and nearly filed a roadmap item to build them a second time. `weighted_avg` is **not** in `SUPPORTED_AGG_TYPES` — see *Aggregations that are stubbed or silently degrade*.
+ **Honest caveat, unchanged:** that count is the *dispatch* surface — every name on it parses, plans and executes, which is not a claim that every one is semantically faithful to ES. The known divergences are enumerated under *Known partials* below, and the ES-YAML conformance suite is the measured answer.
+- **Aggregations: <!-- generated:agg-type-count -->62<!-- /generated:agg-type-count --> types**, likewise generated from `xerj_engine::aggs::SUPPORTED_AGG_TYPES`, printed in [engine/README.md](./engine/README.md#aggregation-types-supported) and [llms-full.txt](https://xerj.org/llms-full.txt), and pinned by the same count test. This includes the full **pipeline family**. `weighted_avg` is **not** in `SUPPORTED_AGG_TYPES` — see *Known partials*.
**Exactness, precisely.** No probabilistic sketch sits in the metric path: `cardinality` is a true distinct count rather than an HLL estimate, and `terms` `doc_count` is precise. Two deliberate exceptions, stated the same way in `engine/README.md` and `llms-full.txt`: (1) the **sampling family** is a sample by definition — `run_sampler` sorts the matched documents by `_score` and keeps the first `shard_size` (default **200**), so every sub-aggregation under `sampler`, `random_sampler` or `diversified_sampler` is computed over that slice rather than the whole match set, `diversified_sampler` additionally caps documents per `field` value, and `random_sampler` shares the `sampler` implementation and **ignores ES's `probability`** (an accepted-and-ignored input, #204); (2) `percentiles` with the `hdr` option returns HdrHistogram-quantized values, deliberately, so ES's own outputs reproduce — the default `tdigest` path sorts every value and interpolates instead.
-- **Dense-vector kNN** (`knn` query and ES 8.x top-level `knn`): unfiltered kNN on a full-precision cosine field (≥1,024 docs) is served by a **persisted HNSW graph with exact rescoring** — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937); filtered/nested kNN, non-cosine similarity, SQ8 fields, and small indexes run the exact brute-force scan (cosine mapped to `(1+cos)/2`). See "Landed since rc-2" below.
-- **Hybrid search** — BM25 + kNN combined in a single request via the `hybrid` **query type**: `{"query":{"hybrid":{"queries":[{"query":…,"weight":…}, …],"fusion":"rrf|linear|learned"}}}`. RRF-fused union verified live. (See *Partial* for the ES-native top-level `query`+`knn` path, which does **not** fuse.)
-- **Columnar storage** — the ZBS2 columnar block (per-column codec) with exactly **9 domain-aware encodings** (`BitsetEnum`, `DeltaTimestamp`, `PackedIp`, `UrlTemplate`, `Varint`, `Dictionary`, `RawString`, `Bitpacked`, `FixedPrecision`), ZSTD/LZ4 codecs, and SQ8 vector quantization — all real and wired into the segment write path.
-- Bulk / scroll / delete-by-query, aliases, index templates, `_cat/*`, `_cluster/health`, `_count` / `_msearch` / `_mget`, `_update` / `_update_by_query` (Painless-style script writes applied) — all live-verified.
-- **A single native binary** — ~36 MB (37,896,272 bytes) statically-linked with the neural embedder built in (a `--no-default-features` slim build is ~23 MB), no JVM, sub-second cold start (readiness within ~100 ms).
-
-## Landed in 1.0.0-rc.2
-
-These three shipped in rc-2 (each conformance-gated — the ES-compat YAML suite is now green at 1,360 / 1,363 — and verified by real requests). Honest limitations are noted.
-
-### 1. Auto-embed on ingest + a built-in embedder ✅ (rc-2)
-
-`semantic_text` now works end to end with **zero external configuration**. Indexing a document into a `semantic_text` field auto-embeds its text (previously returned `405`), and the `semantic` query embeds the query text with the same embedder and runs kNN — no external service required. Live-verified: a `semantic_text` doc indexed with no embedder configured returned `201`, and a `semantic` query ranked the intended doc first. A configured external `/v1/embeddings` proxy is still used, at higher quality, when `embedding.default_endpoint` is set.
-
-- **Limitation:** the **default** embedding mode is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. This is observable live: a vocabulary-sharing query out-scored a true paraphrase. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For real neural semantics you have two drop-in upgrades with no mapping/query change: the built-in **neural** BERT embedder that ships in the binary (`--embed-mode neural`, downloads all-MiniLM-L6-v2 on first use — see "Neural embeddings" below), or the external `/v1/embeddings` **proxy** (`--embed-mode proxy` + `embedding.default_endpoint`).
-
-### 2. Agent-memory REST API ✅ (rc-2)
-
-A namespaced agent-memory API, backed by regular XERJ indices (reusing document + vector + BM25 + metadata-filter paths), working fully offline:
-`POST /_memory/{ns}` (store), `POST /_memory/{ns}/_recall` (kNN by vector or BM25 by text, with optional metadata filter + `k`), `GET /_memory/{ns}` (list), `DELETE /_memory/{ns}/{id}` and `DELETE /_memory/{ns}` (forget / drop). Namespaces are physically isolated — live-verified: recall in an empty namespace returns `hits:[]`, text recall ranks the correct memory first, vector recall returns correct kNN order, and a `metadata.topic` term filter narrows correctly.
-
-- **Limitation:** recall takes exactly one of `query` (BM25, or semantic when `semantic:true`) or `vector` (kNN) — the store uses `text`, so a recall body must say `query`, not `text`, and unknown fields are now rejected (`f1b70fe`). Metadata filters must be a full ES clause (e.g. `{term:{"metadata.topic":…}}`). Recency-blended recall (`recency_weight`) and semantic dedup (`dedup` / `dedup_threshold`) **are** applied on the REST path (`memory_api.rs` `blend_recency`, `ec6dc4a`). Single-node.
-
-### 3. Anomaly detection (`_ml`) ✅ (rc-2)
-
-A real statistical detector replaces the empty compat stubs:
-`PUT /_ml/anomaly_detectors/{id}` (create: source index, time field, function `count|mean|min|max|sum`, bucket span, threshold), `GET` (fetch/list — returns real jobs), `POST /_ml/anomaly_detectors/{id}/_score` (buckets the source over time, builds a moving mean/stddev baseline, flags buckets deviating beyond the threshold with a normalized anomaly score), `DELETE`. Live-verified: a 500-value spike among 24 baseline buckets of ~10 was correctly flagged (`is_anomaly:true`, `anomaly_score:100`), and `DELETE` removed the job from subsequent `GET`s.
-
-- **Limitation:** on-demand scoring only (`POST _score`) — no continuous datafeed scheduler, no forecasting, no influencers/model-plot, single-node config registry. When the baseline std_dev is 0 the z-score is a placeholder (`1000000`). `_cat/ml/datafeeds` and `_cat/ml/trained_models` remain valid empty stubs. (The continuous datafeed scheduler has since landed — see below.)
-
-## Landed in 1.0.0-rc.3
-
-These shipped in the **rc-3 release** (cut 2026-07-10) during the RC3 gap-closure and AI-use-case pass. Each is conformance-gated (full ES-compat YAML suite green) and ships a runnable recipe + docs.
-
-### 4. Real scalar8 vector quantization (serving path) ✅
-
-A `dense_vector` field can opt into **scalar8** (int8) quantization via `index_options.type: int8_hnsw`. The kNN *serving* path scores against 1-byte-per-dimension codes (≈4× smaller vector working set) while `_source` still returns the original float32 vectors. Live-verified on a 128-dim corpus: **recall@10 ≈ 0.99** vs exact float32, footprint 512 → 128 B/vec. Recipe: `recipes/vector_quantization.py`; guide: `docs/recipes/vector-quantization.md`.
-
-- **Limitation:** `binary` (1-bit) is still rejected at startup rather than faked; scalar4/offload remain future work.
-
-### 5. Continuous anomaly datafeeds (`_ml/datafeeds`) ✅
+- **Dense-vector kNN** (`knn` query and ES 8.x top-level `knn`): unfiltered kNN on a full-precision cosine field (≥1,024 docs) is served by a **persisted HNSW graph with exact rescoring** — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937); filtered/nested kNN, non-cosine similarity, SQ8 fields, and small indexes run the exact brute-force scan (cosine mapped to `(1+cos)/2`).
+- **Hybrid search** — BM25 + kNN combined in a single request via the `hybrid` **query type** with `rrf|linear|learned` fusion, verified live. (The ES-native top-level `{query, knn}` body does **not** fuse — see *Known partials*.)
+- **Zero-config folder onboarding** — `xerj autoindex <folder>` sniffs files, infers datasets, and creates one index per dataset: tree-sitter AST extraction for 13 languages (symbols, defs, line numbers — expansion tracked in [#295](https://github.com/xerj-org/xerj/issues/295)), CSV/JSON/JSONL/XML/YAML/SQLite/PDF/DOCX/HTML/log formats, `.gitignore`/`.xerjignore` support, incremental re-runs, and a machine-parseable progress stream.
+- **Agent-memory REST API** (`/_memory/*`), **second-brain knowledge graph** (`/_graph`), **anomaly detection** (`_ml` with continuous datafeeds), **auto-embed on ingest** (default embedder is deterministic **lexical** feature-hashing — never described as neural; `--embed-mode neural` runs the in-binary BERT encoder, `--embed-mode proxy` an external endpoint).
+- **Columnar storage** — the ZBS2 columnar block with 9 domain-aware encodings, ZSTD/LZ4 codecs, and SQ8 vector quantization, wired into the segment write path.
+- Bulk / scroll / delete-by-query, aliases, index templates, **executed** index-lifecycle policies (ISM-modeled, `_ilm/*` + `_plugins/_ism/*`, since rc.15), `_cat/*`, `_cluster/health`, `_count` / `_msearch` / `_mget`, `_update` / `_update_by_query` — all live-verified.
+- **A single native binary**, statically linked, no JVM, sub-second cold start.
-The datafeed scheduler that rc-2 lacked: `PUT/GET/DELETE /_ml/datafeeds/{id}` + `_start`/`_stop`, and `GET /_ml/anomaly_detectors/{job}/results/records`. A background task re-buckets a live index on a timer and appends newly-flagged anomaly records you poll — a second spike is detected with no second call. Live-verified end-to-end. Recipe: `recipes/anomaly_datafeed.py`; guide: `docs/recipes/continuous-anomaly-datafeeds.md`.
+The release-by-release record of how all of this landed (rc.1 through rc.15) is [CHANGELOG.md](./CHANGELOG.md) — this file no longer duplicates it.
-- **Limitation:** single-node scheduler; no forecasting/influencers.
+## Next release — [v1.0.0-rc.16](https://github.com/xerj-org/xerj/milestone/1)
-### 6. Ingest-time chunk-embedding pipeline (per-passage vectors) ✅
+rc.15 shipped with two known defects, stated in its changelog entry rather than hidden. **Both are already fixed on `main`** and ship in rc.16:
-Long `semantic_text` values are split into overlapping passages, embedded **per passage**, and the per-passage vectors persisted (in `<field>_vector_chunks`, only when a value spans >1 passage). A `semantic` query scores each document by its **best-matching passage** (max-sim) instead of a single pooled vector, so a long document competes on any one of its sections. Live-verified: on 40 articles + a compendium of all 40, the compendium reached top-3 for **98%** of single-topic queries with per-passage scoring vs **32%** pooled. Short single-passage values are byte-identical to before. Recipe: `recipes/passage_search.py`; guide: `docs/recipes/passage-retrieval.md`.
+- The agent progress stream could be forged by a crafted filename (no control-character sanitisation) — fixed in [#288](https://github.com/xerj-org/xerj/pull/288).
+- An outer repository's `.gitignore` judged files inside a nested checkout (no repository-boundary stop) — fixed in [#287](https://github.com/xerj-org/xerj/pull/287).
-- **Limitation:** per-passage vectors are only as good as the active embedder. The default is lexical; switch to `--embed-mode neural` (built-in BERT) or `--embed-mode proxy` for neural-quality passage vectors — the chunk-embedding pipeline is backend-agnostic. A field that is *also* scalar8-quantized scores against the pooled vector (per-passage max-sim is exact-f32 only).
+Also on `main` since rc.15: dynamic-mapping opacity for nested objects ([#292](https://github.com/xerj-org/xerj/pull/292)), ILM safety rails ([#291](https://github.com/xerj-org/xerj/pull/291)), and AST capture fixes for TypeScript/JavaScript modules and PHP enums/consts (#284, #285, #286).
-### 7. HNSW-served approximate kNN with exact rescoring ✅
+In flight for the cut (open PRs, merged only on green CI): `index.lifecycle.name: null` as a real ILM detach ([#290](https://github.com/xerj-org/xerj/pull/290)), autoindex estimate-first UX — estimate on the user's machine, ask before a long run, index what matters first ([#281](https://github.com/xerj-org/xerj/pull/281)), real cargo-audit + cargo-fuzz in CI and the findings they surfaced ([#261](https://github.com/xerj-org/xerj/pull/261), closes [#207](https://github.com/xerj-org/xerj/issues/207)), the #204 fail-closed/fail-loud sweep ([#258](https://github.com/xerj-org/xerj/pull/258)), the junk-shrink generation invariant ([#296](https://github.com/xerj-org/xerj/pull/296), closes [#283](https://github.com/xerj-org/xerj/issues/283)), and exact float replay in the durable sync journal ([#297](https://github.com/xerj-org/xerj/pull/297)).
-Unfiltered `knn` (top-level or query form) is now served by a **persisted HNSW graph** instead of the exhaustive scan, with every candidate exact-rescored so returned `_score`s stay bit-identical to the brute path. Measured on the official bench cell (50k × 128-d, cosine, k=10): p50 23,325 ms → **1.87 ms** (ES 8.13.4: 1.57 ms — a tie), recall@10 **1.00** vs ES's 0.80; offline 100-probe recall@10 mean **0.976** / min 0.90 (ES same protocol: 0.937 / 0.70). `num_candidates` is honored as the beam width, floored at 800 to match ES's per-segment candidate semantics (ES's 1.5×k default applies when omitted). The graph is persisted at flush/refresh and reloaded at boot with field/freshness stamps; any ineligibility degrades to the exact scan — never to wrong results.
+Open defects targeted at this milestone: the `--no-graph` durable path indexing zero documents for code files ([#294](https://github.com/xerj-org/xerj/issues/294)) and the JavaScript-vs-TypeScript module-const capture gap ([#293](https://github.com/xerj-org/xerj/issues/293)).
-- **Limitation:** ANN serving covers **unfiltered, cosine, full-precision** kNN on indexes ≥1,024 docs only — filtered/nested kNN, `l2_norm`/`dot_product`/`max_inner_product`, SQ8-quantized fields, small indexes, and `semantic`-query vector scoring stay on the exact brute-force scan (recall 1.00, latency scales with vectors scanned). Recall on the ANN path is **measured, not guaranteed**, and `hits.total` can come back below `k`. A missing or stale graph snapshot serves brute until the next flush/refresh re-save; rebuild-from-WAL, HNSW tombstone compaction, and semantic-query routing through the graph are tracked follow-ups.
+## The road to [v1.0.0 GA](https://github.com/xerj-org/xerj/milestone/2)
-### 8. Stub-surface correctness flips ✅
+The 1.0 bar: **every public claim verified against the release binary, and every input either honoured or refused loudly.** The gate list, each item an issue:
-Five query / agg / memory surfaces the roadmap previously listed as *Partial* or *silently-wrong* now implement real semantics (or fail loud). Verified against HEAD:
+- **Close the accepted-and-ignored class** ([#204](https://github.com/xerj-org/xerj/issues/204) umbrella; the sweep starts in PR #258). Known members still open: `dense_vector` `quantization` accepted, echoed, and ignored ([#275](https://github.com/xerj-org/xerj/issues/275)); `nested` `score_mode` parsed-then-ignored and `inner_hits` unparsed; `random_sampler`'s ignored `probability`; `weighted_avg` returning HTTP 200 with an error buried in the aggregations body instead of a 400.
+- **Deterministic tie-breaking** — `constant_score` on a mixed corpus and five post-sort re-sorts tie by `_id` alone ([#270](https://github.com/xerj-org/xerj/issues/270)).
+- **Security tooling as claimed** — cargo-audit and fuzzing actually in CI ([#207](https://github.com/xerj-org/xerj/issues/207), PR #261 in flight), plus the deferred TLS/auth/symlink hardening items from the Phase-2 security backlog.
+- **CLA history hygiene** — the gate is blind to `Co-authored-by` trailers and the pre-enforcement third-party history is unsigned ([#269](https://github.com/xerj-org/xerj/issues/269)).
+- **The mixed read-under-write p99 gap** — the 4 benchmark losses out of 85 measured comparisons, all the same root cause (reads landing on the live memtable under writer pressure). Written up in [`demo/playbooks/MIXED_READ_UNDER_WRITE_FINDING_2026-07-08.md`](./demo/playbooks/MIXED_READ_UNDER_WRITE_FINDING_2026-07-08.md); the candidate fix is a visibility/parity-mode design decision, not a micro-optimisation, and it stays on the GA gate until fixed or explicitly descoped with the benchmark loss kept public.
+- **Ship-or-descope every entry in *Known partials* below.** GA does not ship with a "partial" section that reads like a feature list.
-- **`percolate`** (rc-3) — a real **reverse-search doc-scan**: each indexed document's stored query is parsed and the supplied `document` / `documents` matched against it (`parse_percolate` → `index.rs` `QueryNode::Percolate`, `e72ac71`). No longer hard-coded to `MatchNone`.
-- **`significant_terms`** (rc-3) — real **JLH significance scoring** over a foreground/background split (`run_significant_terms`, `9379578`), emitting `buckets` with `bg_count` + `score`. No longer empty.
-- **`has_child` / `has_parent`** (rc-3) — now **fail loud with a 400** at parse time (`parse_has_child`, `25c6b6c`) instead of silently returning unfiltered hits. Real parent-child join semantics remain future work.
-- **Agent-memory recall** (rc-3) — the REST `/_memory/{ns}/_recall` path now applies **recency-blended scoring** (`recency_weight`) and **semantic dedup** (`dedup` / `dedup_threshold`), not just pure relevance (`memory_api.rs` `blend_recency`, `ec6dc4a`).
-- **`scripted_metric`** (rc-3 interpreter; RC4 fast path) — a real single-shard **Painless interpreter** runs `init` / `map` / `combine` / `reduce` (`run_scripted_metric`), with a columnar doc-values fast path for the canonical "sum one numeric field" shape (`02e13af`). Genuinely-unsupported script constructs degrade to `{"value":null}`, not a wrong number.
+## Beyond 1.0 — themes
-## Landed since rc-3 (on `main`, unreleased — RC4 hardening)
+- **AST language expansion** — 25 further tree-sitter grammars, tiered by demand, one PR per tier ([#295](https://github.com/xerj-org/xerj/issues/295)); Tier 1 (Kotlin, Swift, Scala, Dart, Lua, Perl, R, Julia, Haskell, Elixir) may land earlier in an RC if the grammar/ABI checks prove out.
+- **Distributed clustering maturity** — embedded Raft handles cluster metadata today, but the default run is **single-node**; multi-node sharding/replication hardening is a post-GA track, and XERJ does not claim multi-node production readiness until it is measured.
+- **Neural embedder ergonomics** — share one loaded model across indices (today each index lazily holds its own `NeuralHandle`), optional pre-warm at startup, a larger default model option.
+- **Log-analytics data path** — the dedicated `xerj-logs` columnar module is still not invoked from non-test engine/server code; log-shaped analytics run through ZBS2 + the generic aggregation suite. Wire it or remove it.
+- **Broader aggregation families** — geo/IP/nested/join coverage beyond the current surface; the conformance suite is the measure.
-The RC4 pass (waves 1–4) is release-hardening on top of rc-3: correctness, durability, resource governance, security, and docs honesty. All conformance-gated (ES-compat YAML suite green at 1,360 / 1,363). Highlights, grounded in the commit log:
+## Known partials
-- **Durability & acked-write integrity** — verified WAL prune + power-loss-ordered publish chain (`e83e316`); `wal_sync="sync"` is now honored on the bulk paths and the `wal_batch_ms` fsync loop implemented (`2e1b0b7`, RC4 W1 #9 — see *Durability posture* below); torn-frame WAL recovery (`bbe9e28`); segment-durable delete tombstones (`a0bddae`); refuse-on-corrupt snapshot + data-dir version marker (`1940232`).
-- **Silent-wrong-data fixes** — real per-doc `_version` + ES `seq_no` wire convention (`5029f76`); `_search` rejects the silent-wrong-query class instead of ignoring unknown clauses (`8fe867d`); Painless string comparison fixed (`159f3b4`); real `sum_other_doc_count` + composite bucket-key typing + `multi_terms` bucket-cap 400s (`06d8695`, `5b3d55e`, `5ff04c0`); ES-exact date resolution for range bounds (`f7c9cb0`).
-- **Resource governance / stability** — parent circuit breaker, global search pool, disk flood-stage watermark (`bd24aa0`); search timeout actually fires (`753d0b5`); scroll / async-search context TTL-sweep + caps (`0e47db9`); Painless depth + source-length guards (`fe04e89`).
-- **Security** — persist API keys across restart + honest role surface (`477c12b`); constant-time admin-key compare (`5d9a31c`); restrictive-default configurable CORS (`964a0ef`); key files created `0600` (`47a4ba1`).
-- **Read-path perf (scorecard campaign)** — `scripted_metric` sum shape → columnar `.dv` (+730 ms → 0 ms, `02e13af`); standalone `prefix` / `wildcard` → `constant_score` for ES-exact `max_score` (`4c69c05`); `more_like_this` → `bool.should` of `match` (`c45dfd4`); `range(@timestamp)` cliff + columnar `rare_terms` / `significant_terms` (`9379578`). These moved the 100k / 1-client matrix to **55 W / 26 T / 4 L (3 N/A)** — see `demo/playbooks/SCORECARD.md`.
-- **API-fidelity minors** — `POST /{index}/_doc/{id}` route (`d5c7bc4`); `_cat/indices` uuid + bytes and snapshot shape (`7427d68`); kNN filter / boost / similarity-cutoff threading + `knn:[…]` arrays (`5aecb27`).
+Honesty section: things that resolve without an error but do not implement full ES semantics. Each must be shipped or explicitly descoped before GA.
-Still open (tracked, **not** closed by RC4): the RSS-runaway ingest-heap ticket (repro unclear; identified mechanisms removed + now observable), the mixed read-under-write p99 gap (the 4 scorecard losses), and full parent-child join semantics.
+Re-verified against `main` 2026-08-11:
-## Partial / in progress
+- **`weighted_avg`** — not in `SUPPORTED_AGG_TYPES`; still returns HTTP 200 with an embedded error instead of executing or returning 400 (the 400 is part of the #258 sweep).
+- **`has_child` / `has_parent`** — recognised and rejected with a 400 (fail-loud by design until real parent-child join semantics exist; `REJECTED_QUERY_TYPES` in `parser.rs`).
-### Query types that dispatch but are not yet semantically faithful
+Carried forward from the 2026-07-12 review, not re-verified live since:
-Counting these toward "supported query types" overstates correctness — they resolve without a `400` but do not implement ES semantics:
-
-- **`nested`** — *matching* is real and was mis-described here until issue #211: the inner query is evaluated **per nested element** (`index.rs`, `QueryNode::Nested` → `arr.iter().any(|elem| doc_matches_query(&inner, elem))`), so a document matches only when one element satisfies the whole inner query — covered by `test_nested_query` in `xerj-engine/tests/integration.rs`. What is missing is ES's separate nested-document indexing: `score_mode` is parsed and then ignored, and `inner_hits` is not parsed at all, so neither affects the response. Those two are accepted-and-ignored inputs (#204) and should either be honoured or rejected. (`percolate` and `has_child` / `has_parent` used to sit here — `percolate` is real now and the join queries fail loud; see *Stub-surface correctness flips* above.)
-- **`span_term` / `span_or` / `span_not`** — return 0 hits **standalone**, even though `span_near` / `span_first` / `span_containing` using the same clauses return correct hits. Only composite span queries work.
-- **`type`** — mapped to `MatchAll` (`parser.rs`, the `"type"` arm of `parse_query`; the line number that used to be quoted here was already stale).
+- **`nested`** — matching is real and per-element (`test_nested_query`), but ES's separate nested-document indexing is missing: `score_mode` is parsed and ignored, `inner_hits` is not parsed (#204 members, above).
+- **`span_term` / `span_or` / `span_not`** — return 0 hits **standalone**, while composite span queries (`span_near` / `span_first` / `span_containing`) using the same clauses return correct hits.
+- **`type`** — mapped to `MatchAll`.
- **`combined_fields`** — mapped to `multi_match cross_fields`; scoring is not exact. `rank_feature` passes through on plain fields (no `rank_feature` field type).
-
-### Aggregations that are stubbed or silently degrade
-
-- **`weighted_avg`** — returns **HTTP 200 with an embedded `{"error":"unsupported aggregation type 'weighted_avg'"}`** buried in the aggregations result instead of a value or a `400`. Silent-failure honesty gap; should `400`. (`scripted_metric` and `significant_terms` used to sit here — both execute for real now; see *Stub-surface correctness flips* above.)
-
-### Hybrid / vector wire-compat
-
-- The **ES-native top-level `{query, knn}`** body does **not** union the kNN hits (live: only the lexical match was returned; the best vector match was dropped). One-request BM25+kNN fusion works only through the explicit `hybrid` query type.
-- `POST /{index}/_doc/{id}` returns `405` (only `PUT`/`GET`/`HEAD`/`DELETE` allowed); real ES accepts `POST` there. Minor wire-compat deviation.
-
-### Distributed clustering maturity
-
-- Embedded Raft (`raft.rs`, `replication.rs`, `transport.rs` — self-contained, no external raft crate) handles cluster metadata today, but the default run is **single-node** (`number_of_nodes:1`); multi-node sharding/replication hardening is ongoing.
-
-### Log analytics data path
-
-- The dedicated `xerj-logs` columnar module (delta-of-delta timestamps + dictionary strings) is declared as an engine dependency but **`xerj_logs::` is never invoked in non-test engine/server source** — effectively unwired. The runtime columnar path is `xerj-storage`'s ZBS2, and log-shaped analytics run through the generic ES aggregation suite (`date_histogram`, etc.). Wiring or removing the dead module is tracked work.
-
-### Benchmark honesty (tracked docs fix)
-
-- The reproducible benchmark of record is `demo/playbooks/SCORECARD.md` / `BENCHMARK_VS_ES.md` (100k / 1-client, cache-off, keep-alive transport): **55 W / 4 L / 26 T (3 N/A)** vs live ES 8.13.4, with a **1.72×** ingest win (191k vs 111k docs/s), a **1.61×** smaller on-disk footprint (176 vs 283 MB), aggregations often order-of-magnitude, and the 4 losses all mixed read-under-write p99. The older website headline perf claims (74× SIEM, 21× memory, 2.8× disk, 89× NGINX, 300× cold start, 56× binary) cited battle-report files (`SIEM_BATTLE_…`, `CLUSTER_BATTLE_…`, `HEAD_TO_HEAD_M3_…`) that **do not exist in the repo**; the reproducible closed-loop numbers now replace them across the public-sector / solutions / resources / brandbook pages (RC4 W4 #10).
-
-## Planned / not yet started
-
-### Neural embeddings & richer ML
-
-- A built-in **neural** embedder has **landed and ships in the default binary**: `--embed-mode neural` runs an in-process BERT sentence encoder (all-MiniLM-L6-v2, 384-dim) via `candle` — pure Rust, no Python, no external service, model auto-downloads on first use (air-gap friendly via `embedding.local_model_dir`). No rebuild and no separate artifact needed; the shipped binary is ~36 MB (a `--no-default-features` slim build without the neural backend is ~23 MB). **Remaining work:** (a) share one loaded model across all indices — today each `Index` holds its own lazily-loaded `NeuralHandle`, so a node serving several *semantic* indices can hold multiple copies of the weights in RAM (loads are lazy, so indices that never receive a semantic query never load it); (b) offer a larger/higher-quality default model option; (c) optionally pre-warm the model at startup so the one-time download happens at launch rather than on the first query.
-- Forecasting for capacity/write-load signals (continuous `_ml` datafeeds have landed — see "Landed since rc-2"; the ingest-time per-passage chunk-embedding pipeline has also landed).
-
-### Correctness of stubbed surface
-
-- Real join / parent-child semantics for `has_child` / `has_parent`, and nested-document indexing behind `nested` (`score_mode` / `inner_hits`).
-- Standalone `span_term` / `span_or` / `span_not`.
-- `weighted_avg` execution (and returning `400` for genuinely-unsupported aggs rather than a buried error). (`percolate`, `scripted_metric`, and `significant_terms` have shipped — see *Stub-surface correctness flips*.)
-- ES-native top-level `{query, knn}` fusion.
-
-### Other tracked items
-
-- **Distributed clustering maturity** — embedded Raft handles cluster metadata today; multi-node sharding/replication hardening is ongoing.
-- **Broader aggregation coverage** — geo/IP/nested/join/span families are partially covered; see the conformance suite and `demo/playbooks/ES_COMPATIBILITY.md` for the current surface.
+- **ES-native top-level `{query, knn}`** — does not union the kNN hits; one-request BM25+kNN fusion works only through the explicit `hybrid` query type.
---