AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs+test: the phantom survived in the examples — fix them, and let the guard see them
xerj-orgd470eecfed44
diff --git a/engine/README.md b/engine/README.mdindex 31c797e6f..57a22920a 100644--- a/engine/README.md+++ b/engine/README.md@@ -63,8 +63,12 @@ to close — sharded flush thresholds + doc-values borrow refactor).## Features-- **100% ES API compatible** — drop-in replacement, existing clients work unchanged-- **BM25 full-text search** with highlighting, aggregations, and all ES query types+- **Broad ES 8.x wire compatibility** — point an existing client at port 9200 and+ most of it just works. Wide, not total: the measured coverage and the gaps are+ in [demo/playbooks/ES_COMPATIBILITY.md](../demo/playbooks/ES_COMPATIBILITY.md)+- **BM25 full-text search** with highlighting, aggregations, and every query type+ in [Query Types Supported](#query-types-supported) below — plus two more that+ are recognised and deliberately refused with a 400 rather than answered wrongly- **Vector search** with HNSW, filtered ANN, and inline embedding- **SQL API** — query with SQL syntax- **WAL persistence** — crash recovery, data survives restarts@@ -197,9 +201,18 @@ relationship). Any other query type answers `unknown query type`.## Aggregation Types SupportedComplete and machine-checked the same way, from-`xerj_engine::aggs::SUPPORTED_AGG_TYPES`. All xerj aggregations are exact —-`cardinality` is a true distinct count, not an HLL estimate, and `terms`-`doc_count` is precise.+`xerj_engine::aggs::SUPPORTED_AGG_TYPES`. No probabilistic sketch sits in the+metric path — `cardinality` is a true distinct count, not an HLL estimate, and+`terms` `doc_count` is precise.++The **sampling family is the deliberate exception**, and it is a sample by+definition: `run_sampler` sorts the matched documents by `_score` and keeps the+first `shard_size` (default 200), so every sub-aggregation under `sampler` or+`random_sampler` is computed over that slice rather than the whole match set.+`diversified_sampler` truncates the same way and additionally caps how many+documents may share a `field` value (`max_docs_per_value`, default 1).+`random_sampler` shares the `sampler` implementation and does not read ES's+`probability`.<!-- generated:agg-types -->Metric — `avg`, `sum`, `min`, `max`, `stats`, `extended_stats`, `value_count`,diff --git a/engine/crates/xerj-engine/tests/docs_capability_lists.rs b/engine/crates/xerj-engine/tests/docs_capability_lists.rsindex c5d499d29..49e67a7a2 100644--- a/engine/crates/xerj-engine/tests/docs_capability_lists.rs+++ b/engine/crates/xerj-engine/tests/docs_capability_lists.rs@@ -390,3 +390,302 @@ fn a_backticked_qualifier_inside_a_region_is_refused() {"<!-- generated:query-types -->\n`knn (HNSW-served)`\n<!-- /generated:query-types -->";documented(doc, "synthetic", "query-types");}++// ---------------------------------------------------------------------------+// Beyond the marked regions: the *examples*.+//+// The marked-region checks above only see a list that opted in. That left a+// hole big enough for the original defect to survive the first fix: while+// `landing/docs/queries.html` stopped advertising a `semantic_search` card,+// `landing/docs/playbooks/vector-search.html` went on publishing+// `{"semantic_search": {"field": …, "text": …}}` as *the* worked example for+// the flagship AI-native feature, and `landing/docs/migration-from-es.html`+// went on naming it in the migrator's "what works day one" list. Sent to a+// live instance, that body answers+// `unknown query type `semantic_search`` with a 400.+//+// So the guard now also reads the prose and the samples, over every published+// surface rather than the three files that carry markers.+// ---------------------------------------------------------------------------++/// Every published surface whose text is scanned for capability *claims*+/// (as opposed to the marked lists): the whole docs site, the agent-facing+/// text files, and the two repository documents an evaluator reads first.+fn doc_surfaces() -> Vec<PathBuf> {+ fn walk(dir: &Path, out: &mut Vec<PathBuf>) {+ let Ok(entries) = std::fs::read_dir(dir) else {+ return;+ };+ for e in entries.flatten() {+ let p = e.path();+ if p.is_dir() {+ walk(&p, out);+ } else {+ let keep = p.extension().is_some_and(|x| x == "html")+ || p.file_name()+ .and_then(|n| n.to_str())+ .is_some_and(|n| n.starts_with("llms") && n.ends_with(".txt"));+ if keep {+ out.push(p);+ }+ }+ }+ }++ let root = repo_root();+ let mut files = Vec::new();+ walk(&root.join("landing"), &mut files);+ files.push(root.join("engine/README.md"));+ files.push(root.join("ROADMAP.md"));+ files.sort();+ assert!(+ files.len() > 20,+ "only {} published surfaces found — this check is reading nothing",+ files.len()+ );+ files+}++fn known_query_types() -> BTreeSet<String> {+ xerj_query::parser::SUPPORTED_QUERY_TYPES+ .iter()+ .chain(xerj_query::parser::REJECTED_QUERY_TYPES)+ .map(|s| s.to_string())+ .collect()+}++/// `true` when the byte at `at` starts a token — i.e. it is not preceded by a+/// character that would make it part of a longer identifier. Keeps+/// `xerj_semantic_search` (a real MCP tool) from reading as `semantic_search`+/// (a query type that has never existed).+fn at_token_start(text: &str, at: usize) -> bool {+ text[..at]+ .chars()+ .next_back()+ .is_none_or(|c| !c.is_ascii_alphanumeric() && c != '_')+}++fn token_ends_at(text: &str, end: usize) -> bool {+ text[end..]+ .chars()+ .next()+ .is_none_or(|c| !c.is_ascii_alphanumeric() && c != '_')+}++fn line_of(text: &str, byte: usize) -> usize {+ text[..byte].matches('\n').count() + 1+}++/// Walk `text` from `from`, skipping whitespace, and require it to spell out+/// `wanted` in order. Returns the byte offset just past the last character, or+/// `None` the moment the text diverges.+fn expect_seq(text: &str, from: usize, wanted: &[char]) -> Option<usize> {+ let mut want = wanted.iter();+ let mut next = want.next()?;+ for (off, ch) in text[from..].char_indices() {+ if ch.is_whitespace() {+ continue;+ }+ if ch != *next {+ return None;+ }+ match want.next() {+ Some(n) => next = n,+ None => return Some(from + off + ch.len_utf8()),+ }+ }+ None+}++/// Query-type names the docs have published that the parser has never+/// dispatched. `semantic_search` (the real clause is `semantic`) and `boosted`+/// (the real clause is `boosting`) both shipped on the docs site; a reader who+/// copied either got a 400.+///+/// This is a denylist rather than a general rule because prose is prose: there+/// is no position in an English sentence that reliably marks a word as a+/// capability claim. It is kept honest by+/// [`the_phantom_list_only_holds_names_the_parser_really_lacks`], which retires+/// an entry the moment the parser grows it.+const PHANTOM_QUERY_TYPES: &[&str] = &["boosted", "semantic_search"];++/// If XERJ ever implements one of these, the denylist must shrink, not silently+/// forbid documenting a real feature.+#[test]+fn the_phantom_list_only_holds_names_the_parser_really_lacks() {+ let known = known_query_types();+ let wrong: Vec<_> = PHANTOM_QUERY_TYPES+ .iter()+ .filter(|p| known.contains(**p))+ .collect();+ assert!(+ wrong.is_empty(),+ "{wrong:?} are now dispatched by the parser — remove them from \+ PHANTOM_QUERY_TYPES and document them instead of banning them"+ );+}++/// No published surface may name a query type that does not exist — not in a+/// list, not in a sentence, not in a copy-pasteable example.+#[test]+fn no_published_surface_names_a_phantom_query_type() {+ let root = repo_root();+ let mut hits = Vec::new();++ for file in doc_surfaces() {+ let text = std::fs::read_to_string(&file).unwrap_or_default();+ for phantom in PHANTOM_QUERY_TYPES {+ for (at, _) in text.match_indices(phantom) {+ if at_token_start(&text, at) && token_ends_at(&text, at + phantom.len()) {+ hits.push(format!(+ "{}:{} names `{phantom}`",+ file.strip_prefix(&root).unwrap_or(&file).display(),+ line_of(&text, at)+ ));+ }+ }+ }+ }++ hits.sort();+ assert!(+ hits.is_empty(),+ "the published docs name query types the parser has never had:\n {}\n\+ The real clauses are `semantic` (field + query, k defaults to 10) and \+ `boosting`. A reader who copies a phantom gets `unknown query type`.",+ hits.join("\n ")+ );+}++/// Anything in *query position* in a published sample must be a real query+/// type. A clause sitting directly under a `"query": { … }` object is a query+/// type by construction — that is the one place in an ES request body where the+/// key's meaning is unambiguous — so it can be checked without a denylist.+#[test]+fn every_query_clause_in_a_published_sample_is_a_real_query_type() {+ let root = repo_root();+ let known = known_query_types();+ let mut bad = Vec::new();+ let mut checked = 0usize;++ for file in doc_surfaces() {+ let text = std::fs::read_to_string(&file).unwrap_or_default();+ for (at, _) in text.match_indices("\"query\"") {+ // `"query"` `:` `{` `"<name>"` — anything else (a string value, as+ // in `match` / `query_string` / `semantic`, or an array) is not a+ // clause position and is skipped.+ let Some(from) = expect_seq(&text, at + "\"query\"".len(), &[':', '{', '"']) else {+ continue;+ };+ let Some(len) = text[from..].find('"') else {+ continue;+ };+ let name = &text[from..from + len];+ if name.is_empty()+ || !name+ .chars()+ .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_')+ {+ continue; // not an ES type name — a doc field, a label, prose+ }+ checked += 1;+ if !known.contains(name) {+ bad.push(format!(+ "{}:{} uses `{name}` in query position",+ file.strip_prefix(&root).unwrap_or(&file).display(),+ line_of(&text, at)+ ));+ }+ }+ }++ assert!(+ checked > 20,+ "only {checked} query clauses found across the docs — this check is reading nothing"+ );+ bad.sort();+ bad.dedup();+ assert!(+ bad.is_empty(),+ "published request samples put a name in query position that \+ `parse_query` does not dispatch:\n {}",+ bad.join("\n ")+ );+}++/// Published *counts* drift exactly like published lists, and were drifting in+/// three different directions at once: `landing/docs/migration-from-es.html`+/// said 32 query types, `landing/pricing/index.html` and+/// `landing/demo/index.html` said 38, and the parser dispatches 50. So a number+/// a reader is expected to trust now lives in a marked region too.+///+/// Unlike the list regions, a file may carry the same count section more than+/// once (the pricing page prints the figure in a plan bullet and again in the+/// comparison table); every occurrence is checked.+const COUNT_DOCS: &[&str] = &[+ "landing/docs/migration-from-es.html",+ "landing/pricing/index.html",+ "landing/demo/index.html",+];++/// The count sections and where each number comes from.+fn count_source_of_truth(section: &str) -> usize {+ match section {+ "query-type-count" => xerj_query::parser::SUPPORTED_QUERY_TYPES.len(),+ "rejected-query-type-count" => xerj_query::parser::REJECTED_QUERY_TYPES.len(),+ "agg-type-count" => xerj_engine::aggs::SUPPORTED_AGG_TYPES.len(),+ other => panic!("no source of truth wired up for count section `{other}`"),+ }+}++const COUNT_SECTIONS: &[&str] = &[+ "query-type-count",+ "rejected-query-type-count",+ "agg-type-count",+];++#[test]+fn published_capability_counts_match_the_constants() {+ let root = repo_root();+ let mut seen = 0usize;++ for rel in COUNT_DOCS {+ let doc = std::fs::read_to_string(root.join(rel))+ .unwrap_or_else(|e| panic!("cannot read {rel}: {e}"));++ for section in COUNT_SECTIONS {+ let open = format!("<!-- generated:{section} -->");+ let close = format!("<!-- /generated:{section} -->");+ for (at, _) in doc.match_indices(&open) {+ let from = at + open.len();+ let len = doc[from..].find(&close).unwrap_or_else(|| {+ panic!(+ "{rel} opens `{open}` at line {} but never closes it",+ line_of(&doc, at)+ )+ });+ let raw = doc[from..from + len].trim();+ let published: usize = raw.parse().unwrap_or_else(|_| {+ panic!(+ "{rel} section `{section}` holds `{raw}`, which is not a number. \+ The region is machine-checked and may hold only the count."+ )+ });+ assert_eq!(+ published,+ count_source_of_truth(section),+ "{rel}:{} publishes {published} for `{section}`; the source has {}",+ line_of(&doc, at),+ count_source_of_truth(section)+ );+ seen += 1;+ }+ }+ }++ assert!(+ seen >= 5,+ "only {seen} marked counts found across {COUNT_DOCS:?} — a page dropped its \+ markers and its number is now unchecked"+ );+}diff --git a/landing/demo/index.html b/landing/demo/index.htmlindex e9737e857..ef9d71b05 100644--- a/landing/demo/index.html+++ b/landing/demo/index.html@@ -1214,7 +1214,7 @@ node demo/scripts/full-demo-flow.js<div class="small">98.2 % of ES 8.13 official YAML conformance tests pass.The remaining 24 are documented in<a href="https://github.com/xerj-org/xerj/blob/main/engine/reports/PATH_TO_100_PCT_v0.6.0_to_v1.0.md"><code>PATH_TO_100_PCT</code></a>.- This includes search DSL, all 38 query types, all aggs, bulk, scroll,+ This includes search DSL, all <!-- generated:query-type-count -->50<!-- /generated:query-type-count --> query types, all aggs, bulk, scroll,index templates, aliases, knn, hybrid, semantic.</div><div class="source">measured: <code>cargo run -p es-yaml-runner</code></div></div>diff --git a/landing/docs/aggregations.html b/landing/docs/aggregations.htmlindex 76ca5cf95..67c016790 100644--- a/landing/docs/aggregations.html+++ b/landing/docs/aggregations.html@@ -119,7 +119,7 @@<h1>Aggregations</h1>- <p>Every bucket aggregation returns an exact count — no HyperLogLog approximations in the bucket path. Once an index passes 10,000 live documents, <code class="inline">terms</code> and the other columnar-eligible aggregations read a doc-values ordinal histogram instead of scanning postings (<code class="inline">fast_aggs.rs</code>); below that the JSON path answers directly and stays the reference implementation.</p>+ <p>Every bucket aggregation returns an exact count — no HyperLogLog approximations in the bucket path. The one deliberate exception is the sampling family below (<code class="inline">sampler</code>, <code class="inline">random_sampler</code>, <code class="inline">diversified_sampler</code>), which is a sample by definition: it truncates to the top <code class="inline">shard_size</code> matched documents by <code class="inline">_score</code>, and its sub-aggregations see only that slice. Once an index passes 10,000 live documents, <code class="inline">terms</code> and the other columnar-eligible aggregations read a doc-values ordinal histogram instead of scanning postings (<code class="inline">fast_aggs.rs</code>); below that the JSON path answers directly and stays the reference implementation.</p><p>The list below is the <em>complete</em> set of aggregation keys the engine dispatches, generated from <code class="inline">xerj_engine::aggs::SUPPORTED_AGG_TYPES</code> and held to it by a test. Being listed means the aggregation runs and returns the ES response shape — it is not a promise that every parameter of every type is honoured.</p>@@ -173,9 +173,9 @@<div class="item">geo_distance</div><div class="item">geohash_grid</div><div class="item">geotile_grid</div>- <div class="item">sampler</div>- <div class="item">diversified_sampler</div>- <div class="item">random_sampler</div>+ <div class="item">sampler<span class="dsc">The one deliberate exception to exactness: keeps the top shard_size matched docs by _score (default 200), so sub-aggregations run over that sample.</span></div>+ <div class="item">diversified_sampler<span class="dsc">Same truncation, plus a cap of max_docs_per_value docs sharing one field value (default 1).</span></div>+ <div class="item">random_sampler<span class="dsc">Shares the sampler implementation — same top-by-_score slice; ES's probability parameter is not read.</span></div><div class="item">time_series</div></div>diff --git a/landing/docs/migration-from-es.html b/landing/docs/migration-from-es.htmlindex 450e1e3d9..5f76fdd6d 100644--- a/landing/docs/migration-from-es.html+++ b/landing/docs/migration-from-es.html@@ -126,8 +126,8 @@<li><strong>Index CRUD:</strong> <code class="inline">PUT /:index</code>, <code class="inline">DELETE /:index</code>, <code class="inline">GET /:index</code>, <code class="inline">PUT /:index/_mapping</code>, <code class="inline">GET /:index/_mapping</code>.</li><li><strong>Document CRUD:</strong> <code class="inline">POST /:index/_doc</code>, <code class="inline">PUT /:index/_doc/:id</code>, <code class="inline">GET /:index/_doc/:id</code>, <code class="inline">DELETE /:index/_doc/:id</code>.</li><li><strong>Bulk:</strong> <code class="inline">POST /:index/_bulk</code> with index/create/update/delete actions.</li>- <li><strong>Query DSL:</strong> 32 query types including match, bool, range, wildcard, phrase, span, fuzzy, and AI-native (knn, semantic_search, hybrid).</li>- <li><strong>Aggregations:</strong> terms, range, histogram, date_histogram, filter, missing, composite, plus the metric family (avg, sum, min, max, stats, cardinality, percentiles).</li>+ <li><strong>Query DSL:</strong> <!-- generated:query-type-count -->50<!-- /generated:query-type-count --> query types including match, bool, range, wildcard, phrase, span, fuzzy, and AI-native (knn, semantic, hybrid) — the full list is on <a href="/docs/queries.html">Query types</a>. A further <!-- generated:rejected-query-type-count -->2<!-- /generated:rejected-query-type-count --> are recognised and <em>deliberately</em> refused with a 400 (<code class="inline">has_child</code>, <code class="inline">has_parent</code>): XERJ never materialises a parent/child join, so running them would match against flat documents and return silently wrong hits.</li>+ <li><strong>Aggregations:</strong> <!-- generated:agg-type-count -->62<!-- /generated:agg-type-count --> aggregations across the metric, bucket, sampling, scope, geo <em>and pipeline</em> families — <code class="inline">bucket_script</code>, <code class="inline">derivative</code>, <code class="inline">moving_fn</code> and the rest of the pipeline family included. Full list on <a href="/docs/aggregations.html">Aggregations</a>.</li><li><strong>Cluster endpoints:</strong> <code class="inline">GET /_cluster/health</code>, <code class="inline">GET /_cat/indices</code>.</li><li><strong>Delete by query:</strong> <code class="inline">POST /:index/_delete_by_query</code>.</li></ul>diff --git a/landing/docs/playbooks/vector-search.html b/landing/docs/playbooks/vector-search.htmlindex 2883e6824..a21127b18 100644--- a/landing/docs/playbooks/vector-search.html+++ b/landing/docs/playbooks/vector-search.html@@ -157,24 +157,32 @@ timeout_ms = 5000</pre>}</pre><h2>Hybrid · BM25 + kNN fusion</h2>- <p>One request, one planner pass, one round trip.</p>+ <p>One request, one planner pass, one round trip. <code class="inline">hybrid</code> is a query clause, so it goes inside <code class="inline">query</code>, and each entry of <code class="inline">queries</code> wraps its clause in <code class="inline">query</code> (an optional <code class="inline">weight</code> sits beside it for <code class="inline">"fusion": "linear"</code>).</p><pre class="code">{- "hybrid": {- "fusion": "rrf",- "queries": [- { "match": { "text": "kernel panic after kernel 6.1 upgrade" } },- { "knn": { "field": "embedding", "query_vector": [...], "k": 50 } }- ]+ "query": {+ "hybrid": {+ "fusion": "rrf",+ "queries": [+ { "query": { "match": { "text": "kernel panic after kernel 6.1 upgrade" } } },+ { "query": { "knn": { "field": "embedding", "query_vector": [0.12, 0.08, -0.31], "k": 50 } } }+ ]+ }},"size": 10}</pre><h2>Semantic search · embed at query time</h2>- <p>Skip the client-side embedding step. XERJ embeds the query using the configured endpoint.</p>+ <p>Skip the client-side embedding step. Map a field as <code class="inline">semantic_text</code> and XERJ embeds both the document at ingest and the query at search time — with the built-in lexical feature-hashing embedder by default, or with the endpoint configured in <code class="inline">[embedding]</code> above. The clause is <code class="inline">semantic</code>; it takes <code class="inline">field</code> and <code class="inline">query</code>, and <code class="inline">k</code> defaults to 10.</p>+<pre class="code">$ curl -sX PUT http://localhost:9200/docs/_mapping \+ -H 'Content-Type: application/json' \+ -d '{ "properties": { "summary": { "type": "semantic_text" } } }'</pre><pre class="code">{- "semantic_search": {- "field": "embedding",- "text": "how do I rotate api keys without downtime"+ "query": {+ "semantic": {+ "field": "summary",+ "query": "how do I rotate api keys without downtime",+ "k": 10+ }}}</pre>diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 7f3f25e6c..22f630e41 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -481,8 +481,14 @@ Any other query type answers `unknown query type`.Search body options: from, size, sort, _source (include/exclude), highlight, aggs,track_total_hits, and (via dedicated endpoints) scroll and point-in-time.-Aggregations (complete; all exact — cardinality is a true distinct count, not an-HLL estimate, and terms doc_count is precise):+Aggregations (complete; no probabilistic sketch in the metric path — cardinality+is a true distinct count, not an HLL estimate, and terms doc_count is precise).+The Sampling family below is the deliberate exception: sampler and+random_sampler keep only the top `shard_size` matched docs by _score (default+200) and diversified_sampler additionally caps docs per field value, so+sub-aggregations under any of the three run over that sample, not the whole+match set — and random_sampler shares the sampler implementation and ignores+ES's `probability`:<!-- generated:agg-types -->Metric `avg`, `sum`, `min`, `max`, `stats`, `extended_stats`, `value_count`,diff --git a/landing/pricing/index.html b/landing/pricing/index.htmlindex bf6fc443e..974711d8c 100644--- a/landing/pricing/index.html+++ b/landing/pricing/index.html@@ -56,7 +56,7 @@<p class="body" style="color:var(--z-mute); margin-top:4px;">POC · dev · OSS projects · single node</p><ul><li>Full feature set, single-node</li>- <li>ES 8.13 wire protocol · 38 query types</li>+ <li>ES 8.13 wire protocol · <!-- generated:query-type-count -->50<!-- /generated:query-type-count --> query types</li><li>kNN (HNSW + exact) + SQ8 · BM25 · columnar logs · agent memory</li><li>Community support · GitHub · forum</li><li>No SLA · best-effort response</li>@@ -135,7 +135,7 @@<tr class="section-row"><td colspan="4">Platform</td></tr><tr>- <td class="cell-feat">ES 8.13 wire protocol <span class="desc">drop-in for existing clients · 38 query types</span></td>+ <td class="cell-feat">ES 8.13 wire protocol <span class="desc">drop-in for existing clients · <!-- generated:query-type-count -->50<!-- /generated:query-type-count --> query types</span></td><td class="cell-check">✓</td><td class="cell-check">✓</td><td class="cell-check">✓</td></tr><tr>