AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(security): finish the #207 date-math fix and close the same defect in the agg script tokenisers
xerj-org69d7170cf361
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d866b90a0..d7ba4b1ca 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -19,11 +19,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0that was not there.- New CI job `security-audit` runs `cargo audit` on every push and pullrequest; a RUSTSEC vulnerability advisory fails the build.- - New CI job `fuzz` builds and runs six libFuzzer harnesses+ - New CI job `fuzz` builds and runs seven libFuzzer harnesses(`engine/fuzz/`) over checked-in seeds: the Elasticsearch query DSL, theLucene `query_string` grammar, date math and date-format patterns,- index-name date math, SQL, and Painless scripts. `.github/scripts/fuzz-smoke.sh` is the same entry- point locally and in CI.+ index-name date math, SQL, Painless scripts, and the aggregation engine's+ two separate script tokenisers. `.github/scripts/fuzz-smoke.sh` is the same+ entry point locally and in CI.- `xerj-engine/tests/security_tooling_claims.rs` fails if a documented toolstops running, if a fuzz target ships without a seed corpus, or if the"all input parsing paths" overclaim reappears in any prose file.@@ -140,10 +141,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0code (`parser.rs:931`), the release profile sets `panic = "abort"`, and theresult is an unauthenticated remote denial of service on a released binary.Fixed with the fallible `try_*` constructors. Grepping for the same pattern- found two more copies in the index-name date-math resolvers- (`xerj-engine/src/index.rs`, `xerj-api/src/es_compat.rs`), reachable from a- request URI as `<logs-{now+9999999999999d}>`; both are fixed too, and their- `n * 30` / `n * 365` multiplications are now checked.+ found **three** more copies in the index-name date-math resolvers —+ `xerj-engine/src/index.rs`, and *both* branches of+ `xerj-api/src/es_compat.rs`: the `now` form (`<logs-{now+9999999999999d}>`)+ and the anchored form (`<logs-{2026-01-01||+9999999999999d}>`), which are+ handled by two different functions sixty lines apart. All three are fixed,+ their `n * 30` / `n * 365` multiplications are `checked_mul`, the unit is read+ as one character rather than one byte, and the addition goes through+ `checked_add_signed`. `es_compat`'s resolver is the one that runs on the wire+ — the index path parameter of every create/index call and the `_search` index+ list — so `crates/xerj-api/tests/index_name_date_math_is_total.rs` now drives+ it at the crate boundary; the same-file fix that missed the anchored branch+ passed because nothing exercised the public entry point.++- **A `_search` aggregation script containing one non-ASCII character aborted+ the process.** `painless::tokenize` was not the only Painless-subset scanner+ in the tree: `aggs.rs` holds two more with the identical byte-walk /+ `&str`-slice shape — `lex_script` (`scripted_metric`) and `tokenize_script`+ (`bucket_script` / `bucket_selector`) — and both took `&src[i..i + 2]` to test+ for a two-character operator at a byte index that can sit inside a multi-byte+ character, because every arm above it is ASCII-only.+ `{"m":{"scripted_metric":{"map_script":"中"}}}` was `end byte index 2 is not a+ char boundary`. Both also skipped whitespace with `is_whitespace()` on+ `bytes[i] as char`, which is true for Latin-1 NEL and NBSP — bytes that in+ valid UTF-8 are only ever *continuation* bytes. The two-character probe now+ requires both bytes to be ASCII (every such operator is ASCII, so nothing is+ lost) and whitespace is `is_ascii_whitespace`. A new `agg_script` fuzz target+ covers all three entry points, which is how the next one gets found instead of+ reported.- **An XML entity nobody declared no longer discards the text around it.**Fallout from the `quick-xml` upgrade above, which is a breaking change to howdiff --git a/SECURITY.md b/SECURITY.mdindex 1df655077..464c617db 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -86,9 +86,11 @@ worth reporting.harnesses in [`engine/fuzz/`](engine/fuzz/), covering the parsers anunauthenticated request reaches: the Elasticsearch query DSL, the Lucene`query_string` grammar, date math and date-format patterns, index-name date- math, SQL, and Painless scripts. Each target replays a checked-in seed corpus- and then fuzzes for a bounded budget. That list is the coverage — it is not- every parser in the engine.+ math, SQL, `xerj_engine::painless` scripts, and the two separate script+ tokenisers inside the aggregation engine (`scripted_metric`, and+ `bucket_script` / `bucket_selector`). Each target replays a checked-in seed+ corpus and then fuzzes for a bounded budget. That list is the coverage — it is+ not every parser in the engine.Neither is a substitute for a report. Fuzzing finds crashes and hangs, not logicflaws, and an advisory database only knows about vulnerabilities somebody hasdiff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 3279cd622..d44f88252 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -14722,6 +14722,26 @@ fn resolve_date_math_expr(expr: &str) -> String {/// Apply the math portion of an anchored expression (everything after `||`)./// Handles `-1d`, `+2h`, `/d`, and empty tail.+///+/// Third copy of the date-math defect the `date_math` fuzz target found in+/// `xerj_query::dates::add_unit` (#207), and the one the first pass at that+/// issue missed: this is the *anchored* branch of the index-name resolver+/// (`<logs-{2026-01-01||+1d}>`), sixty lines above the `now` branch that was+/// patched. Everything below is reached from a request URI through the public+/// `resolve_date_math_index` — the index path parameter of every create/index+/// call and the `_search` index list — and the release profile sets+/// `panic = "abort"`, so each of these was an unauthenticated process kill:+///+/// * `+9999999999999d` → `TimeDelta::days out of bounds`+/// * `+99999999999h` → `DateTime + TimeDelta overflowed`+/// * `+9223372036854775807M` → `n * 30` overflows before `Duration` is asked+/// * `+1中` / `/中` → `byte index 1 is not a char boundary`+///+/// The fixes are the same three the `now` branch got: read the unit as one+/// CHARACTER, build the offset with the fallible `try_*` constructors and+/// `checked_mul`, and add it with `checked_add_signed`. An unrepresentable+/// offset degrades to "no offset", which is what this resolver already does+/// with an unrecognised unit — it has no error channel to report into.fn apply_date_math_tail(tail: &str,base: chrono::DateTime<chrono::Utc>,@@ -14735,28 +14755,30 @@ fn apply_date_math_tail(let (num_str, r2) =r.split_at(r.find(|c: char| !c.is_ascii_digit()).unwrap_or(r.len()));let n: i64 = num_str.parse().unwrap_or(0) * sign;- let (unit, r3) = if !r2.is_empty() {- (&r2[..1], &r2[1..])- } else {- ("", r2)+ // One character, not one byte: the digit scan stops at the first+ // non-digit, which may be multi-byte, and `r2[..1]` would split it.+ let (unit, r3) = match r2.chars().next() {+ Some(c) => r2.split_at(c.len_utf8()),+ None => ("", r2),};let offset = match unit {- "d" => Duration::days(n),- "h" => Duration::hours(n),- "m" => Duration::minutes(n),- "s" => Duration::seconds(n),- "w" => Duration::weeks(n),- "M" => Duration::days(n * 30),- "y" => Duration::days(n * 365),- _ => Duration::zero(),- };- dt += offset;+ "d" => Duration::try_days(n),+ "h" => Duration::try_hours(n),+ "m" => Duration::try_minutes(n),+ "s" => Duration::try_seconds(n),+ "w" => Duration::try_weeks(n),+ "M" => n.checked_mul(30).and_then(Duration::try_days),+ "y" => n.checked_mul(365).and_then(Duration::try_days),+ _ => Some(Duration::zero()),+ }+ .unwrap_or_else(Duration::zero);+ dt = dt.checked_add_signed(offset).unwrap_or(dt);rest = r3;} else if let Some(r) = rest.strip_prefix('/') {- let (unit, r2) = if !r.is_empty() {- (&r[..1], &r[1..])- } else {- ("", r)+ // Same one-character read as above: `/中` panicked on `r[..1]`.+ let (unit, r2) = match r.chars().next() {+ Some(c) => r.split_at(c.len_utf8()),+ None => ("", r),};dt = round_date_down(dt, unit);rest = r2;diff --git a/engine/crates/xerj-api/tests/index_name_date_math_is_total.rs b/engine/crates/xerj-api/tests/index_name_date_math_is_total.rsnew file mode 100644index 000000000..f3cc7d3f8--- /dev/null+++ b/engine/crates/xerj-api/tests/index_name_date_math_is_total.rs@@ -0,0 +1,121 @@+//! `resolve_date_math_index` must be total on every index name a request can+//! carry.+//!+//! Issue #207. The `date_math` fuzz target found that `chrono`'s+//! `Duration::days`/`hours`/… **panic** on a count that does not fit a+//! `TimeDelta`, and that a byte-indexed `[..1]` splits a multi-byte unit+//! character in half. Three copies of that pattern existed. The first pass at+//! #207 fixed two of them — `xerj_engine::index::resolve_date_math` and the+//! `now` branch here — and missed the third: `apply_date_math_tail`, the+//! *anchored* branch (`<logs-{2026-01-01||+1d}>`) sixty lines above the hunk+//! that was patched.+//!+//! That matters because this resolver is the one that runs on the wire. It is+//! called on the index path parameter in `create_index` (`es_compat.rs:671`),+//! on alias resolution (`:825`) and on the `_search` index list (`:6763`), so+//! every input below arrives in a request URI from an unauthenticated client,+//! and the release profile sets `panic = "abort"` — each one was the process,+//! not a 400.+//!+//! This test is at the crate boundary on purpose: the defect survived a+//! same-file fix because nothing exercised the public entry point.++use std::panic;+use xerj_api::es_compat::resolve_date_math_index;++/// Resolve `name`, converting a panic into a reportable string instead of+/// letting it abort the test binary under `panic = "abort"`-adjacent settings.+fn resolve(name: &str) -> Result<String, String> {+ let prev = panic::take_hook();+ panic::set_hook(Box::new(|_| {}));+ let out = panic::catch_unwind(|| resolve_date_math_index(name));+ panic::set_hook(prev);+ out.map_err(|e| {+ e.downcast_ref::<String>()+ .cloned()+ .or_else(|| e.downcast_ref::<&str>().map(|s| s.to_string()))+ .unwrap_or_else(|| "<non-string panic>".to_string())+ })+}++/// Before the fix every one of these aborts the process.+#[test]+fn an_anchored_index_name_never_panics() {+ let mut bad = Vec::new();+ for name in [+ // Offset that does not fit a TimeDelta — `Duration::days` panics.+ "<logs-{2026-01-01||+9999999999999d}>",+ "<logs-{2026-01-01||-9999999999999d}>",+ "<logs-{2026-01-01||+99999999999h}>",+ "<logs-{2026-01-01||+999999999999999m}>",+ "<logs-{2026-01-01||+99999999999999999s}>",+ "<logs-{2026-01-01||+9999999999999w}>",+ // `n * 30` / `n * 365` overflow i64 before `Duration` is even asked.+ "<logs-{2026-01-01||+9223372036854775807M}>",+ "<logs-{2026-01-01||+9223372036854775807y}>",+ // Representable offset, but the resulting DateTime is out of range.+ "<logs-{2026-01-01||+100000000000d}>",+ // Multi-byte character where the unit is read — `r2[..1]` splits it.+ "<logs-{2026-01-01||+1\u{4e2d}}>",+ "<logs-{2026-01-01||+1\u{1be}}>",+ "<logs-{2026-01-01||-1\u{1f600}}>",+ "<logs-{2026-01-01||/\u{4e2d}}>",+ "<logs-{2026-01-01||\u{4e2d}}>",+ // Same shapes with a timestamp anchor and a format suffix.+ "<logs-{2026-01-01T00:00:00||+9999999999999d{yyyy.MM.dd}}>",+ "<logs-{2026-01-01T00:00:00||/\u{4e2d}{yyyy}}>",+ // Chained tails, so the loop runs more than once.+ "<logs-{2026-01-01||+1d+9999999999999d/d}>",+ "<logs-{2026-01-01||/d+1\u{4e2d}-1d}>",+ // Controls: the `now` branch the first pass did fix.+ "<logs-{now+9999999999999d}>",+ "<logs-{now+1\u{4e2d}}>",+ ] {+ if let Err(msg) = resolve(name) {+ bad.push(format!("{name} -> PANIC: {msg}"));+ }+ }+ assert!(+ bad.is_empty(),+ "index names that panic resolve_date_math_index. `panic = \"abort\"` is \+ set in the release profile, so each of these is process death from a \+ request URI:\n {}",+ bad.join("\n ")+ );+}++/// The fix must not have made anchored date math stop working: an+/// unrepresentable offset degrades to "no offset", everything representable+/// still resolves.+#[test]+fn anchored_date_math_still_resolves() {+ assert_eq!(+ resolve("<logs-{2026-01-01||+1d}>").unwrap(),+ "logs-2026.01.02"+ );+ assert_eq!(+ resolve("<logs-{2026-01-01||-1d}>").unwrap(),+ "logs-2025.12.31"+ );+ assert_eq!(+ resolve("<logs-{2026-01-15||/M{yyyy.MM}}>").unwrap(),+ "logs-2026.01"+ );+ assert_eq!(+ resolve("<logs-{2026-01-01||+2M}>").unwrap(),+ // 2 * 30 days, which is what this resolver's calendar-free `M` means.+ "logs-2026.03.02"+ );+ // Out-of-range offset degrades to the anchor rather than to a crash.+ assert_eq!(+ resolve("<logs-{2026-01-01||+9999999999999d}>").unwrap(),+ "logs-2026.01.01"+ );+ // An unreadable multi-byte unit is not a recognised unit, so likewise.+ assert_eq!(+ resolve("<logs-{2026-01-01||+1\u{4e2d}}>").unwrap(),+ "logs-2026.01.01"+ );+ // Names without date math pass through untouched.+ assert_eq!(resolve("my-index").unwrap(), "my-index");+}diff --git a/engine/crates/xerj-engine/src/aggs.rs b/engine/crates/xerj-engine/src/aggs.rsindex 4ff756655..0ab9f24a4 100644--- a/engine/crates/xerj-engine/src/aggs.rs+++ b/engine/crates/xerj-engine/src/aggs.rs@@ -1499,7 +1499,15 @@ fn tokenize_script(s: &str, params: &HashMap<String, f64>) -> Option<Vec<Tok>> {let mut i = 0;while i < bytes.len() {let c = bytes[i] as char;- if c.is_whitespace() {+ // ASCII-only, for the same reason `painless::tokenize` is (#207): this+ // scanner walks BYTES (`bytes[i] as char` reinterprets each byte as+ // Latin-1) but slices `&s[..]` as a `&str`. `is_whitespace()` is true+ // for Latin-1 NEL (0x85) and NBSP (0xA0), and in valid UTF-8 both of+ // those bytes are only ever *continuation* bytes — accepting them steps+ // `i` into the middle of a character, and the next slice panics on a+ // char boundary. No real script loses anything: those are the only two+ // non-ASCII bytes the old test accepted.+ if c.is_ascii_whitespace() {i += 1;continue;}@@ -1538,7 +1546,19 @@ fn tokenize_script(s: &str, params: &HashMap<String, f64>) -> Option<Vec<Tok>> {continue;}// Two-character operators first.- if i + 1 < bytes.len() {+ //+ // `&s[i..i + 2]` is a `&str` slice on a byte index, so it panics unless+ // BOTH ends sit on a character boundary. Every arm above this one is+ // ASCII-only, so a multi-byte character's lead byte falls straight+ // through to here and `i + 2` lands inside it —+ // `{"bs":{"bucket_script":{"buckets_path":{"x":"s"},"script":"1 中 2"}}}`+ // in an ordinary `_search` body was `end byte index 2 is not a char+ // boundary`, and `panic = "abort"` makes that the process (#207).+ // Every two-character operator is pure ASCII, so requiring both bytes+ // to be ASCII keeps the slice on a boundary by construction; a+ // non-ASCII byte falls to the `_ => return None` arm below, which is+ // the right answer for an operator this subset cannot evaluate.+ if i + 1 < bytes.len() && bytes[i].is_ascii() && bytes[i + 1].is_ascii() {let two = &s[i..i + 2];match two {">=" | "<=" | "==" | "!=" | "&&" | "||" => {@@ -12247,7 +12267,11 @@ fn lex_script(src: &str) -> Result<Vec<PlTok>, String> {let mut out = Vec::new();while i < bytes.len() {let c = bytes[i] as char;- if c.is_whitespace() {+ // ASCII-only — see the identical note in `tokenize_script` above and in+ // `painless::tokenize`. Latin-1 NEL/NBSP read as whitespace here but are+ // continuation bytes in UTF-8, so accepting them walks `i` into the+ // middle of a character (#207).+ if c.is_ascii_whitespace() {i += 1;continue;}@@ -12304,8 +12328,17 @@ fn lex_script(src: &str) -> Result<Vec<PlTok>, String> {out.push(PlTok::Ident(src[start..i].to_string()));continue;}- // Two-char operators- if i + 1 < bytes.len() {+ // Two-char operators.+ //+ // Both bytes must be ASCII before `&src[i..i + 2]` can be taken: this+ // is a `&str` slice on a byte index and every arm above is ASCII-only,+ // so a multi-byte character's lead byte reaches here and `i + 2` lands+ // inside it. `{"m":{"scripted_metric":{"map_script":"中"}}}` in an+ // ordinary `_search` body was `end byte index 2 is not a char+ // boundary` — the process, not a 400, under `panic = "abort"` (#207).+ // The operators are all ASCII, so nothing is lost; a non-ASCII byte+ // now reaches the `unexpected char` error below.+ if i + 1 < bytes.len() && bytes[i].is_ascii() && bytes[i + 1].is_ascii() {let two = &src[i..i + 2];let op: Option<&'static str> = match two {">=" => Some(">="),diff --git a/engine/crates/xerj-engine/tests/agg_script_tokenisers_are_total.rs b/engine/crates/xerj-engine/tests/agg_script_tokenisers_are_total.rsnew file mode 100644index 000000000..06368962a--- /dev/null+++ b/engine/crates/xerj-engine/tests/agg_script_tokenisers_are_total.rs@@ -0,0 +1,166 @@+//! The aggregation script tokenisers must be total on any `_search` body.+//!+//! Issue #207. `painless::tokenize` was not the only Painless-subset scanner+//! walking bytes and slicing `&str`. `aggs.rs` holds two more, reached from an+//! ordinary aggregation body and from nowhere else:+//!+//! * `aggs::lex_script` — `scripted_metric`+//! * `aggs::tokenize_script` — `bucket_script` / `bucket_selector`+//!+//! Both took `&src[i..i + 2]` to test for a two-character operator. Every arm+//! above that point is ASCII-only, so the lead byte of a multi-byte character+//! fell straight through and `i + 2` landed inside it:+//! `end byte index 2 is not a char boundary`. Both also skipped whitespace with+//! `is_whitespace()` on `bytes[i] as char`, which is true for Latin-1 NEL and+//! NBSP — bytes that in valid UTF-8 are only ever *continuation* bytes, so the+//! scanner could step into the middle of a character on its own.+//!+//! The release profile sets `panic = "abort"`, so this was not a 400 on a bad+//! script: it was the process, from an unauthenticated `_search`.++use serde_json::json;+use std::panic;++fn run(label: String, aggs: serde_json::Value, docs: Vec<serde_json::Value>) -> Option<String> {+ let prev = panic::take_hook();+ panic::set_hook(Box::new(|_| {}));+ let out = panic::catch_unwind(move || xerj_engine::aggs::run_aggs(&aggs, &docs));+ panic::set_hook(prev);+ out.err().map(|e| {+ format!(+ "{label}: {}",+ e.downcast_ref::<String>()+ .cloned()+ .or_else(|| e.downcast_ref::<&str>().map(|s| s.to_string()))+ .unwrap_or_else(|| "<non-string panic>".into())+ )+ })+}++fn docs() -> Vec<serde_json::Value> {+ vec![+ json!({ "k": "a", "v": 1.0 }),+ json!({ "k": "a", "v": 2.0 }),+ json!({ "k": "b", "v": 3.0 }),+ ]+}++/// Before the fix each of these aborts the test process.+#[test]+fn a_non_ascii_aggregation_script_is_an_error_not_a_panic() {+ let mut bad = Vec::new();++ for src in [+ // Bare multi-byte character: the lead byte reaches the two-char slice.+ "\u{4e2d}",+ // Multi-byte between two operands, and after an operator.+ "1 \u{4e2d} 2",+ "params.x \u{4e2d}",+ "params.x > \u{4e2d}",+ // Four-byte character, so `i + 2` is two bytes short of the end.+ "params.x \u{1f600}",+ // NBSP: `is_whitespace()` on the 0xA0 continuation byte used to+ // advance `i` into the middle of the character.+ "params.x \u{a0}> 1",+ "\u{a0}\u{a0}\u{a0}",+ // NEL (U+0085), the other byte `is_whitespace()` accepted.+ "params.x \u{85}> 1",+ // Multi-byte at the very end, where the two-char probe runs off.+ "1 +\u{e9}",+ "\u{e9}",+ ] {+ for (label, aggs) in [+ (+ "scripted_metric",+ json!({ "m": { "scripted_metric": { "map_script": src } } }),+ ),+ (+ "bucket_script",+ json!({+ "s": { "sum": { "field": "v" } },+ "bs": { "bucket_script": { "buckets_path": { "x": "s" }, "script": src } },+ }),+ ),+ (+ "bucket_selector",+ json!({ "by": {+ "terms": { "field": "k" },+ "aggs": {+ "s": { "sum": { "field": "v" } },+ "sel": { "bucket_selector": {+ "buckets_path": { "x": "s" }, "script": src } },+ },+ }}),+ ),+ ] {+ if let Some(msg) = run(format!("{label} script={src:?}"), aggs, docs()) {+ bad.push(msg);+ }+ }+ }++ assert!(+ bad.is_empty(),+ "aggregation scripts that panic run_aggs. `panic = \"abort\"` is set in \+ the release profile, so each of these is process death from a _search \+ body:\n {}",+ bad.join("\n ")+ );+}++/// The narrowed scan must not have narrowed the language: ASCII scripts,+/// including the two-character operators the guard now gates, still evaluate.+#[test]+fn ascii_aggregation_scripts_still_evaluate() {+ // bucket_script over a sibling sum, using a two-character operator path+ // (`>=` is tokenised by the same guarded branch).+ let out = xerj_engine::aggs::run_aggs(+ &json!({+ "s": { "sum": { "field": "v" } },+ "bs": { "bucket_script": {+ "buckets_path": { "x": "s" },+ "script": "params.x >= 6 ? params.x * 2 : params.x",+ }},+ }),+ &docs(),+ );+ assert_eq!(+ out["bs"]["value"].as_f64(),+ Some(12.0),+ "bucket_script over sum(v)=6 with `>=` and a ternary: {out}"+ );++ // bucket_selector keeps the bucket whose sum passes the predicate and+ // drops the other one.+ let out = xerj_engine::aggs::run_aggs(+ &json!({ "by": {+ "terms": { "field": "k" },+ "aggs": {+ "s": { "sum": { "field": "v" } },+ "sel": { "bucket_selector": {+ "buckets_path": { "x": "s" },+ "script": "params.x >= 3 && params.x != 99",+ }},+ },+ }}),+ &docs(),+ );+ let buckets = out["by"]["buckets"].as_array().expect("buckets");+ assert_eq!(buckets.len(), 2, "both buckets sum to >= 3: {out}");++ let out = xerj_engine::aggs::run_aggs(+ &json!({ "by": {+ "terms": { "field": "k" },+ "aggs": {+ "s": { "sum": { "field": "v" } },+ "sel": { "bucket_selector": {+ "buckets_path": { "x": "s" },+ "script": "params.x > 3",+ }},+ },+ }}),+ &docs(),+ );+ let buckets = out["by"]["buckets"].as_array().expect("buckets");+ assert_eq!(buckets.len(), 0, "neither bucket sums above 3: {out}");+}diff --git a/engine/fuzz/Cargo.toml b/engine/fuzz/Cargo.tomlindex e1be09fa0..a264bbdc0 100644--- a/engine/fuzz/Cargo.toml+++ b/engine/fuzz/Cargo.toml@@ -14,6 +14,10 @@ libfuzzer-sys = "0.4"serde_json = "1"xerj-query = { path = "../crates/xerj-query" }xerj-engine = { path = "../crates/xerj-engine" }+# The index-name date-math resolver that actually runs on the wire lives here,+# not in xerj-engine — see the note at the top of+# fuzz_targets/index_name_date_math.rs.+xerj-api = { path = "../crates/xerj-api" }# Keep the harnesses out of the engine workspace: they only build under a# nightly toolchain with sanitizers, so a `cargo build --workspace` on stable@@ -64,3 +68,10 @@ path = "fuzz_targets/index_name_date_math.rs"test = falsedoc = falsebench = false++[[bin]]+name = "agg_script"+path = "fuzz_targets/agg_script.rs"+test = false+doc = false+bench = falsediff --git a/engine/fuzz/README.md b/engine/fuzz/README.mdindex 364c78e39..899a7497c 100644--- a/engine/fuzz/README.md+++ b/engine/fuzz/README.md@@ -46,7 +46,15 @@ input libFuzzer minimised the `TimeDelta::hours out of bounds` panic down to.| `date_math` | `xerj_query::dates` | `"gte": "now-1d/d"` and `"format": "…"` are caller-supplied; both compile through hand-written tokenisers || `sql` | `xerj_engine::sql::parse_sql` | `/_sql` takes a statement from the wire; ES's CVE-2024-43709 was an OOM from a crafted SQL query || `painless` | `xerj_engine::painless::{check_script_limits, eval_painless}` | attacker-supplied *code* inside a query body, plus the depth/op/deadline budgets that are supposed to bound it |-| `index_name_date_math` | `xerj_engine::index::resolve_date_math` | `<logs-{now/d}>` arrives in a request URI — a second date-math implementation with its own brace scanner |+| `index_name_date_math` | `xerj_api::es_compat::resolve_date_math_index`, then `xerj_engine::index::resolve_date_math` | `<logs-{now/d}>` arrives in a request URI — a second date-math implementation with its own brace scanner |+| `agg_script` | `xerj_engine::aggs::run_aggs` with a fuzzed script in `scripted_metric`, `bucket_script`, `bucket_selector` | two *more* Painless-subset tokenisers, distinct from `xerj_engine::painless` and reached only through an aggregation body |++The `es_compat` entry point above is named first for a reason. The first version+of `index_name_date_math` fuzzed `xerj_engine::index::resolve_date_math` alone —+which has no production callers — so it passed while the resolver that runs on+the wire still aborted on `<logs-{2026-01-01||+9999999999999d}>`. **Fuzz the+function the request reaches**, and check that it is that function: `grep` for+callers before you write the harness.This list is the coverage. It is not every parser in the engine, and thesecurity docs must name these rather than claim the parser surface generally —diff --git a/engine/fuzz/fuzz_targets/agg_script.rs b/engine/fuzz/fuzz_targets/agg_script.rsnew file mode 100644index 000000000..325f3a19e--- /dev/null+++ b/engine/fuzz/fuzz_targets/agg_script.rs@@ -0,0 +1,74 @@+//! Fuzz the two script tokenisers that live in the aggregation engine.+//!+//! `xerj_engine::painless::tokenize` is not the only Painless-subset scanner in+//! the tree. `aggs.rs` holds two more with the same byte-walk / `&str`-slice+//! shape, reached from an ordinary `_search` body and nothing else:+//!+//! * `aggs::lex_script` — `scripted_metric` (`init/map/combine/reduce`)+//! * `aggs::tokenize_script` — `bucket_script` / `bucket_selector`+//!+//! Neither was covered when the `painless` target found its char-boundary+//! crash, and both had the identical defect: `&src[i..i + 2]` for a two-char+//! operator, taken at a byte index that can sit inside a multi-byte character+//! because every arm above it is ASCII-only. `{"map_script":"中"}` was+//! `end byte index 2 is not a char boundary` — the process, under+//! `panic = "abort"` (#207).+//!+//! The harness supplies only the *script source* and wraps it in fixed agg+//! bodies over fixed documents. That is deliberate: fuzzing whole aggregation+//! JSON would spend its budget on `date_histogram` intervals and bucket counts+//! and turn a security gate into an out-of-memory lottery, while the parsers+//! this file exists to protect would barely be reached.+#![no_main]++use libfuzzer_sys::fuzz_target;+use serde_json::json;++fuzz_target!(|data: &[u8]| {+ let Ok(src) = std::str::from_utf8(data) else {+ return;+ };+ // Both evaluators reject longer sources on a length check before they+ // tokenize; past that bound the fuzzer is only exercising `len()`.+ if src.len() > 4096 {+ return;+ }++ let docs = vec![+ json!({ "k": "a", "v": 1.0 }),+ json!({ "k": "a", "v": 2.0 }),+ json!({ "k": "b", "v": 3.0 }),+ ];++ // scripted_metric -> lex_script+ let _ = xerj_engine::aggs::run_aggs(+ &json!({ "m": { "scripted_metric": {+ "init_script": src,+ "map_script": src,+ "combine_script": src,+ "reduce_script": src,+ }}}),+ &docs,+ );++ // bucket_script (sibling pipeline agg) -> tokenize_script+ let _ = xerj_engine::aggs::run_aggs(+ &json!({+ "s": { "sum": { "field": "v" } },+ "bs": { "bucket_script": { "buckets_path": { "x": "s" }, "script": src } },+ }),+ &docs,+ );++ // bucket_selector (parent pipeline agg) -> tokenize_script+ let _ = xerj_engine::aggs::run_aggs(+ &json!({ "by": {+ "terms": { "field": "k" },+ "aggs": {+ "s": { "sum": { "field": "v" } },+ "sel": { "bucket_selector": { "buckets_path": { "x": "s" }, "script": src } },+ },+ }}),+ &docs,+ );+});diff --git a/engine/fuzz/fuzz_targets/index_name_date_math.rs b/engine/fuzz/fuzz_targets/index_name_date_math.rsindex 178729d89..ad008457a 100644--- a/engine/fuzz/fuzz_targets/index_name_date_math.rs+++ b/engine/fuzz/fuzz_targets/index_name_date_math.rs@@ -5,6 +5,18 @@//! independent implementation of date math with its own hand-written brace//! scanner on top. Both halves are attacker-controlled: the arithmetic//! (`now+9999999999999d`) and the brace/format structure (`<a{b{c}}>`).+//!+//! **The resolver that runs on the wire is `xerj_api::es_compat`'s.** The first+//! version of this harness fuzzed `xerj_engine::index::resolve_date_math`+//! instead, which is a re-implementation with no production callers — grep+//! `crates/` and the only hits are `lib.rs`'s re-export, its own `#[cfg(test)]`+//! module and this file. So the harness passed while the resolver that actually+//! runs on the index path parameter (`es_compat.rs` `create_index`) and on the+//! `_search` index list still aborted the process on+//! `<logs-{2026-01-01||+9999999999999d}>`. `resolve_date_math_index` is fuzzed+//! first here for that reason; the engine copy is kept because it is exported+//! from `xerj-engine`'s public API and costs nothing to drive with the same+//! input (#207).#![no_main]use libfuzzer_sys::fuzz_target;@@ -16,5 +28,8 @@ fuzz_target!(|data: &[u8]| {if name.len() > 1024 {return;}+ // The one that runs on the wire.+ let _ = xerj_api::es_compat::resolve_date_math_index(name);+ // The public engine re-implementation, same input.let _ = xerj_engine::index::resolve_date_math(name);});diff --git a/engine/fuzz/seeds/agg_script/bucket_script_arith b/engine/fuzz/seeds/agg_script/bucket_script_arithnew file mode 100644index 000000000..4eb9d4fb0--- /dev/null+++ b/engine/fuzz/seeds/agg_script/bucket_script_arith@@ -0,0 +1 @@+params.x / params.y * 100\ No newline at end of filediff --git a/engine/fuzz/seeds/agg_script/bucket_selector_compare b/engine/fuzz/seeds/agg_script/bucket_selector_comparenew file mode 100644index 000000000..430f54371--- /dev/null+++ b/engine/fuzz/seeds/agg_script/bucket_selector_compare@@ -0,0 +1 @@+params.x > 100\ No newline at end of filediff --git a/engine/fuzz/seeds/agg_script/comments b/engine/fuzz/seeds/agg_script/commentsnew file mode 100644index 000000000..f77bd824a--- /dev/null+++ b/engine/fuzz/seeds/agg_script/comments@@ -0,0 +1,2 @@+// comment+/* block */ return 1;\ No newline at end of filediff --git a/engine/fuzz/seeds/agg_script/regression-207-multibyte-between-operands b/engine/fuzz/seeds/agg_script/regression-207-multibyte-between-operandsnew file mode 100644index 000000000..79a8ce291--- /dev/null+++ b/engine/fuzz/seeds/agg_script/regression-207-multibyte-between-operands@@ -0,0 +1 @@+1 中 2\ No newline at end of filediff --git a/engine/fuzz/seeds/agg_script/regression-207-multibyte-lead-byte b/engine/fuzz/seeds/agg_script/regression-207-multibyte-lead-bytenew file mode 100644index 000000000..ec92f2542--- /dev/null+++ b/engine/fuzz/seeds/agg_script/regression-207-multibyte-lead-byte@@ -0,0 +1 @@+中\ No newline at end of filediff --git a/engine/fuzz/seeds/agg_script/regression-207-nbsp-continuation-byte b/engine/fuzz/seeds/agg_script/regression-207-nbsp-continuation-bytenew file mode 100644index 000000000..c1e331081--- /dev/null+++ b/engine/fuzz/seeds/agg_script/regression-207-nbsp-continuation-byte@@ -0,0 +1 @@+params.x > 1\ No newline at end of filediff --git a/engine/fuzz/seeds/agg_script/scripted_metric_map b/engine/fuzz/seeds/agg_script/scripted_metric_mapnew file mode 100644index 000000000..e141a7525--- /dev/null+++ b/engine/fuzz/seeds/agg_script/scripted_metric_map@@ -0,0 +1 @@+double d = doc["v"].value; if (d > 1.0) { return d; } return 0;\ No newline at end of filediff --git a/engine/fuzz/seeds/agg_script/scripted_metric_reduce b/engine/fuzz/seeds/agg_script/scripted_metric_reducenew file mode 100644index 000000000..dd8c8cec1--- /dev/null+++ b/engine/fuzz/seeds/agg_script/scripted_metric_reduce@@ -0,0 +1 @@+state.sum = 0; for (s : states) { state.sum += s; } return state.sum;\ No newline at end of filediff --git a/engine/fuzz/seeds/agg_script/two_char_operators b/engine/fuzz/seeds/agg_script/two_char_operatorsnew file mode 100644index 000000000..d81a1e3bf--- /dev/null+++ b/engine/fuzz/seeds/agg_script/two_char_operators@@ -0,0 +1 @@+params.x >= 1 && params.x != 2 || params.x <= 3\ No newline at end of filediff --git a/engine/fuzz/seeds/index_name_date_math/anchored_rounding b/engine/fuzz/seeds/index_name_date_math/anchored_roundingnew file mode 100644index 000000000..f1449eb5e--- /dev/null+++ b/engine/fuzz/seeds/index_name_date_math/anchored_rounding@@ -0,0 +1 @@+<logs-{2026-01-01T00:00:00||-1M/M{yyyy.MM}}>\ No newline at end of filediff --git a/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-datetime-overflow b/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-datetime-overflownew file mode 100644index 000000000..39fac05b7--- /dev/null+++ b/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-datetime-overflow@@ -0,0 +1 @@+<logs-{2026-01-01||+100000000000d/中}>\ No newline at end of filediff --git a/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-multibyte-unit b/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-multibyte-unitnew file mode 100644index 000000000..004dbea91--- /dev/null+++ b/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-multibyte-unit@@ -0,0 +1 @@+<logs-{2026-01-01||+1中}>\ No newline at end of filediff --git a/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-offset-overflow b/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-offset-overflownew file mode 100644index 000000000..0c779d6eb--- /dev/null+++ b/engine/fuzz/seeds/index_name_date_math/regression-207-anchored-offset-overflow@@ -0,0 +1 @@+<logs-{2026-01-01||+9999999999999d}>\ No newline at end of filediff --git a/landing/docs/config.html b/landing/docs/config.htmlindex ac05248d9..3450b5202 100644--- a/landing/docs/config.html+++ b/landing/docs/config.html@@ -162,9 +162,9 @@ flush_size_mb = 256 # flush memtable at 256 MiBflush_interval_secs = 30</pre><h2 id="merge-section">[merge]</h2>- <p>Segment compaction. <code class="inline">size_tiered</code> is the right default: it merges same-size segments, which is cheap and write-optimal. Switch to <code class="inline">log_structured</code> if your reads are doing a lot of full-range scans.</p>+ <p>Segment compaction. <code class="inline">size_tiered</code> is the only implemented policy: it merges same-size segments, which is cheap and write-optimal. <code class="inline">log_structured</code> is a name in the enum with no merge policy behind it, so the server <strong>refuses to start</strong> on it rather than silently running size-tiered merging for an operator who picked a levelled policy for its read amplification. If full-range scans are what hurts, that is a real gap in this build — setting the key will not change it.</p><pre class="code">[merge]-strategy = "size_tiered"+strategy = "size_tiered" # the only one implemented — log_structured is refused at startupmin_segments = 10 # accepted, not wiredmax_segment_mb = 8192 # 8 GiB capio_rate_mb_per_sec = 100 # accepted, not wired — merges are not throttleddiff --git a/landing/docs/storage.html b/landing/docs/storage.htmlindex 091ee2f54..9b68f654d 100644--- a/landing/docs/storage.html+++ b/landing/docs/storage.html@@ -160,7 +160,8 @@<h2>Merges</h2><p><code class="inline">[merge] max_segment_mb</code> excludes anything that big from further merging (default 8192). <code class="inline">tier_floor_mb</code> (default 4) sets the tier boundaries, <code class="inline">min_merge_count</code> (default 4) is the per-tier trigger, and <code class="inline">max_merge_count</code> (default 16) caps how many segments one batch merges — and so caps peak merge RAM. Those four are what the merge path reads.</p>- <p><strong>Three keys in this section are accepted but not wired</strong> and setting them logs a warning at startup: <code class="inline">strategy</code> (only <code class="inline">size_tiered</code> exists — <code class="inline">log_structured</code> is refused rather than silently substituted), <code class="inline">min_segments</code> (superseded by <code class="inline">min_merge_count</code>), and <code class="inline">io_rate_mb_per_sec</code> — <strong>merge I/O is not throttled in this build</strong>. <code class="inline">max_concurrent</code> is likewise unread; merge parallelism comes from the <code class="inline">XERJ_MERGE_PARALLELISM</code> environment variable (default 1).</p>+ <p><code class="inline">strategy</code> has exactly one implemented value: <code class="inline">size_tiered</code>. <code class="inline">log_structured</code> is <strong>refused at startup</strong> rather than silently substituted, so a config that asks for it fails loudly instead of running the other policy.</p>+ <p><strong>Three further keys in this section are accepted but not wired</strong>, and setting any of them away from its default logs a warning at startup: <code class="inline">min_segments</code> (superseded by <code class="inline">min_merge_count</code>), <code class="inline">io_rate_mb_per_sec</code> — <strong>merge I/O is not throttled in this build</strong>; the rate limiter that honours it sits in a storage-crate merge executor the engine never constructs — and <code class="inline">max_concurrent</code>, since merge parallelism comes from the <code class="inline">XERJ_MERGE_PARALLELISM</code> environment variable (default 1).</p><h2>Cluster metadata</h2><p>In single-node mode there is no <code class="inline">cluster/</code> directory — everything the engine needs is right next to the index data. When the server is started with a cluster config, a sibling <code class="inline">cluster/</code> directory holds the embedded Raft log and snapshots. Index <em>data</em> is never in Raft — only the <strong>metadata</strong> (index schemas, shard assignments, node roster). See <a href="/docs/clustering.html">Clustering</a>.</p>diff --git a/user-feedback/09-security/cves-and-vulnerabilities.md b/user-feedback/09-security/cves-and-vulnerabilities.mdindex 69e6ff5d2..067b7b155 100644--- a/user-feedback/09-security/cves-and-vulnerabilities.md+++ b/user-feedback/09-security/cves-and-vulnerabilities.md@@ -67,16 +67,19 @@- `cargo-audit` runs on every push and pull request (`.github/workflows/ci.yml`,job `security-audit`). A RUSTSEC **vulnerability** advisory against anything in`engine/Cargo.lock` fails the build; informational advisories (unmaintained,- unsound, yanked) are printed in the job log and do not+ unsound, yanked) are printed in the job log and do not fail it, because they+ land in transitive crates we cannot upgrade unilaterally and a gate that is+ red for reasons nobody can act on gets ignored, then switched off.- `cargo-fuzz` harnesses live in `engine/fuzz/` and run on every push and pullrequest (job `fuzz`, script `.github/scripts/fuzz-smoke.sh`). They cover theparsers an unauthenticated request can reach: the Elasticsearch query DSL, theLucene `query_string` / `simple_query_string` grammar, date math and date-format- patterns, index-name date math (`<logs-{now/d}>`), SQL, and Painless scripts.- Each target replays a checked-in seed corpus and then fuzzes for a bounded- budget. **That list is the coverage** —- it is not every parser in the engine, and this line should be edited, not- broadened, when targets are added or removed+ patterns, index-name date math (`<logs-{now/d}>`), SQL, `xerj_engine::painless`+ scripts, and the two separate script tokenisers in the aggregation engine+ (`scripted_metric`, `bucket_script` / `bucket_selector`). Each target replays a+ checked-in seed corpus and then fuzzes for a bounded budget. **That list is the+ coverage** — it is not every parser in the engine, and this line should be+ edited, not broadened, when targets are added or removed.## Sources- Elastic Security Advisory ESA-2021-31