Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into unblock-261-work

xerj-orga01f90cbc77a
diff --cc CHANGELOG.md
index fe0520c4e,e380144d1..80b35923e
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,210 -7,28 +7,230 @@@ and this project adheres to [Semantic V
## [Unreleased]
+### Security
+
+- **`cargo-audit` and `cargo-fuzz` now actually run in CI — they were only ever
+ documented** ([#207](https://github.com/xerj-org/xerj/issues/207)).
+ `user-feedback/09-security/cves-and-vulnerabilities.md` listed "`cargo-audit`
+ for dependency vulnerability scanning" and "fuzz testing (cargo-fuzz) on all
+ input parsing paths" as part of XERJ's answer to Elasticsearch's CVE history.
+ Neither existed: no audit job, no fuzz job, no fuzz target anywhere in the
+ tree. A reader comparing the two engines priced in a fuzzed parser surface
+ that was not there.
+ - New CI job `security-audit` runs `cargo audit` on every push and pull
+ request; a RUSTSEC vulnerability advisory fails the build.
+ - New CI job `fuzz` builds and runs seven libFuzzer harnesses
+ (`engine/fuzz/`) over checked-in seeds: the Elasticsearch query DSL, the
+ Lucene `query_string` grammar, date math and date-format patterns,
+ index-name date math, SQL, Painless scripts, and the aggregation engine's
+ two separate script tokenisers. `.github/scripts/fuzz-smoke.sh` is the same
+ entry point locally and in CI.
+ - `xerj-engine/tests/security_tooling_claims.rs` fails if a documented tool
+ stops running, if a fuzz target ships without a seed corpus, or if the
+ "all input parsing paths" overclaim reappears in any prose file.
+ - The claim itself is now specific about what is and is not covered.
+ - **Known gap, stated because the point of this entry is not to overclaim
+ again:** seven harnesses are seven parsers, not the engine's whole input
+ surface, and one parser outside them is already known to abort.
+ `aggs::parse_time_zone_offset` slices a `time_zone` string at byte 2 after
+ a *byte*-length check, so a `date_histogram` with `"time_zone": "+中a"` in
+ an unauthenticated `_search` body dies on a character boundary. It is
+ pre-existing, is not touched by this change, and is tracked in
+ [#272](https://github.com/xerj-org/xerj/issues/272) with a fix and a
+ proposed harness for the aggregation date parsers.
+
+- **Fixed two high-severity advisories in the document-ingest XML parser.**
+ Turning the audit gate on found `quick-xml` 0.36.2 in `xerj-autoindex`, which
+ parses `.docx`, `.xlsx` and `.xml` files supplied by whoever owns the folder
+ being indexed: RUSTSEC-2026-0195 (unbounded namespace-declaration allocation →
+ memory-exhaustion DoS, CVSS 7.5) and RUSTSEC-2026-0194 (quadratic run time on
+ duplicate attribute names, CVSS 7.5). Upgraded to 0.41, which also
+ de-duplicates the crate — 0.41 was already in the tree via `pdf_oxide`.
+ `crossbeam-epoch` moved 0.9.18 → 0.9.20 for RUSTSEC-2026-0204.
+
### Changed
+- **`merge.strategy = "log_structured"` is now refused at startup instead of
+ silently running the size-tiered policy**
+ ([#207](https://github.com/xerj-org/xerj/issues/207)). Nothing in the tree
+ ever read `merge.strategy`; `run_merge_once` always builds a
+ `SizeTieredMergePolicy`. An operator who chose a levelled policy for its read
+ amplification got the other one, quietly. This follows the `storage.backend`
+ and `vector.default_quantization` guards already in `Config::validate`.
+
+- **The three dormant `[merge]` settings now say so, in the code, in the shipped
+ TOML, on the docs site, and in the log at startup**
+ ([#207](https://github.com/xerj-org/xerj/issues/207)). The issue reported
+ merge I/O rate limiting as "claimed but dormant" and it is: the `RateLimiter`
+ that honours `io_rate_mb_per_sec` is wired only into `xerj-storage`'s legacy
+ `MergeExecutor`, which the engine never constructs. `min_segments` and
+ `max_concurrent` are unread too — the real per-tier trigger is
+ `min_merge_count`, and merge parallelism comes from `XERJ_MERGE_PARALLELISM`.
+ `MergeConfig::dormant_overrides` names any of the three the operator has moved
+ off its default and `xerj-server` logs each at WARN, so the operator throttling
+ merges to protect query latency finds out instead of guessing. They stay
+ accepted rather than becoming hard errors because the wrong value costs
+ latency, not data, and `io_rate_mb_per_sec = 100` has shipped in
+ `xerj.default.toml` since v0.1. Wiring a real throttle into `run_merge_once`
+ is the fix that makes this list shorter.
+
+- **`engine/xerj.default.toml` disagreed with the defaults it claims to
+ document, in four places.** `max_segment_mb` said 5120 against a real 8192,
+ `wal_max_size_mb` 512 against 1024, `flush_size_mb` 256 against 512, and
+ `default_quantization` `"scalar8"` against `"none"` — so `cp xerj.default.toml
+ xerj.toml`, the documented first step, silently changed four engine behaviours
+ including turning on 8-bit vector quantization. The file is corrected and
+ `shipped_default_config_documents_the_real_defaults` now diffs every leaf key
+ against `Config::default()`, so it cannot drift again.
+
+- **The docs site had drifted the same way, and now has the same guard.**
+ `landing/docs/config.html` shipped `wal_max_size_mb = 512`,
+ `flush_size_mb = 256` and `default_quantization = "scalar8"` in its
+ copy-pasteable `[storage]` and `[vector]` blocks while its own DEFAULT
+ table — thirty lines up, on the same page — said 1024, 512 and `"none"`;
+ `landing/docs/storage.html` said the WAL rolls at 512 MiB;
+ `landing/docs/vectors.html` called `scalar8` "the default" and listed a
+ `scalar4` mode that no config or mapping value can reach; and
+ `engine/README.md` put `flush_size_mb` at 256 MiB. All four are corrected,
+ along with the docs search-index blob duplicated across 44 landing pages,
+ which offered `scalar4` as a selectable quantization mode.
+ A second guard, `the_docs_site_config_page_agrees_with_the_real_defaults`,
+ now diffs *both* halves of that page — every DEFAULT cell in the reference
+ table and every assignment in the example blocks — against
+ `Config::default()`. An example that deliberately differs (the blocks whose
+ point is switching TLS or cluster mode on) has to carry `# not a default` on
+ the line, so the reader is told as well as the test.
+
+- **Four other claims from [#207](https://github.com/xerj-org/xerj/issues/207)
+ now match the source.**
+ - *Settings count.* `config.rs` said 38 in its header and 56 twenty lines
+ down while the count test asserted 60 — and the test asserted it by
+ re-adding a hardcoded sum (`5 + 2 + 3 + … - 1 == 38`), an identity that
+ could not fail whatever `Config` held. There was a second copy of the same
+ non-test in `config.rs` itself (`count_user_facing_settings`, `61 == 61`).
+ Both now serialise `Config::default()` and count leaf keys, section by
+ section. The measured figure is **104** (103 when this work was done; the
+ rc.14 `server.allow_insecure_network_bind` key makes it 104), and 38 / 50 /
+ 56 / 60 / 61 / "<50"
+ are corrected in `config.rs`, `xerj-common/src/lib.rs`, `engine/README.md`,
+ `xerj.default.toml` and the feedback responses, along with the stale
+ per-sub-config annotations (limits 3 → 13, storage 5 → 10, embedding 4 → 19,
+ merge 5 → 8, cluster 4 → 5, tls 3 → 4, auth 2 → 3). This closes item 10 of
+ `user-feedback/ROADMAP-TO-GA.md`. 104 versus 3,000+ is still the winning
+ story, told truthfully.
+ - *Wire-compatibility test framing.* `journey_es_migration` was described as
+ proving "the same curl commands" work. It calls the Rust engine API
+ directly and never makes an HTTP request. The response-shape assertions are
+ real and stay; the comment now says what the test does and points at the
+ ES-compat YAML conformance suite, which is what actually covers the wire.
+ - *OpenAPI spec.* `landing/openapi.json` declared `version: 1.0` while
+ specifying 7 routes out of the 200-plus the router registers. It is now
+ titled and versioned as a partial spec, and says so.
+
+ - **`autoindex` parses each PDF once per fresh run instead of twice.** Phase A
+ already paid a *complete* parse for every PDF — `extract::extract` routes
+ `Family::Pdf` straight to the isolated worker and drops the sampling limit,
+ so only delivery ever stopped early — and phase B then spawned the worker
+ again for the same bytes. Phase A now retains each validated worker response
+ in an anonymous temporary file under `--state-dir` and phase B replays it.
+ Reuse is an optional accelerator, never correctness-critical state: it is
+ bounded to 384 MiB of retained-plus-in-flight bytes and to an artifact-handle
+ cap derived from live `RLIMIT_NOFILE` and open-descriptor counts, both
+ re-probed at every admission; a 4 GiB-or-half-free filesystem floor is
+ reserved for the journal and phase-B staging first; and every artifact is
+ verified (physical length, content digest, JSON decode, worker-protocol
+ identity) before a single record is published. Anything that cannot be
+ measured conservatively declines and reparses — which is why the optimization
+ is **Linux-only** today: other platforms have no live descriptor evidence.
+ A restart has a frozen plan but no trusted handle, so it parses again; the
+ spool is deliberately not journal state. `--json` gains a
+ `pdf_extraction_reuse` block so the behaviour is observable rather than
+ inferred from timing. Original work by Leonid Bugaev (@buger).
+
+### Fixed
+
+- **A `query_string` containing one non-ASCII character aborted the process.**
+ `{"query_string":{"query":"_\u0660"}}` — or the same text in `?q=` — took the
+ server down. The Lucene tokeniser walks bytes but slices `&q[start..i]` as a
+ `&str`, and it broke on `is_whitespace()`, which is true for Latin-1 NBSP
+ (0xA0). In valid UTF-8 that byte is only ever a *continuation* byte, so the
+ scan stopped between the two bytes of U+0660 and the slice panicked on a
+ character boundary. Every delimiter test in that scanner is now `is_ascii_*`,
+ which makes "the scan can only stop on a character boundary" true by
+ construction. Nothing is lost: the only non-ASCII bytes the old test accepted
+ were 0x85 and 0xA0, neither of which is a standalone character in UTF-8.
+
+- **An index name of `<{}{}>` aborted the process.** Index-name date math finds
+ the first `{` and the last `}` inside the braces; when the `}` comes first,
+ `date_part[2..0]` is a panic, not an empty slice. Reachable from any request
+ that names an index. The closing brace is now required to come after the
+ opening one.
+
+- **A Painless script containing one non-ASCII character aborted the process.**
+ The second thing the new fuzz harnesses found, inside a second. The tokeniser
+ walks *bytes* (`bytes[i] as char` reinterprets each byte as Latin-1) but
+ slices `&src[start..i]` as a `&str`. Every continuation byte of a multi-byte
+ character reads as an alphabetic Latin-1 char, so an identifier scan could
+ start or stop inside one and the slice panicked on a char boundary —
+ `end byte index 24 is not a char boundary; it is inside 'ʋ'`. Scripts arrive
+ in ordinary search and update bodies and the release profile sets
+ `panic = "abort"`, so this was not a 400, it was the whole server. The
+ identifier scan is now ASCII-only, which keeps both ends on a character
+ boundary by construction; a non-ASCII byte reaches the tokeniser's
+ `unexpected char` error instead. String literals still carry any Unicode.
+
+- **A search body could abort the whole process: `{"range":{"ts":{"gte":"now+33333333333333H"}}}`.**
+ The first thing the new `date_math` fuzz target found, 20 seconds into its
+ first run. `chrono`'s `Duration::hours` and its siblings **panic** when the
+ count does not fit a `TimeDelta`, and they run before the
+ `checked_add_signed` that was supposed to make
+ `xerj_query::dates::add_unit` total — so the careful checking around them
+ bought nothing. A range bound goes straight from the request body into this
+ code (`parser.rs:931`), the release profile sets `panic = "abort"`, and the
+ result is an unauthenticated remote denial of service on a released binary.
+ Fixed with the fallible `try_*` constructors. Grepping for the same pattern
+ found **three** more copies in the index-name date-math resolvers —
+ `xerj-engine/src/index.rs`, and *both* branches of
+ `xerj-api/src/es_compat.rs`: the `now` form (`<logs-{now+9999999999999d}>`)
+ and the anchored form (`<logs-{2026-01-01||+9999999999999d}>`), which are
+ handled by two different functions sixty lines apart. All three are fixed,
+ their `n * 30` / `n * 365` multiplications are `checked_mul`, the unit is read
+ as one character rather than one byte, and the addition goes through
+ `checked_add_signed`. `es_compat`'s resolver is the one that runs on the wire
+ — the index path parameter of every create/index call and the `_search` index
+ list — so `crates/xerj-api/tests/index_name_date_math_is_total.rs` now drives
+ it at the crate boundary; the same-file fix that missed the anchored branch
+ passed because nothing exercised the public entry point.
+
+- **A `_search` aggregation script containing one non-ASCII character aborted
+ the process.** `painless::tokenize` was not the only Painless-subset scanner
+ in the tree: `aggs.rs` holds two more with the identical byte-walk /
+ `&str`-slice shape — `lex_script` (`scripted_metric`) and `tokenize_script`
+ (`bucket_script` / `bucket_selector`) — and both took `&src[i..i + 2]` to test
+ for a two-character operator at a byte index that can sit inside a multi-byte
+ character, because every arm above it is ASCII-only.
+ `{"m":{"scripted_metric":{"map_script":"中"}}}` was `end byte index 2 is not a
+ char boundary`. Both also skipped whitespace with `is_whitespace()` on
+ `bytes[i] as char`, which is true for Latin-1 NEL and NBSP — bytes that in
+ valid UTF-8 are only ever *continuation* bytes. The two-character probe now
+ requires both bytes to be ASCII (every such operator is ASCII, so nothing is
+ lost) and whitespace is `is_ascii_whitespace`. A new `agg_script` fuzz target
+ covers all three entry points, which is how the next one gets found instead of
+ reported.
+
+- **An XML entity nobody declared no longer discards the text around it.**
+ Fallout from the `quick-xml` upgrade above, which is a breaking change to how
+ character data arrives: 0.38 stopped folding `&amp;` into the surrounding
+ `Event::Text` and began emitting each reference as its own
+ `Event::GeneralRef`. Entity handling is preserved across that change (a
+ reader that ignored the new event would have silently dropped every `&`, `<`,
+ `>` and `&#233;` while the words around them still arrived), and one case
+ gets better: 0.36's `BytesText::unescape()` failed the whole text node on an
+ entity no DTD declared for us, and `unwrap_or_default()` then discarded it —
+ now only the reference itself is dropped. Character data is also reassembled
+ from its fragments before being emitted, so an entity-bearing field stays one
+ value instead of becoming an array of pieces.
+
## [1.0.0-rc.14] - 2026-08-10
### Changed