AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge remote-tracking branch 'origin/main' into unblock-261-work
xerj-org96007a83e347
diff --cc CHANGELOG.mdindex 760fd5877,ef625a381..efb2b3637--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -164,89 -29,55 +164,138 @@@ and this project adheres to [Semantic V`pdf_extraction_reuse` block so the behaviour is observable rather thaninferred from timing. Original work by Leonid Bugaev (@buger).+ ### Changed — autoindex refuses reruns that would strand documents++ - **Deleting an indexed file and rerunning `xerj autoindex` now fails instead+ of exiting 0.** Nothing in the pipeline removes the documents, aliases,+ graph edges or catalog entries that the deleted file published, so a rerun+ that ignored the deletion left them live and searchable with no source file+ behind them. The rerun is now refused before any remote call other than the+ endpoint-readiness ping: no mapping, delete-by-query, bulk, refresh, graph+ or catalog write is attempted, and the journal is not appended to. The error+ names the removed files and their content keys — the first ten, then an+ `… and N more` tail, since a whole unmounted subtree can vanish at once —+ and gives three recovery routes: restore the files and rerun, rebuild in+ place by deleting the named indices and the state directory, or rebuild+ under a new state directory, prefix and brain. `--fresh` is refused for the+ same case, because it does not delete those documents either. `--json` emits+ the same facts — every removed entry, uncapped — as+ `xerj.autoindex.unsupported_sync_delta.v1` on stdout with exit 1. Deleting a+ file that was only ever *skipped* is not refused: a junk file publishes no+ documents, and the stale junk-catalog sweep removes its one catalog row, so+ nothing is stranded.+ - **Files added after a plan was frozen are now called out on stderr.** They+ are still not indexed by a rerun that resumes an existing plan — the plan is+ a crash-resume boundary, not a folder-sync generation — but the run now+ lists them, records them as skipped (exit 3, completed-with-junk) and points+ at `--fresh`, which rebuilds the plan in place and picks them up. Adding or+ changing files and rerunning keeps working; only removals are refused.+ - **`xerj brain` no longer turns an absent or zero node-count probe into an+ automatic reset.** Journal/server disagreement now fails with the journal,+ URL, prefix, and brain identity plus recovery instructions, including the+ explicit `--fresh` rerun for a genuinely wiped data directory. Probe+ transport and malformed-response failures remain errors instead of being+ reported as an empty destination. A nodes index that was deleted out from+ under a surviving brain meta doc reads as absence and reaches that recovery+ text, rather than surfacing as a raw HTTP 404.+ - **`--fresh` still recovers a state directory whose journal cannot be+ parsed.** The rerun gate reads the durable plan before the run starts, and a+ plan that is malformed, or recorded for a different root/URL/prefix, is fatal+ to a resume — but not to `--fresh`, which deletes that journal unread. The+ preflight is now no more fatal than the open it precedes: under `--fresh` the+ unreadable plan is reported on stderr and rebuilt from the current folder.+ Without `--fresh` the refusal is unchanged. Because the removal gate has no+ comparison basis in that case, the warning says so: documents already+ published for files that are now gone cannot be identified from an unreadable+ plan and are not deleted.+ - **Refusal and skip listings are capped at ten entries plus an "and N more"+ tail.** Unmounting a bind mount under an indexed root vanishes every content+ group at once, so the uncapped listing was one rendered entry per journalled+ file. `--json` still carries every entry.++### Fixed++- **A `query_string` containing one non-ASCII character aborted the process.**+ `{"query_string":{"query":"_\u0660"}}` — or the same text in `?q=` — took the+ server down. The Lucene tokeniser walks bytes but slices `&q[start..i]` as a+ `&str`, and it broke on `is_whitespace()`, which is true for Latin-1 NBSP+ (0xA0). In valid UTF-8 that byte is only ever a *continuation* byte, so the+ scan stopped between the two bytes of U+0660 and the slice panicked on a+ character boundary. Every delimiter test in that scanner is now `is_ascii_*`,+ which makes "the scan can only stop on a character boundary" true by+ construction. Nothing is lost: the only non-ASCII bytes the old test accepted+ were 0x85 and 0xA0, neither of which is a standalone character in UTF-8.++- **An index name of `<{}{}>` aborted the process.** Index-name date math finds+ the first `{` and the last `}` inside the braces; when the `}` comes first,+ `date_part[2..0]` is a panic, not an empty slice. Reachable from any request+ that names an index. The closing brace is now required to come after the+ opening one.++- **A Painless script containing one non-ASCII character aborted the process.**+ The second thing the new fuzz harnesses found, inside a second. The tokeniser+ walks *bytes* (`bytes[i] as char` reinterprets each byte as Latin-1) but+ slices `&src[start..i]` as a `&str`. Every continuation byte of a multi-byte+ character reads as an alphabetic Latin-1 char, so an identifier scan could+ start or stop inside one and the slice panicked on a char boundary —+ `end byte index 24 is not a char boundary; it is inside 'ʋ'`. Scripts arrive+ in ordinary search and update bodies and the release profile sets+ `panic = "abort"`, so this was not a 400, it was the whole server. The+ identifier scan is now ASCII-only, which keeps both ends on a character+ boundary by construction; a non-ASCII byte reaches the tokeniser's+ `unexpected char` error instead. String literals still carry any Unicode.++- **A search body could abort the whole process: `{"range":{"ts":{"gte":"now+33333333333333H"}}}`.**+ The first thing the new `date_math` fuzz target found, 20 seconds into its+ first run. `chrono`'s `Duration::hours` and its siblings **panic** when the+ count does not fit a `TimeDelta`, and they run before the+ `checked_add_signed` that was supposed to make+ `xerj_query::dates::add_unit` total — so the careful checking around them+ bought nothing. A range bound goes straight from the request body into this+ code (`parser.rs:931`), the release profile sets `panic = "abort"`, and the+ result is an unauthenticated remote denial of service on a released binary.+ Fixed with the fallible `try_*` constructors. Grepping for the same pattern+ found **three** more copies in the index-name date-math resolvers —+ `xerj-engine/src/index.rs`, and *both* branches of+ `xerj-api/src/es_compat.rs`: the `now` form (`<logs-{now+9999999999999d}>`)+ and the anchored form (`<logs-{2026-01-01||+9999999999999d}>`), which are+ handled by two different functions sixty lines apart. All three are fixed,+ their `n * 30` / `n * 365` multiplications are `checked_mul`, the unit is read+ as one character rather than one byte, and the addition goes through+ `checked_add_signed`. `es_compat`'s resolver is the one that runs on the wire+ — the index path parameter of every create/index call and the `_search` index+ list — so `crates/xerj-api/tests/index_name_date_math_is_total.rs` now drives+ it at the crate boundary; the same-file fix that missed the anchored branch+ passed because nothing exercised the public entry point.++- **A `_search` aggregation script containing one non-ASCII character aborted+ the process.** `painless::tokenize` was not the only Painless-subset scanner+ in the tree: `aggs.rs` holds two more with the identical byte-walk /+ `&str`-slice shape — `lex_script` (`scripted_metric`) and `tokenize_script`+ (`bucket_script` / `bucket_selector`) — and both took `&src[i..i + 2]` to test+ for a two-character operator at a byte index that can sit inside a multi-byte+ character, because every arm above it is ASCII-only.+ `{"m":{"scripted_metric":{"map_script":"中"}}}` was `end byte index 2 is not a+ char boundary`. Both also skipped whitespace with `is_whitespace()` on+ `bytes[i] as char`, which is true for Latin-1 NEL and NBSP — bytes that in+ valid UTF-8 are only ever *continuation* bytes. The two-character probe now+ requires both bytes to be ASCII (every such operator is ASCII, so nothing is+ lost) and whitespace is `is_ascii_whitespace`. A new `agg_script` fuzz target+ covers all three entry points, which is how the next one gets found instead of+ reported.++- **An XML entity nobody declared no longer discards the text around it.**+ Fallout from the `quick-xml` upgrade above, which is a breaking change to how+ character data arrives: 0.38 stopped folding `&` into the surrounding+ `Event::Text` and began emitting each reference as its own+ `Event::GeneralRef`. Entity handling is preserved across that change (a+ reader that ignored the new event would have silently dropped every `&`, `<`,+ `>` and `é` while the words around them still arrived), and one case+ gets better: 0.36's `BytesText::unescape()` failed the whole text node on an+ entity no DTD declared for us, and `unwrap_or_default()` then discarded it —+ now only the reference itself is dropped. Character data is also reassembled+ from its fragments before being emitted, so an entity-bearing field stays one+ value instead of becoming an array of pieces.## [1.0.0-rc.14] - 2026-08-10