Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into unblock-261-work

xerj-org96007a83e347
diff --cc CHANGELOG.md
index 760fd5877,ef625a381..efb2b3637
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -164,89 -29,55 +164,138 @@@ and this project adheres to [Semantic V
`pdf_extraction_reuse` block so the behaviour is observable rather than
inferred from timing. Original work by Leonid Bugaev (@buger).
+ ### Changed — autoindex refuses reruns that would strand documents
+
+ - **Deleting an indexed file and rerunning `xerj autoindex` now fails instead
+ of exiting 0.** Nothing in the pipeline removes the documents, aliases,
+ graph edges or catalog entries that the deleted file published, so a rerun
+ that ignored the deletion left them live and searchable with no source file
+ behind them. The rerun is now refused before any remote call other than the
+ endpoint-readiness ping: no mapping, delete-by-query, bulk, refresh, graph
+ or catalog write is attempted, and the journal is not appended to. The error
+ names the removed files and their content keys — the first ten, then an
+ `… and N more` tail, since a whole unmounted subtree can vanish at once —
+ and gives three recovery routes: restore the files and rerun, rebuild in
+ place by deleting the named indices and the state directory, or rebuild
+ under a new state directory, prefix and brain. `--fresh` is refused for the
+ same case, because it does not delete those documents either. `--json` emits
+ the same facts — every removed entry, uncapped — as
+ `xerj.autoindex.unsupported_sync_delta.v1` on stdout with exit 1. Deleting a
+ file that was only ever *skipped* is not refused: a junk file publishes no
+ documents, and the stale junk-catalog sweep removes its one catalog row, so
+ nothing is stranded.
+ - **Files added after a plan was frozen are now called out on stderr.** They
+ are still not indexed by a rerun that resumes an existing plan — the plan is
+ a crash-resume boundary, not a folder-sync generation — but the run now
+ lists them, records them as skipped (exit 3, completed-with-junk) and points
+ at `--fresh`, which rebuilds the plan in place and picks them up. Adding or
+ changing files and rerunning keeps working; only removals are refused.
+ - **`xerj brain` no longer turns an absent or zero node-count probe into an
+ automatic reset.** Journal/server disagreement now fails with the journal,
+ URL, prefix, and brain identity plus recovery instructions, including the
+ explicit `--fresh` rerun for a genuinely wiped data directory. Probe
+ transport and malformed-response failures remain errors instead of being
+ reported as an empty destination. A nodes index that was deleted out from
+ under a surviving brain meta doc reads as absence and reaches that recovery
+ text, rather than surfacing as a raw HTTP 404.
+ - **`--fresh` still recovers a state directory whose journal cannot be
+ parsed.** The rerun gate reads the durable plan before the run starts, and a
+ plan that is malformed, or recorded for a different root/URL/prefix, is fatal
+ to a resume — but not to `--fresh`, which deletes that journal unread. The
+ preflight is now no more fatal than the open it precedes: under `--fresh` the
+ unreadable plan is reported on stderr and rebuilt from the current folder.
+ Without `--fresh` the refusal is unchanged. Because the removal gate has no
+ comparison basis in that case, the warning says so: documents already
+ published for files that are now gone cannot be identified from an unreadable
+ plan and are not deleted.
+ - **Refusal and skip listings are capped at ten entries plus an "and N more"
+ tail.** Unmounting a bind mount under an indexed root vanishes every content
+ group at once, so the uncapped listing was one rendered entry per journalled
+ file. `--json` still carries every entry.
+
+### Fixed
+
+- **A `query_string` containing one non-ASCII character aborted the process.**
+ `{"query_string":{"query":"_\u0660"}}` — or the same text in `?q=` — took the
+ server down. The Lucene tokeniser walks bytes but slices `&q[start..i]` as a
+ `&str`, and it broke on `is_whitespace()`, which is true for Latin-1 NBSP
+ (0xA0). In valid UTF-8 that byte is only ever a *continuation* byte, so the
+ scan stopped between the two bytes of U+0660 and the slice panicked on a
+ character boundary. Every delimiter test in that scanner is now `is_ascii_*`,
+ which makes "the scan can only stop on a character boundary" true by
+ construction. Nothing is lost: the only non-ASCII bytes the old test accepted
+ were 0x85 and 0xA0, neither of which is a standalone character in UTF-8.
+
+- **An index name of `<{}{}>` aborted the process.** Index-name date math finds
+ the first `{` and the last `}` inside the braces; when the `}` comes first,
+ `date_part[2..0]` is a panic, not an empty slice. Reachable from any request
+ that names an index. The closing brace is now required to come after the
+ opening one.
+
+- **A Painless script containing one non-ASCII character aborted the process.**
+ The second thing the new fuzz harnesses found, inside a second. The tokeniser
+ walks *bytes* (`bytes[i] as char` reinterprets each byte as Latin-1) but
+ slices `&src[start..i]` as a `&str`. Every continuation byte of a multi-byte
+ character reads as an alphabetic Latin-1 char, so an identifier scan could
+ start or stop inside one and the slice panicked on a char boundary —
+ `end byte index 24 is not a char boundary; it is inside 'ʋ'`. Scripts arrive
+ in ordinary search and update bodies and the release profile sets
+ `panic = "abort"`, so this was not a 400, it was the whole server. The
+ identifier scan is now ASCII-only, which keeps both ends on a character
+ boundary by construction; a non-ASCII byte reaches the tokeniser's
+ `unexpected char` error instead. String literals still carry any Unicode.
+
+- **A search body could abort the whole process: `{"range":{"ts":{"gte":"now+33333333333333H"}}}`.**
+ The first thing the new `date_math` fuzz target found, 20 seconds into its
+ first run. `chrono`'s `Duration::hours` and its siblings **panic** when the
+ count does not fit a `TimeDelta`, and they run before the
+ `checked_add_signed` that was supposed to make
+ `xerj_query::dates::add_unit` total — so the careful checking around them
+ bought nothing. A range bound goes straight from the request body into this
+ code (`parser.rs:931`), the release profile sets `panic = "abort"`, and the
+ result is an unauthenticated remote denial of service on a released binary.
+ Fixed with the fallible `try_*` constructors. Grepping for the same pattern
+ found **three** more copies in the index-name date-math resolvers —
+ `xerj-engine/src/index.rs`, and *both* branches of
+ `xerj-api/src/es_compat.rs`: the `now` form (`<logs-{now+9999999999999d}>`)
+ and the anchored form (`<logs-{2026-01-01||+9999999999999d}>`), which are
+ handled by two different functions sixty lines apart. All three are fixed,
+ their `n * 30` / `n * 365` multiplications are `checked_mul`, the unit is read
+ as one character rather than one byte, and the addition goes through
+ `checked_add_signed`. `es_compat`'s resolver is the one that runs on the wire
+ — the index path parameter of every create/index call and the `_search` index
+ list — so `crates/xerj-api/tests/index_name_date_math_is_total.rs` now drives
+ it at the crate boundary; the same-file fix that missed the anchored branch
+ passed because nothing exercised the public entry point.
+
+- **A `_search` aggregation script containing one non-ASCII character aborted
+ the process.** `painless::tokenize` was not the only Painless-subset scanner
+ in the tree: `aggs.rs` holds two more with the identical byte-walk /
+ `&str`-slice shape — `lex_script` (`scripted_metric`) and `tokenize_script`
+ (`bucket_script` / `bucket_selector`) — and both took `&src[i..i + 2]` to test
+ for a two-character operator at a byte index that can sit inside a multi-byte
+ character, because every arm above it is ASCII-only.
+ `{"m":{"scripted_metric":{"map_script":"中"}}}` was `end byte index 2 is not a
+ char boundary`. Both also skipped whitespace with `is_whitespace()` on
+ `bytes[i] as char`, which is true for Latin-1 NEL and NBSP — bytes that in
+ valid UTF-8 are only ever *continuation* bytes. The two-character probe now
+ requires both bytes to be ASCII (every such operator is ASCII, so nothing is
+ lost) and whitespace is `is_ascii_whitespace`. A new `agg_script` fuzz target
+ covers all three entry points, which is how the next one gets found instead of
+ reported.
+
+- **An XML entity nobody declared no longer discards the text around it.**
+ Fallout from the `quick-xml` upgrade above, which is a breaking change to how
+ character data arrives: 0.38 stopped folding `&amp;` into the surrounding
+ `Event::Text` and began emitting each reference as its own
+ `Event::GeneralRef`. Entity handling is preserved across that change (a
+ reader that ignored the new event would have silently dropped every `&`, `<`,
+ `>` and `&#233;` while the words around them still arrived), and one case
+ gets better: 0.36's `BytesText::unescape()` failed the whole text node on an
+ entity no DTD declared for us, and `unwrap_or_default()` then discarded it —
+ now only the reference itself is dropped. Character data is also reassembled
+ from its fragments before being emitted, so an entity-bearing field stays one
+ value instead of becoming an array of pieces.
## [1.0.0-rc.14] - 2026-08-10