AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge remote-tracking branch 'origin/main' into HEAD
xerj-orgf8ed64db711e
diff --cc CHANGELOG.mdindex 10e2a5739,bae4e4abd..94932cb6a--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,25 -7,52 +7,71 @@@ and this project adheres to [Semantic V## [Unreleased]+### Measured++- **The #1062 wrong-and-confident detection gate was run through the real+ `_watcher` path and it FAILS** (issue+ [#1108](https://github.com/xerj-org/xerj/issues/1108)) — the unmeasured half+ of the rc.80 `_watcher` work. On the rc.94 release binary, a real+ `PUT /_watcher/watch` with `condition.xerj_decide` (one detection watch per+ label, one-vs-rest) evaluated a 120-email boundary tier and wrote 120 fire+ records into `.xerj_alert_fires`; the measurement reads only those fires.+ **Wrong-and-confident = 0.2449** (24 of 98 fires with confidence ≥ 0.8 were+ wrong) against the **≤ 0.05** bar. The confidence compared is the raw+ `/_decide` confidence the watcher fires on; `p_cal` was absent (the vote+ history has no labelled outcome pairs to fit), and because 18 of the 24+ confident errors sit at raw `p` = 1.000 a monotonic `p_cal` could not change+ the verdict anyway. Published under the loses-does-not-ship rule: the+ `_watcher` mechanism is real, but its default vote is not good enough to ship a+ detection watch as a "confidently correct" default, and no doc claims it is.+ Raw results, corpus generator and harness in+ `benchmarks/systemone-watcher-gate/`. Closes #1108.+ ## [1.0.0-rc.95] - 2026-10-12++ Window since v1.0.0-rc.94: legacy Microsoft Office support in autoindex,+ hardened against hostile input.++ ### Added++ - **Legacy Microsoft Office formats in `xerj autoindex`** (PR+ [#1324](https://github.com/xerj-org/xerj/pull/1324)) — OLE2 Compound File+ attachments, detected by content (never by extension), are now indexed+ instead of dropped as binary:+ - **`.xls`** (BIFF, via `calamine`) — one dataset per worksheet, one typed+ record per row, mirroring the `.xlsx` extractor.+ - **`.doc`** (Word 97+) — document text recovered through the FIB piece+ table, titled from `SummaryInformation`. Text-only by design: no+ formatting, tables-as-rows, headers/footers, footnotes, or pre-Word-97+ binaries (those fall back to a lossy read or are skipped).+ - Other OLE2 containers (e.g. `.epf` 1C data-processors) are **named and+ skipped**, not mis-parsed.++ ### Security++ - **`.xls` parsing is process-isolated** (PR+ [#1324](https://github.com/xerj-org/xerj/pull/1324)) — `calamine` 0.26.1 is+ not panic-safe on hostile BIFF (crafted `DIMENSIONS`/`SST` records force+ multi-terabyte allocations; malformed `Lbl`/`ExternSheet` records panic), and+ the engine is `panic=abort`, so a single hostile spreadsheet in an untrusted+ dump could abort the whole indexer. The BIFF read now runs in a short-lived+ `xerj __extract-xls` worker under `RLIMIT_AS`/`RLIMIT_CPU` + a wall-clock+ watchdog + process-group kill; the parent reads records back as NDJSON and+ degrades any worker crash/timeout to a skipped file, never calling `calamine`+ on the raw bytes. A `biff_allocations_bounded` pre-filter rejects the known+ over-allocation shapes before the spawn. Adversarially verified: the indexer+ survives every crafted vector while benign workbooks still extract. (Windows+ keeps the in-process pre-filter as a documented best-effort.)++ ### Housekeeping++ - Landing footer version stamps swept to rc.94 (PR+ [#1325](https://github.com/xerj-org/xerj/pull/1325)); the sweep is now part+ of the release-cut recipe.++ ### Known gaps++ - OpenDocument `.ods`/`.odt` (zip-based) are detected and **skipped** (no+ extractor yet); `.rtf` indexes as raw text. Tracked for a follow-up.## [1.0.0-rc.94] - 2026-10-11diff --cc ROADMAP.mdindex 2ebf61372,59c71db02..fd88016d7--- a/ROADMAP.md+++ b/ROADMAP.md@@@ -280,7 -294,7 +294,7 @@@ and the console review's), the #1062 wr`xerj-decide-v1` Hugging Face publication (operator credentials), and the GAbar itself — *The road to v1.0.0 GA* below.--**Open defects.** Nine, all live on the tracker — the++**Open defects.** Eight, all live on the tracker — theshortlist the release-notes gate checks. From the rc.92 window:[#1250](https://github.com/xerj-org/xerj/issues/1250) (multi-indexingest oscillates on the parent RSS breaker; root cause re-attributed@@@ -304,9 -318,8 +318,6 @@@ From the rc.81/82 windowspanic line). From the cluster-mode testing:[#1170](https://github.com/xerj-org/xerj/issues/1170) (the Raft ring isnever connected to the engine). Filed from the corpus-program work:--[#1108](https://github.com/xerj-org/xerj/issues/1108) (the #1062- wrong-and-confident detection gate — now measured on rc.94 and failed- at 0.2449 vs ≤ 5 %, published; closing on this PR),-wrong-and-confident detection gate, unmeasured),[#1110](https://github.com/xerj-org/xerj/issues/1110) (thegeneralization proof: second-domain pack + fresh-machine replay) and[#1183](https://github.com/xerj-org/xerj/issues/1183) (corpus index