Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into HEAD

xerj-orgf8ed64db711e
diff --cc CHANGELOG.md
index 10e2a5739,bae4e4abd..94932cb6a
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,25 -7,52 +7,71 @@@ and this project adheres to [Semantic V
## [Unreleased]
+### Measured
+
+- **The #1062 wrong-and-confident detection gate was run through the real
+ `_watcher` path and it FAILS** (issue
+ [#1108](https://github.com/xerj-org/xerj/issues/1108)) — the unmeasured half
+ of the rc.80 `_watcher` work. On the rc.94 release binary, a real
+ `PUT /_watcher/watch` with `condition.xerj_decide` (one detection watch per
+ label, one-vs-rest) evaluated a 120-email boundary tier and wrote 120 fire
+ records into `.xerj_alert_fires`; the measurement reads only those fires.
+ **Wrong-and-confident = 0.2449** (24 of 98 fires with confidence ≥ 0.8 were
+ wrong) against the **≤ 0.05** bar. The confidence compared is the raw
+ `/_decide` confidence the watcher fires on; `p_cal` was absent (the vote
+ history has no labelled outcome pairs to fit), and because 18 of the 24
+ confident errors sit at raw `p` = 1.000 a monotonic `p_cal` could not change
+ the verdict anyway. Published under the loses-does-not-ship rule: the
+ `_watcher` mechanism is real, but its default vote is not good enough to ship a
+ detection watch as a "confidently correct" default, and no doc claims it is.
+ Raw results, corpus generator and harness in
+ `benchmarks/systemone-watcher-gate/`. Closes #1108.
+ ## [1.0.0-rc.95] - 2026-10-12
+
+ Window since v1.0.0-rc.94: legacy Microsoft Office support in autoindex,
+ hardened against hostile input.
+
+ ### Added
+
+ - **Legacy Microsoft Office formats in `xerj autoindex`** (PR
+ [#1324](https://github.com/xerj-org/xerj/pull/1324)) — OLE2 Compound File
+ attachments, detected by content (never by extension), are now indexed
+ instead of dropped as binary:
+ - **`.xls`** (BIFF, via `calamine`) — one dataset per worksheet, one typed
+ record per row, mirroring the `.xlsx` extractor.
+ - **`.doc`** (Word 97+) — document text recovered through the FIB piece
+ table, titled from `SummaryInformation`. Text-only by design: no
+ formatting, tables-as-rows, headers/footers, footnotes, or pre-Word-97
+ binaries (those fall back to a lossy read or are skipped).
+ - Other OLE2 containers (e.g. `.epf` 1C data-processors) are **named and
+ skipped**, not mis-parsed.
+
+ ### Security
+
+ - **`.xls` parsing is process-isolated** (PR
+ [#1324](https://github.com/xerj-org/xerj/pull/1324)) — `calamine` 0.26.1 is
+ not panic-safe on hostile BIFF (crafted `DIMENSIONS`/`SST` records force
+ multi-terabyte allocations; malformed `Lbl`/`ExternSheet` records panic), and
+ the engine is `panic=abort`, so a single hostile spreadsheet in an untrusted
+ dump could abort the whole indexer. The BIFF read now runs in a short-lived
+ `xerj __extract-xls` worker under `RLIMIT_AS`/`RLIMIT_CPU` + a wall-clock
+ watchdog + process-group kill; the parent reads records back as NDJSON and
+ degrades any worker crash/timeout to a skipped file, never calling `calamine`
+ on the raw bytes. A `biff_allocations_bounded` pre-filter rejects the known
+ over-allocation shapes before the spawn. Adversarially verified: the indexer
+ survives every crafted vector while benign workbooks still extract. (Windows
+ keeps the in-process pre-filter as a documented best-effort.)
+
+ ### Housekeeping
+
+ - Landing footer version stamps swept to rc.94 (PR
+ [#1325](https://github.com/xerj-org/xerj/pull/1325)); the sweep is now part
+ of the release-cut recipe.
+
+ ### Known gaps
+
+ - OpenDocument `.ods`/`.odt` (zip-based) are detected and **skipped** (no
+ extractor yet); `.rtf` indexes as raw text. Tracked for a follow-up.
## [1.0.0-rc.94] - 2026-10-11
diff --cc ROADMAP.md
index 2ebf61372,59c71db02..fd88016d7
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@@ -280,7 -294,7 +294,7 @@@ and the console review's), the #1062 wr
`xerj-decide-v1` Hugging Face publication (operator credentials), and the GA
bar itself — *The road to v1.0.0 GA* below.
--**Open defects.** Nine, all live on the tracker — the
++**Open defects.** Eight, all live on the tracker — the
shortlist the release-notes gate checks. From the rc.92 window:
[#1250](https://github.com/xerj-org/xerj/issues/1250) (multi-index
ingest oscillates on the parent RSS breaker; root cause re-attributed
@@@ -304,9 -318,8 +318,6 @@@ From the rc.81/82 windows
panic line). From the cluster-mode testing:
[#1170](https://github.com/xerj-org/xerj/issues/1170) (the Raft ring is
never connected to the engine). Filed from the corpus-program work:
--[#1108](https://github.com/xerj-org/xerj/issues/1108) (the #1062
- wrong-and-confident detection gate — now measured on rc.94 and failed
- at 0.2449 vs ≤ 5 %, published; closing on this PR),
-wrong-and-confident detection gate, unmeasured),
[#1110](https://github.com/xerj-org/xerj/issues/1110) (the
generalization proof: second-domain pack + fresh-machine replay) and
[#1183](https://github.com/xerj-org/xerj/issues/1183) (corpus index