Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

merge: bring in main (rc.14 + #248/#249), keeping both Unreleased entries

xerj-org0c7a366f099f
diff --cc CHANGELOG.md
index 46fccaf44,e380144d1..f172f4453
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,73 -7,28 +7,95 @@@ and this project adheres to [Semantic V
## [Unreleased]
+ ### Changed
+
+ - **`autoindex` parses each PDF once per fresh run instead of twice.** Phase A
+ already paid a *complete* parse for every PDF — `extract::extract` routes
+ `Family::Pdf` straight to the isolated worker and drops the sampling limit,
+ so only delivery ever stopped early — and phase B then spawned the worker
+ again for the same bytes. Phase A now retains each validated worker response
+ in an anonymous temporary file under `--state-dir` and phase B replays it.
+ Reuse is an optional accelerator, never correctness-critical state: it is
+ bounded to 384 MiB of retained-plus-in-flight bytes and to an artifact-handle
+ cap derived from live `RLIMIT_NOFILE` and open-descriptor counts, both
+ re-probed at every admission; a 4 GiB-or-half-free filesystem floor is
+ reserved for the journal and phase-B staging first; and every artifact is
+ verified (physical length, content digest, JSON decode, worker-protocol
+ identity) before a single record is published. Anything that cannot be
+ measured conservatively declines and reparses — which is why the optimization
+ is **Linux-only** today: other platforms have no live descriptor evidence.
+ A restart has a frozen plan but no trusted handle, so it parses again; the
+ spool is deliberately not journal state. `--json` gains a
+ `pdf_extraction_reuse` block so the behaviour is observable rather than
+ inferred from timing. Original work by Leonid Bugaev (@buger).
+
+### Fixed
+
+- **An index whose metadata will not parse is refused instead of silently
+ reopened with an empty mapping**
+ ([#202](https://github.com/xerj-org/xerj/issues/202)). `Index::open` treated
+ "this file is not there" and "this file does not parse" as the same thing:
+ `load_schema`/`load_settings` mapped ENOENT, EACCES, EIO and malformed JSON
+ alike to one anonymous error, and the caller answered all of them with a
+ fresh dynamic mapping. Measured on a field deliberately mapped `keyword`:
+ after `schema.json` was truncated to half its bytes the index still opened
+ `Ok` with `field_count = 0`, the field came back `None`, and one further
+ document re-inferred it as `long` — a mapping silently replaced by a
+ different one, with nothing in any response saying so. Absent stays absent
+ (indices predating create-time schema persistence legitimately have no
+ `schema.json`); present-but-unparseable now fails the open with an error
+ naming the file. `es_mapping.json` — the full-fidelity mapping behind
+ `GET /_mapping` — was logged-and-ignored on a parse failure and now fails the
+ index too, on the boot scan, on snapshot restore and on a retry.
+
+ **Visible on upgrade:** a node whose data dir already holds a corrupt sidecar
+ now boots **red** with that index unserved, where it previously came up green
+ with an empty mapping. It joins the failed set
+ [#206](https://github.com/xerj-org/xerj/issues/206) introduced, so every
+ surface that set feeds already reports it — measured on a two-index node with
+ one truncated `schema.json`:
+
+ ```
+ GET /_cluster/health -> red, unassigned_primary_shards: 1
+ GET /_cluster/health?level=indices
+ -> victim red, unassigned_info.details = the
+ absolute path of schema.json and the parse error
+ GET /_cat/indices -> "red open victim <uuid> 1 0 0 0 0b 0b 0b"
+ GET /_cluster/indices/failed -> the reason, plus the retry and delete calls
+ PUT /victim, POST /victim/_doc, POST /victim/_search
+ -> 503 no_shard_available_action_exception
+ carrying that same reason
+ GET /health/ready -> 200 "ready (degraded): 1 of 2 indices
+ serving, 1 failed to open"
+ POST /healthy/_search -> 200
+ ```
+
+ Recovery is the three doors #206 added, both verified end to end here: repair
+ the file and `POST /_cluster/indices/failed/{index}/_retry` (503 while it is
+ still torn, `200 {"reopened": true}` once it is not, health back to green), or
+ restore the index from a snapshot, or `DELETE /{index}`. The node stays in
+ kubelet rotation as long as any index is still serving.
+
+- **Concurrent sidecar writes can no longer manufacture the torn file the reader
+ now refuses.** `write_file_atomic` staged every write in
+ `path.with_extension("tmp")` — one shared name for all writers of that file —
+ so two concurrent writers of one sidecar (`PUT /{index}/_settings` racing
+ another settings update, two API-key mints for `api_keys.json`) both opened it
+ `O_TRUNC`, interleaved their bytes, and each renamed it into place. The rename
+ is atomic; the content being renamed was not one writer's. Measured with the
+ old shared name, 4 threads × two different-length settings bodies × 200
+ rounds, over four runs: **86–294 of 800 writes failed with ENOENT** (the loser
+ renaming a file the winner had already moved) and, with those errors swallowed
+ as the callers did, **1–16 of 200 rounds left an unparseable `settings.json`**.
+ The counts are race-dependent and vary with machine load; what does not vary is
+ that both are non-zero on every run with the shared name and **0 and 0** on
+ every run with a unique staging name. Each write now stages
+ in its own sibling file (`<file>.tmp.<pid>.<seq>`) in the target's directory
+ and removes its own debris on failure — which for `api_keys.json` is key
+ material. `update_settings` also writes `settings.json` through
+ `write_file_atomic` rather than a plain `fs::write`; it was the last
+ non-atomic sidecar writer left.
+
## [1.0.0-rc.14] - 2026-08-10
### Changed