AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
merge: bring in main (#254 + the autoindex rerun-refusal wave)
xerj-org252857162b05
diff --cc CHANGELOG.mdindex 76e5feed1,ef625a381..1a32cf870--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -29,84 -29,55 +29,133 @@@ and this project adheres to [Semantic V`pdf_extraction_reuse` block so the behaviour is observable rather thaninferred from timing. Original work by Leonid Bugaev (@buger).+ ### Changed — autoindex refuses reruns that would strand documents++ - **Deleting an indexed file and rerunning `xerj autoindex` now fails instead+ of exiting 0.** Nothing in the pipeline removes the documents, aliases,+ graph edges or catalog entries that the deleted file published, so a rerun+ that ignored the deletion left them live and searchable with no source file+ behind them. The rerun is now refused before any remote call other than the+ endpoint-readiness ping: no mapping, delete-by-query, bulk, refresh, graph+ or catalog write is attempted, and the journal is not appended to. The error+ names the removed files and their content keys — the first ten, then an+ `… and N more` tail, since a whole unmounted subtree can vanish at once —+ and gives three recovery routes: restore the files and rerun, rebuild in+ place by deleting the named indices and the state directory, or rebuild+ under a new state directory, prefix and brain. `--fresh` is refused for the+ same case, because it does not delete those documents either. `--json` emits+ the same facts — every removed entry, uncapped — as+ `xerj.autoindex.unsupported_sync_delta.v1` on stdout with exit 1. Deleting a+ file that was only ever *skipped* is not refused: a junk file publishes no+ documents, and the stale junk-catalog sweep removes its one catalog row, so+ nothing is stranded.+ - **Files added after a plan was frozen are now called out on stderr.** They+ are still not indexed by a rerun that resumes an existing plan — the plan is+ a crash-resume boundary, not a folder-sync generation — but the run now+ lists them, records them as skipped (exit 3, completed-with-junk) and points+ at `--fresh`, which rebuilds the plan in place and picks them up. Adding or+ changing files and rerunning keeps working; only removals are refused.+ - **`xerj brain` no longer turns an absent or zero node-count probe into an+ automatic reset.** Journal/server disagreement now fails with the journal,+ URL, prefix, and brain identity plus recovery instructions, including the+ explicit `--fresh` rerun for a genuinely wiped data directory. Probe+ transport and malformed-response failures remain errors instead of being+ reported as an empty destination. A nodes index that was deleted out from+ under a surviving brain meta doc reads as absence and reaches that recovery+ text, rather than surfacing as a raw HTTP 404.+ - **`--fresh` still recovers a state directory whose journal cannot be+ parsed.** The rerun gate reads the durable plan before the run starts, and a+ plan that is malformed, or recorded for a different root/URL/prefix, is fatal+ to a resume — but not to `--fresh`, which deletes that journal unread. The+ preflight is now no more fatal than the open it precedes: under `--fresh` the+ unreadable plan is reported on stderr and rebuilt from the current folder.+ Without `--fresh` the refusal is unchanged. Because the removal gate has no+ comparison basis in that case, the warning says so: documents already+ published for files that are now gone cannot be identified from an unreadable+ plan and are not deleted.+ - **Refusal and skip listings are capped at ten entries plus an "and N more"+ tail.** Unmounting a bind mount under an indexed root vanishes every content+ group at once, so the uncapped listing was one rendered entry per journalled+ file. `--json` still carries every entry.++### Fixed++- **An index whose metadata will not parse is refused instead of silently+ reopened with an empty mapping**+ ([#202](https://github.com/xerj-org/xerj/issues/202)). `Index::open` treated+ "this file is not there" and "this file does not parse" as the same thing:+ `load_schema`/`load_settings` mapped ENOENT, EACCES, EIO and malformed JSON+ alike to one anonymous error, and the caller answered all of them with a+ fresh dynamic mapping. Measured on a field deliberately mapped `keyword`:+ after `schema.json` was truncated to half its bytes the index still opened+ `Ok` with `field_count = 0`, the field came back `None`, and one further+ document re-inferred it as `long` — a mapping silently replaced by a+ different one, with nothing in any response saying so. Absent stays absent+ (indices predating create-time schema persistence legitimately have no+ `schema.json`); present-but-unparseable now fails the open with an error+ naming the file. `es_mapping.json` — the full-fidelity mapping behind+ `GET /_mapping` — was logged-and-ignored on a parse failure and now fails the+ index too, on the boot scan, on snapshot restore and on a retry.++ **Visible on upgrade:** a node whose data dir already holds a corrupt sidecar+ now boots **red** with that index unserved, where it previously came up green+ with an empty mapping. It joins the failed set+ [#206](https://github.com/xerj-org/xerj/issues/206) introduced, so every+ surface that set feeds already reports it — measured on a node holding two+ user indices, `victim` with a truncated `schema.json`:++ ```+ GET /_cluster/health -> red, unassigned_primary_shards: 1+ GET /_cluster/health?level=indices+ -> victim red, unassigned_info.details = the+ absolute path of schema.json and the parse error+ GET /_cat/indices -> "red open victim <uuid> 1 0 0 0 0b 0b 0b"+ GET /_cluster/indices/failed -> the reason, plus the retry and delete calls+ PUT /victim, POST /victim/_doc, POST /victim/_search+ -> 503 no_shard_available_action_exception+ carrying that same reason+ GET /health/ready -> 200 "ready (degraded): 1 of 2 indices+ serving, 1 failed to open; see GET+ /_cluster/indices/failed"+ POST /healthy/_search -> 200+ ```++ The readiness counts above are the two-index test router's. A running node+ also holds its internal `.xerj_*` indices, so the same condition on the shipped+ binary reads `ready (degraded): 15 of 16 indices serving, 1 failed to open` —+ the 200 and its meaning are unchanged, only the totals move with what else the+ node holds.++ Recovery is the three doors #206 added. Two were measured end to end on the+ shipped binary: repair the file and+ `POST /_cluster/indices/failed/{index}/_retry` (503 while it is still torn,+ `200 {"reopened": true}` once it is not, health back to green), and+ `DELETE /{index}` (200, directory removed, health back to green). The third —+ restore the index from a snapshot — is the path this change touches (a+ successful restore now clears the recorded failure, which it has to or health+ would stay red after the repair worked); it is covered by an engine-level+ test rather than measured over HTTP. The node stays in kubelet rotation as+ long as any index is still serving.++- **Concurrent sidecar writes can no longer manufacture the torn file the reader+ now refuses.** `write_file_atomic` staged every write in+ `path.with_extension("tmp")` — one shared name for all writers of that file —+ so two concurrent writers of one sidecar (`PUT /{index}/_settings` racing+ another settings update, two API-key mints for `api_keys.json`) both opened it+ `O_TRUNC`, interleaved their bytes, and each renamed it into place. The rename+ is atomic; the content being renamed was not one writer's. Measured with the+ old shared name, 4 threads × two different-length settings bodies × 200+ rounds, over four runs: **86–294 of 800 writes failed with ENOENT** (the loser+ renaming a file the winner had already moved) and, with those errors swallowed+ as the callers did, **1–16 of 200 rounds left an unparseable `settings.json`**.+ The counts are race-dependent and vary with machine load; what does not vary is+ that both are non-zero on every run with the shared name and **0 and 0** on+ every run with a unique staging name. Each write now stages+ in its own sibling file (`<file>.tmp.<pid>.<seq>`) in the target's directory+ and removes its own debris on failure — which for `api_keys.json` is key+ material. `update_settings` also writes `settings.json` through+ `write_file_atomic` rather than a plain `fs::write`; it was the last+ non-atomic sidecar writer left.## [1.0.0-rc.14] - 2026-08-10