AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
merge main into fix/issue-940-rrf-stable-ties
xerj-orgb0d0c47aeff5
diff --cc CHANGELOG.mdindex 6834fe86f,c5adc4349..1daefdd37--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -25,25 -245,179 +245,197 @@@ and this project adheres to [Semantic Vnever silently degrades to one leg. The MCP `xerj_memory_recall` toolexposes the same two parameters. Raised by @Vinz2168 from a shared-memoryagent integration where neither single mode was enough.+ - **`rerank` stage on `_search`: hand the top hits to an external relevance+ judge and reorder by the calibrated probability that comes back.**+ `"rerank": {}` sends the question and the text of the top `window` hits+ (default 30, max 300) to TypeSafe AI's Jev; the 0–1 probability replaces+ `_score`, so `rerank.min_score` is an absolute cut-off, which a BM25 score+ cannot be. **It is the only search-time feature that sends document text+ off the node** (`[embedding] default_endpoint` and the WAL tap also send text+ off the node when an operator configures them; docs/RERANK.md lists every+ outbound connection a node can open, and a test fails when the engine source+ gains one that list does not name): inert until+ an operator sets `[rerank] api_key` (or+ `TYPESAFE_API_KEY`; config wins over env), opt-in per request, forbidden+ outright by `[rerank] enabled = false`, and only fields the response returns+ are sent — `rerank.fields` is an exhaustive allow-list. Failure policy is+ *degrade on deadline, surface on contract*: a slow provider is a 200 with the+ engine's order and `_rerank.applied: false`; no key is 503, disabled 403, a+ rejected key or malformed body 502, all with no hits; a provider that answers+ for nothing it was sent is a 200 with `applied: false`. When applied, every+ `_score` is a probability or `null` (a hit with no verdict, counted in+ `_rerank.unjudged`, sorts last) — never an engine score beside probabilities.+ `hits.total` and `aggs`+ stay the engine's; paging happens inside the window; `sort`, `search_after`,+ `collapse`, scroll and `size: 0` are 400s; and `_msearch`, search templates,+ `_async_search`, `_rank_eval` (per request, under `failures`), the native+ `/v1` search API and gRPC refuse the block instead of dropping it. Every+ caller-chosen cost knob has a server-side ceiling, the strings included:+ `instructions` is capped at 2,000 characters because the provider's wire+ format repeats it once per judged document, the question at 4,000 and+ `model` at 128; a provider response over 2 MiB is a 502, not an allocation.+ `GET /_xerj/rerank` reports whether a provider is configured+ and never the key; `/v1/metrics` gains `xerj_rerank_requests_total{outcome}`,+ `xerj_rerank_documents_judged_total` and+ `xerj_rerank_provider_tokens_total{kind}`; the MCP `xerj_search` and+ `xerj_hybrid_search` tools take an optional `rerank` argument. **Ranking+ quality with the real model is not verified** — no provider key was+ available, and every HTTP test runs against an in-process test double.+ New crate `xerj-rerank`; reference in `docs/RERANK.md`; two benchmarks,+ `benchmarks/beir-hybrid` and `benchmarks/decisions-as-retrieval`, both+ measured with `--embed-mode neural`, not the default lexical embedder. The+ one-question-per-document request shape follows `hev/jev-rerank`+ (Apache-2.0); the failure policy and the retry back-off constants follow+ Meilisearch's personalization module (MIT; adapted, cited in code).++ ### Fixed++ - **`xerj autoindex`: one dataset the server refuses no longer aborts the run,+ `--no-graph` progress reports the phase it is in, and `xc-index.sh --fresh`+ works on a corpus that was indexed before**+ ([#929](https://github.com/xerj-org/xerj/issues/929),+ [#931](https://github.com/xerj-org/xerj/issues/931),+ [#930](https://github.com/xerj-org/xerj/issues/930)). On rc.74 a single HTTP+ 400 on one dataset's mapping ended a 48,533-file run with `exit=1+ reason=aborted` and nothing indexed. A 400 on create-index / put-mapping is+ now a refusal of *that dataset*: its files are recorded as junk with the+ server's reason, every other dataset is indexed, the run exits 3, and+ `xerj-done`, the catalog run document and `xerj autoindex map` all carry+ `datasets_refused` / `files_refused`, so a generation that lacks a dataset+ cannot read as a whole one. 401/403/404/408/429/5xx still abort. On the+ `--no-graph` path the stream used to print `phase=scan pct=100.0+ eta_quality=stalled` for the whole time documents were landing (48 such lines+ in the rc.74 capture) — indistinguishable from a real hang; mapping install,+ sealing, indexing and the read-back barrier are now the `prepare`, `snapshot`,+ `index`, `finalize-catalog`, `finalize-refresh` and `finalize-verify` phases+ with real denominators. `tools/xerj-code/scripts/xc-index.sh --fresh` no+ longer forwards the flag to autoindex (which refuses it once a generation has+ committed): it builds a replacement beside the old index, verifies+ `_count > 0`, switches the state file atomically and only then retires the old+ indices by exact name; a record count the node does not answer is never read+ as zero, so it cannot get a working index retired or a finished build+ deleted. A first build that is interrupted after writing+ records is kept rather than leave no corpus, and `xc.py` now reads the+ `salvaged` / `autoindex_exit` the script always recorded: it warns on every+ query that coverage is INCOMPLETE, so a miss against a partial index cannot+ read as "this code does not exist". Found while verifying the above on the full corpus+ ([#944](https://github.com/xerj-org/xerj/issues/944)): a per-item HTTP 429+ inside one bulk — the node's memory circuit breaker, which engages and+ releases within seconds — aborted the 48,533-file run at 60% after 85+ minutes. The rejected items are now re-sent, and only those, after a backoff,+ until the node takes them or the bulk's 600 s of patience runs out — a+ 429 on the whole bulk request is the same back-pressure, not six transport+ attempts (that split ended the resumed run);+ only then is it exit 1, resumable, with an error line that says so. The+ terminal line carries `bulk_retries=N` when it happened, and the+ `raising bulk concurrency` line is printed at most once per 10 s (117 lines+ for 11 shrinks in the capture). Captures: `benchmarks/autoindex-resilience/`.++ - **`xerj autoindex`: a corpus whose catalog holds more documents than the+ server takes in one request no longer fails at the very end**+ ([#955](https://github.com/xerj-org/xerj/issues/955)). Both indexing paths+ sent the catalog (one document per file, per dataset, per run) as ONE+ `_bulk`. On the 48,533-file reference corpus that was 51,129 actions in+ 31.9 MB against the engine's default `limits.max_actions_per_bulk` of 50,000:+ the `--no-graph` run applied all 47,444 operations, then ended+ `exit=1 reason=aborted` in `finalize-catalog` after 10,336 s. On the default+ path the same answer was silent: exit 0 with an empty catalog (reproduced by+ a test against a stub that returns the engine's literal 413). Every `_bulk`+ body now goes out in windows of at most 10,000 actions, and the catalog also+ under `--bulk-mb`. A request the server still refuses as too large (HTTP 413,+ or one item answered 413 for a body of several actions) is halved and+ re-sent, and the bound is kept for the rest of the run; the terminal line+ carries `bulk_splits=N` when that happened. Resuming that same generation+ with the fix committed it (`ok=true exit=3 … records=821840`, 51,129 catalog+ documents). A `--no-graph` run that runs out of back-pressure patience now+ ends `reason=server-backpressure` with `ops_applied` / `ops_remaining`+ instead of `reason=aborted`; it is still exit 1 and resumable.++ - **Reference-code passage windows preserve their match score.** `xc.py`+ could select relevant source and then discard its matches while aligning+ the excerpt to line boundaries, including on long source lines. Line+ alignment now keeps the selected term score or falls back to the bounded+ original window. Window scoring also uses original-source offsets when+ Unicode lowercasing expands characters. This affects the fallback when no+ matching symbol is available and explicit `--no-symbol` output.++ - **`xc.py --mode hybrid` keeps BM25 results when the optional vector arm has+ a transport failure.** Connection failures, read timeouts, and interrupted+ HTTP responses during semantic mapping discovery or search now take the+ existing BM25-only fallback instead of aborting and discarding valid hits.+ Primary BM25 failures still exit with an error. Standalone `--mode semantic`+ now reports mapping/search HTTP and transport failures as errors (exit `2`)+ instead of treating failed requests as empty search results (exit `1`).++ - **`xc.py --json` emits JSON for empty search results.** Previously, a query+ with no hits printed the human-readable no-match message before reaching the+ JSON output branch, breaking callers that parse stdout. Empty results now+ preserve the JSON response and still exit `1`; matching results exit `0`, and+ a corpus with no live indices retains its distinct exit `3` diagnostic.++ ### Documentation++ - **`llms.txt` proposals, revised and fact-checked**+ (`docs/research/llms-txt-2026-09/proposals/`). A proposed `llms.txt`, an+ `llms-install.md` and ten paste-ready install prompts, following the study's+ twelve rules: three complete entry paths (shell, MCP-only, HTTP-only), the+ lexical-by-default correction first, per-client MCP registration with the+ key, a verify line after every step, and a feedback ask that is optional+ and needs no git. Every quote is re-fetched (259 claims, 258 confirmed, the+ one failure removed) and the XERJ commands the proposals print were run on+ Linux against v1.0.0-rc.74, except the ones the record's own "Not run" list+ names — the `curl | sh` installer, macOS and Windows, `--embed-mode neural`,+ `claude mcp add --scope local` and the per-client registration lines, which+ are quoted from each client's own documentation with a fact-check id. Two product findings came out of it:+ `xerj feedback --open-pr` branches, commits and pushes in whatever+ repository it is run from, and `xerj init` writes an MCP entry without+ `XERJ_AUTH`, so against a default node every tool call returns 401. The+ live `landing/llms.txt` is unchanged; the report's ship checklist says what+ has to exist first.++ - **ROADMAP: the zero-token direction, with every status checked against the+ tree** ([#941](https://github.com/xerj-org/xerj/issues/941)). A new roadmap+ section and [docs/ZERO_TOKEN_DIRECTION.md](./docs/ZERO_TOKEN_DIRECTION.md)+ lay out judged search, share links and a guest reading room, mail ingest,+ semantic detections, a real object-storage backend, a block index mode for+ logs, user-code ingest plugins and a corpus hub of signed packs — and say+ plainly what exists today: `S3Backend` was a local-directory simulation and+ `storage.backend = "s3"` refuses to start on purpose — the first half of that+ is no longer true in this same release, see the object-storage backend entry+ above; the refusal stands, because the index segment path is still local —+ `_watcher` stores+ watches and never evaluates them, alert rules have schemas and no+ evaluator, `xerj-logs` has no caller, and there is no wasmtime backend in+ the tree. The measuring behind it+ ([benchmarks/neural-path-triage/](./benchmarks/neural-path-triage/)) filed+ four defects with literal reproductions:+ [#937](https://github.com/xerj-org/xerj/issues/937) a declared analyzer+ stops applying at flush, [#938](https://github.com/xerj-org/xerj/issues/938)+ neural ingest keeps ~3.4 of 32 threads busy,+ [#939](https://github.com/xerj-org/xerj/issues/939) `semantic` over+ multi-passage documents is an exact scan that copies every `_source`+ (~410 ms p50 on 5,183 documents, forward pass ~14 ms), and+ [#940](https://github.com/xerj-org/xerj/issues/940) tied RRF scores change+ order across a restart. All four stay open; nothing is fixed by this entry.+### Fixed++- **A `hybrid` query returns the same page on every run and after a+ restart** ([#940](https://github.com/xerj-org/xerj/issues/940)).+ `fusion: "rrf"` and `fusion: "linear"` drained their accumulator from a+ hash map into a score-only sort, so documents with an identical fused+ score came back in hash-iteration order, which is seeded per process.+ Under RRF such ties are structural (one leg's rank *r* and the other+ leg's rank *r* score the same), and on BEIR SciFact 21 of 40 queries+ changed their top 10 after a restart on unchanged data. Fused results now+ use the order every score-ranked page uses since #270: score descending,+ then arrival (`_seq_no`), then `_id`; a NaN fused score sorts last+ instead of making the sort panic. Measured on the fixed build, lexical+ embedder: the 300 SciFact test queries (1,509 tied adjacent pairs in+ their top 100) returned identical result lists across a flush and two+ restarts. The hybrid recipe's worked example was also re-run and its+ numbers reconciled with the current engine.+## [1.0.0-rc.74] - 2026-09-08### Added