Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

merge main into fix/issue-940-rrf-stable-ties

xerj-orgb0d0c47aeff5
diff --cc CHANGELOG.md
index 6834fe86f,c5adc4349..1daefdd37
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -25,25 -245,179 +245,197 @@@ and this project adheres to [Semantic V
never silently degrades to one leg. The MCP `xerj_memory_recall` tool
exposes the same two parameters. Raised by @Vinz2168 from a shared-memory
agent integration where neither single mode was enough.
+ - **`rerank` stage on `_search`: hand the top hits to an external relevance
+ judge and reorder by the calibrated probability that comes back.**
+ `"rerank": {}` sends the question and the text of the top `window` hits
+ (default 30, max 300) to TypeSafe AI's Jev; the 0–1 probability replaces
+ `_score`, so `rerank.min_score` is an absolute cut-off, which a BM25 score
+ cannot be. **It is the only search-time feature that sends document text
+ off the node** (`[embedding] default_endpoint` and the WAL tap also send text
+ off the node when an operator configures them; docs/RERANK.md lists every
+ outbound connection a node can open, and a test fails when the engine source
+ gains one that list does not name): inert until
+ an operator sets `[rerank] api_key` (or
+ `TYPESAFE_API_KEY`; config wins over env), opt-in per request, forbidden
+ outright by `[rerank] enabled = false`, and only fields the response returns
+ are sent — `rerank.fields` is an exhaustive allow-list. Failure policy is
+ *degrade on deadline, surface on contract*: a slow provider is a 200 with the
+ engine's order and `_rerank.applied: false`; no key is 503, disabled 403, a
+ rejected key or malformed body 502, all with no hits; a provider that answers
+ for nothing it was sent is a 200 with `applied: false`. When applied, every
+ `_score` is a probability or `null` (a hit with no verdict, counted in
+ `_rerank.unjudged`, sorts last) — never an engine score beside probabilities.
+ `hits.total` and `aggs`
+ stay the engine's; paging happens inside the window; `sort`, `search_after`,
+ `collapse`, scroll and `size: 0` are 400s; and `_msearch`, search templates,
+ `_async_search`, `_rank_eval` (per request, under `failures`), the native
+ `/v1` search API and gRPC refuse the block instead of dropping it. Every
+ caller-chosen cost knob has a server-side ceiling, the strings included:
+ `instructions` is capped at 2,000 characters because the provider's wire
+ format repeats it once per judged document, the question at 4,000 and
+ `model` at 128; a provider response over 2 MiB is a 502, not an allocation.
+ `GET /_xerj/rerank` reports whether a provider is configured
+ and never the key; `/v1/metrics` gains `xerj_rerank_requests_total{outcome}`,
+ `xerj_rerank_documents_judged_total` and
+ `xerj_rerank_provider_tokens_total{kind}`; the MCP `xerj_search` and
+ `xerj_hybrid_search` tools take an optional `rerank` argument. **Ranking
+ quality with the real model is not verified** — no provider key was
+ available, and every HTTP test runs against an in-process test double.
+ New crate `xerj-rerank`; reference in `docs/RERANK.md`; two benchmarks,
+ `benchmarks/beir-hybrid` and `benchmarks/decisions-as-retrieval`, both
+ measured with `--embed-mode neural`, not the default lexical embedder. The
+ one-question-per-document request shape follows `hev/jev-rerank`
+ (Apache-2.0); the failure policy and the retry back-off constants follow
+ Meilisearch's personalization module (MIT; adapted, cited in code).
+
+ ### Fixed
+
+ - **`xerj autoindex`: one dataset the server refuses no longer aborts the run,
+ `--no-graph` progress reports the phase it is in, and `xc-index.sh --fresh`
+ works on a corpus that was indexed before**
+ ([#929](https://github.com/xerj-org/xerj/issues/929),
+ [#931](https://github.com/xerj-org/xerj/issues/931),
+ [#930](https://github.com/xerj-org/xerj/issues/930)). On rc.74 a single HTTP
+ 400 on one dataset's mapping ended a 48,533-file run with `exit=1
+ reason=aborted` and nothing indexed. A 400 on create-index / put-mapping is
+ now a refusal of *that dataset*: its files are recorded as junk with the
+ server's reason, every other dataset is indexed, the run exits 3, and
+ `xerj-done`, the catalog run document and `xerj autoindex map` all carry
+ `datasets_refused` / `files_refused`, so a generation that lacks a dataset
+ cannot read as a whole one. 401/403/404/408/429/5xx still abort. On the
+ `--no-graph` path the stream used to print `phase=scan pct=100.0
+ eta_quality=stalled` for the whole time documents were landing (48 such lines
+ in the rc.74 capture) — indistinguishable from a real hang; mapping install,
+ sealing, indexing and the read-back barrier are now the `prepare`, `snapshot`,
+ `index`, `finalize-catalog`, `finalize-refresh` and `finalize-verify` phases
+ with real denominators. `tools/xerj-code/scripts/xc-index.sh --fresh` no
+ longer forwards the flag to autoindex (which refuses it once a generation has
+ committed): it builds a replacement beside the old index, verifies
+ `_count > 0`, switches the state file atomically and only then retires the old
+ indices by exact name; a record count the node does not answer is never read
+ as zero, so it cannot get a working index retired or a finished build
+ deleted. A first build that is interrupted after writing
+ records is kept rather than leave no corpus, and `xc.py` now reads the
+ `salvaged` / `autoindex_exit` the script always recorded: it warns on every
+ query that coverage is INCOMPLETE, so a miss against a partial index cannot
+ read as "this code does not exist". Found while verifying the above on the full corpus
+ ([#944](https://github.com/xerj-org/xerj/issues/944)): a per-item HTTP 429
+ inside one bulk — the node's memory circuit breaker, which engages and
+ releases within seconds — aborted the 48,533-file run at 60% after 85
+ minutes. The rejected items are now re-sent, and only those, after a backoff,
+ until the node takes them or the bulk's 600 s of patience runs out — a
+ 429 on the whole bulk request is the same back-pressure, not six transport
+ attempts (that split ended the resumed run);
+ only then is it exit 1, resumable, with an error line that says so. The
+ terminal line carries `bulk_retries=N` when it happened, and the
+ `raising bulk concurrency` line is printed at most once per 10 s (117 lines
+ for 11 shrinks in the capture). Captures: `benchmarks/autoindex-resilience/`.
+
+ - **`xerj autoindex`: a corpus whose catalog holds more documents than the
+ server takes in one request no longer fails at the very end**
+ ([#955](https://github.com/xerj-org/xerj/issues/955)). Both indexing paths
+ sent the catalog (one document per file, per dataset, per run) as ONE
+ `_bulk`. On the 48,533-file reference corpus that was 51,129 actions in
+ 31.9 MB against the engine's default `limits.max_actions_per_bulk` of 50,000:
+ the `--no-graph` run applied all 47,444 operations, then ended
+ `exit=1 reason=aborted` in `finalize-catalog` after 10,336 s. On the default
+ path the same answer was silent: exit 0 with an empty catalog (reproduced by
+ a test against a stub that returns the engine's literal 413). Every `_bulk`
+ body now goes out in windows of at most 10,000 actions, and the catalog also
+ under `--bulk-mb`. A request the server still refuses as too large (HTTP 413,
+ or one item answered 413 for a body of several actions) is halved and
+ re-sent, and the bound is kept for the rest of the run; the terminal line
+ carries `bulk_splits=N` when that happened. Resuming that same generation
+ with the fix committed it (`ok=true exit=3 … records=821840`, 51,129 catalog
+ documents). A `--no-graph` run that runs out of back-pressure patience now
+ ends `reason=server-backpressure` with `ops_applied` / `ops_remaining`
+ instead of `reason=aborted`; it is still exit 1 and resumable.
+
+ - **Reference-code passage windows preserve their match score.** `xc.py`
+ could select relevant source and then discard its matches while aligning
+ the excerpt to line boundaries, including on long source lines. Line
+ alignment now keeps the selected term score or falls back to the bounded
+ original window. Window scoring also uses original-source offsets when
+ Unicode lowercasing expands characters. This affects the fallback when no
+ matching symbol is available and explicit `--no-symbol` output.
+
+ - **`xc.py --mode hybrid` keeps BM25 results when the optional vector arm has
+ a transport failure.** Connection failures, read timeouts, and interrupted
+ HTTP responses during semantic mapping discovery or search now take the
+ existing BM25-only fallback instead of aborting and discarding valid hits.
+ Primary BM25 failures still exit with an error. Standalone `--mode semantic`
+ now reports mapping/search HTTP and transport failures as errors (exit `2`)
+ instead of treating failed requests as empty search results (exit `1`).
+
+ - **`xc.py --json` emits JSON for empty search results.** Previously, a query
+ with no hits printed the human-readable no-match message before reaching the
+ JSON output branch, breaking callers that parse stdout. Empty results now
+ preserve the JSON response and still exit `1`; matching results exit `0`, and
+ a corpus with no live indices retains its distinct exit `3` diagnostic.
+
+ ### Documentation
+
+ - **`llms.txt` proposals, revised and fact-checked**
+ (`docs/research/llms-txt-2026-09/proposals/`). A proposed `llms.txt`, an
+ `llms-install.md` and ten paste-ready install prompts, following the study's
+ twelve rules: three complete entry paths (shell, MCP-only, HTTP-only), the
+ lexical-by-default correction first, per-client MCP registration with the
+ key, a verify line after every step, and a feedback ask that is optional
+ and needs no git. Every quote is re-fetched (259 claims, 258 confirmed, the
+ one failure removed) and the XERJ commands the proposals print were run on
+ Linux against v1.0.0-rc.74, except the ones the record's own "Not run" list
+ names — the `curl | sh` installer, macOS and Windows, `--embed-mode neural`,
+ `claude mcp add --scope local` and the per-client registration lines, which
+ are quoted from each client's own documentation with a fact-check id. Two product findings came out of it:
+ `xerj feedback --open-pr` branches, commits and pushes in whatever
+ repository it is run from, and `xerj init` writes an MCP entry without
+ `XERJ_AUTH`, so against a default node every tool call returns 401. The
+ live `landing/llms.txt` is unchanged; the report's ship checklist says what
+ has to exist first.
+
+ - **ROADMAP: the zero-token direction, with every status checked against the
+ tree** ([#941](https://github.com/xerj-org/xerj/issues/941)). A new roadmap
+ section and [docs/ZERO_TOKEN_DIRECTION.md](./docs/ZERO_TOKEN_DIRECTION.md)
+ lay out judged search, share links and a guest reading room, mail ingest,
+ semantic detections, a real object-storage backend, a block index mode for
+ logs, user-code ingest plugins and a corpus hub of signed packs — and say
+ plainly what exists today: `S3Backend` was a local-directory simulation and
+ `storage.backend = "s3"` refuses to start on purpose — the first half of that
+ is no longer true in this same release, see the object-storage backend entry
+ above; the refusal stands, because the index segment path is still local —
+ `_watcher` stores
+ watches and never evaluates them, alert rules have schemas and no
+ evaluator, `xerj-logs` has no caller, and there is no wasmtime backend in
+ the tree. The measuring behind it
+ ([benchmarks/neural-path-triage/](./benchmarks/neural-path-triage/)) filed
+ four defects with literal reproductions:
+ [#937](https://github.com/xerj-org/xerj/issues/937) a declared analyzer
+ stops applying at flush, [#938](https://github.com/xerj-org/xerj/issues/938)
+ neural ingest keeps ~3.4 of 32 threads busy,
+ [#939](https://github.com/xerj-org/xerj/issues/939) `semantic` over
+ multi-passage documents is an exact scan that copies every `_source`
+ (~410 ms p50 on 5,183 documents, forward pass ~14 ms), and
+ [#940](https://github.com/xerj-org/xerj/issues/940) tied RRF scores change
+ order across a restart. All four stay open; nothing is fixed by this entry.
+### Fixed
+
+- **A `hybrid` query returns the same page on every run and after a
+ restart** ([#940](https://github.com/xerj-org/xerj/issues/940)).
+ `fusion: "rrf"` and `fusion: "linear"` drained their accumulator from a
+ hash map into a score-only sort, so documents with an identical fused
+ score came back in hash-iteration order, which is seeded per process.
+ Under RRF such ties are structural (one leg's rank *r* and the other
+ leg's rank *r* score the same), and on BEIR SciFact 21 of 40 queries
+ changed their top 10 after a restart on unchanged data. Fused results now
+ use the order every score-ranked page uses since #270: score descending,
+ then arrival (`_seq_no`), then `_id`; a NaN fused score sorts last
+ instead of making the sort panic. Measured on the fixed build, lexical
+ embedder: the 300 SciFact test queries (1,509 tied adjacent pairs in
+ their top 100) returned identical result lists across a flush and two
+ restarts. The hybrid recipe's worked example was also re-run and its
+ numbers reconciled with the current engine.
+
## [1.0.0-rc.74] - 2026-09-08
### Added