AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge origin/main into codex/xc-passage-match-preservation
xerj-orgab4e07c68e00
diff --cc .github/workflows/ci.ymlindex 4048e9e65,1e4f17d19..91114d6b6--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@@ -96,9 -96,9 +96,12 @@@ jobs- name: Corpus manifest round-triprun: tools/xerj-code/tests/test_xc_corpus.sh+ - name: Matching passage window regressions+ run: python3 tools/xerj-code/tests/test_passage_window.py++ - name: Retrieval output and state-ledger regressions+ run: python3 tools/xerj-code/tests/test_state_ledger.py+- name: Vetted hub manifests validaterun: python3 tools/xerj-code/tests/validate_manifest.py --hub tools/xerj-code/hub/*.jsondiff --cc CHANGELOG.mdindex e31eb1b20,8f7b12f1a..4a222d18d--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -28,14 -28,62 +28,70 @@@ and this project adheres to [Semantic V### Fixed+- **Reference-code passage windows preserve their match score.** `xc.py`+ could select relevant source and then discard its matches while aligning+ the excerpt to line boundaries, including on long source lines. Line+ alignment now keeps the selected term score or falls back to the bounded+ original window. Window scoring also uses original-source offsets when+ Unicode lowercasing expands characters. This affects the fallback when no+ matching symbol is available and explicit `--no-symbol` output.++ - **`xc.py --mode hybrid` keeps BM25 results when the optional vector arm has+ a transport failure.** Connection failures, read timeouts, and interrupted+ HTTP responses during semantic mapping discovery or search now take the+ existing BM25-only fallback instead of aborting and discarding valid hits.+ Primary BM25 failures still exit with an error. Standalone `--mode semantic`+ now reports mapping/search HTTP and transport failures as errors (exit `2`)+ instead of treating failed requests as empty search results (exit `1`).++ - **`xc.py --json` emits JSON for empty search results.** Previously, a query+ with no hits printed the human-readable no-match message before reaching the+ JSON output branch, breaking callers that parse stdout. Empty results now+ preserve the JSON response and still exit `1`; matching results exit `0`, and+ a corpus with no live indices retains its distinct exit `3` diagnostic.++ ### Documentation++ - **`llms.txt` proposals, revised and fact-checked**+ (`docs/research/llms-txt-2026-09/proposals/`). A proposed `llms.txt`, an+ `llms-install.md` and ten paste-ready install prompts, following the study's+ twelve rules: three complete entry paths (shell, MCP-only, HTTP-only), the+ lexical-by-default correction first, per-client MCP registration with the+ key, a verify line after every step, and a feedback ask that is optional+ and needs no git. Every quote is re-fetched (259 claims, 258 confirmed, the+ one failure removed) and the XERJ commands the proposals print were run on+ Linux against v1.0.0-rc.74, except the ones the record's own "Not run" list+ names — the `curl | sh` installer, macOS and Windows, `--embed-mode neural`,+ `claude mcp add --scope local` and the per-client registration lines, which+ are quoted from each client's own documentation with a fact-check id. Two product findings came out of it:+ `xerj feedback --open-pr` branches, commits and pushes in whatever+ repository it is run from, and `xerj init` writes an MCP entry without+ `XERJ_AUTH`, so against a default node every tool call returns 401. The+ live `landing/llms.txt` is unchanged; the report's ship checklist says what+ has to exist first.++ - **ROADMAP: the zero-token direction, with every status checked against the+ tree** ([#941](https://github.com/xerj-org/xerj/issues/941)). A new roadmap+ section and [docs/ZERO_TOKEN_DIRECTION.md](./docs/ZERO_TOKEN_DIRECTION.md)+ lay out judged search, share links and a guest reading room, mail ingest,+ semantic detections, a real object-storage backend, a block index mode for+ logs, user-code ingest plugins and a corpus hub of signed packs — and say+ plainly what exists today: `S3Backend` is a local-directory simulation and+ `storage.backend = "s3"` refuses to start on purpose, `_watcher` stores+ watches and never evaluates them, alert rules have schemas and no+ evaluator, `xerj-logs` has no caller, and there is no wasmtime backend in+ the tree. The measuring behind it+ ([benchmarks/neural-path-triage/](./benchmarks/neural-path-triage/)) filed+ four defects with literal reproductions:+ [#937](https://github.com/xerj-org/xerj/issues/937) a declared analyzer+ stops applying at flush, [#938](https://github.com/xerj-org/xerj/issues/938)+ neural ingest keeps ~3.4 of 32 threads busy,+ [#939](https://github.com/xerj-org/xerj/issues/939) `semantic` over+ multi-passage documents is an exact scan that copies every `_source`+ (~410 ms p50 on 5,183 documents, forward pass ~14 ms), and+ [#940](https://github.com/xerj-org/xerj/issues/940) tied RRF scores change+ order across a restart. All four stay open; nothing is fixed by this entry.+## [1.0.0-rc.74] - 2026-09-08### Addeddiff --cc tools/xerj-code/README.mdindex 22e9cb966,19fdd1197..38da24646--- a/tools/xerj-code/README.md+++ b/tools/xerj-code/README.md@@@ -88,19 -88,28 +88,40 @@@ already knows, it is overhead. `SKILL.m[`docs/case-studies/reference-coding/`](../../docs/case-studies/reference-coding/)carry the measured numbers, including the cases where this loses.+## Passage windows++When no matching symbol is available, or with `--no-symbol`, `xc.py` selects a+window of source around query terms, capped at 800 characters by default.+It aligns the window to whole lines only when doing so preserves+the selected window's term score; a partial line is preferable to dropping the+match. Window offsets refer to the original source, including when Unicode+lowercasing changes its length. `--full N` changes the character limit, and+`--full 0` keeps the explicit file-head mode.++ ## Hybrid retrieval fallback++ With `--mode hybrid`, the optional vector arm falls back to the already+ retrieved BM25 passages if its mapping lookup or search has a transport error:+ connection failures, read timeouts, and interrupted HTTP responses. Prose and+ MEATL output label the fallback as `BM25 only`; `--json` keeps its existing hits+ format. A failed primary BM25 search still exits with an error. In standalone+ `--mode semantic`, mapping and search HTTP/transport failures exit `2`, rather+ than reporting no match; there are no BM25 results to fall back to.++ ## Machine-readable retrieval++ Pass `--json` to return the search result as JSON, including when no passages+ match. Check both stdout and the exit status: matching results exit `0`, while+ an empty `hits.hits` array exits `1`. A corpus with no live indices still exits+ `3` with a diagnostic on stderr; it is not an empty search result.+## Tests```shtools/xerj-code/tests/test_xc_corpus.sh # offline; local git fixtures over file://+python3 tools/xerj-code/tests/test_passage_window.py # offline passage extraction regressions+++ python3 tools/xerj-code/tests/test_hybrid_fallback.py # offline; simulated HTTP failures++ python3 tools/xerj-code/tests/test_state_ledger.py # offline; mocked HTTP responses```