Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge origin/main into codex/xc-passage-match-preservation

xerj-orgab4e07c68e00
diff --cc .github/workflows/ci.yml
index 4048e9e65,1e4f17d19..91114d6b6
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@@ -96,9 -96,9 +96,12 @@@ jobs
- name: Corpus manifest round-trip
run: tools/xerj-code/tests/test_xc_corpus.sh
+ - name: Matching passage window regressions
+ run: python3 tools/xerj-code/tests/test_passage_window.py
+
+ - name: Retrieval output and state-ledger regressions
+ run: python3 tools/xerj-code/tests/test_state_ledger.py
+
- name: Vetted hub manifests validate
run: python3 tools/xerj-code/tests/validate_manifest.py --hub tools/xerj-code/hub/*.json
diff --cc CHANGELOG.md
index e31eb1b20,8f7b12f1a..4a222d18d
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -28,14 -28,62 +28,70 @@@ and this project adheres to [Semantic V
### Fixed
+- **Reference-code passage windows preserve their match score.** `xc.py`
+ could select relevant source and then discard its matches while aligning
+ the excerpt to line boundaries, including on long source lines. Line
+ alignment now keeps the selected term score or falls back to the bounded
+ original window. Window scoring also uses original-source offsets when
+ Unicode lowercasing expands characters. This affects the fallback when no
+ matching symbol is available and explicit `--no-symbol` output.
+
+ - **`xc.py --mode hybrid` keeps BM25 results when the optional vector arm has
+ a transport failure.** Connection failures, read timeouts, and interrupted
+ HTTP responses during semantic mapping discovery or search now take the
+ existing BM25-only fallback instead of aborting and discarding valid hits.
+ Primary BM25 failures still exit with an error. Standalone `--mode semantic`
+ now reports mapping/search HTTP and transport failures as errors (exit `2`)
+ instead of treating failed requests as empty search results (exit `1`).
+
+ - **`xc.py --json` emits JSON for empty search results.** Previously, a query
+ with no hits printed the human-readable no-match message before reaching the
+ JSON output branch, breaking callers that parse stdout. Empty results now
+ preserve the JSON response and still exit `1`; matching results exit `0`, and
+ a corpus with no live indices retains its distinct exit `3` diagnostic.
+
+ ### Documentation
+
+ - **`llms.txt` proposals, revised and fact-checked**
+ (`docs/research/llms-txt-2026-09/proposals/`). A proposed `llms.txt`, an
+ `llms-install.md` and ten paste-ready install prompts, following the study's
+ twelve rules: three complete entry paths (shell, MCP-only, HTTP-only), the
+ lexical-by-default correction first, per-client MCP registration with the
+ key, a verify line after every step, and a feedback ask that is optional
+ and needs no git. Every quote is re-fetched (259 claims, 258 confirmed, the
+ one failure removed) and the XERJ commands the proposals print were run on
+ Linux against v1.0.0-rc.74, except the ones the record's own "Not run" list
+ names — the `curl | sh` installer, macOS and Windows, `--embed-mode neural`,
+ `claude mcp add --scope local` and the per-client registration lines, which
+ are quoted from each client's own documentation with a fact-check id. Two product findings came out of it:
+ `xerj feedback --open-pr` branches, commits and pushes in whatever
+ repository it is run from, and `xerj init` writes an MCP entry without
+ `XERJ_AUTH`, so against a default node every tool call returns 401. The
+ live `landing/llms.txt` is unchanged; the report's ship checklist says what
+ has to exist first.
+
+ - **ROADMAP: the zero-token direction, with every status checked against the
+ tree** ([#941](https://github.com/xerj-org/xerj/issues/941)). A new roadmap
+ section and [docs/ZERO_TOKEN_DIRECTION.md](./docs/ZERO_TOKEN_DIRECTION.md)
+ lay out judged search, share links and a guest reading room, mail ingest,
+ semantic detections, a real object-storage backend, a block index mode for
+ logs, user-code ingest plugins and a corpus hub of signed packs — and say
+ plainly what exists today: `S3Backend` is a local-directory simulation and
+ `storage.backend = "s3"` refuses to start on purpose, `_watcher` stores
+ watches and never evaluates them, alert rules have schemas and no
+ evaluator, `xerj-logs` has no caller, and there is no wasmtime backend in
+ the tree. The measuring behind it
+ ([benchmarks/neural-path-triage/](./benchmarks/neural-path-triage/)) filed
+ four defects with literal reproductions:
+ [#937](https://github.com/xerj-org/xerj/issues/937) a declared analyzer
+ stops applying at flush, [#938](https://github.com/xerj-org/xerj/issues/938)
+ neural ingest keeps ~3.4 of 32 threads busy,
+ [#939](https://github.com/xerj-org/xerj/issues/939) `semantic` over
+ multi-passage documents is an exact scan that copies every `_source`
+ (~410 ms p50 on 5,183 documents, forward pass ~14 ms), and
+ [#940](https://github.com/xerj-org/xerj/issues/940) tied RRF scores change
+ order across a restart. All four stay open; nothing is fixed by this entry.
+
## [1.0.0-rc.74] - 2026-09-08
### Added
diff --cc tools/xerj-code/README.md
index 22e9cb966,19fdd1197..38da24646
--- a/tools/xerj-code/README.md
+++ b/tools/xerj-code/README.md
@@@ -88,19 -88,28 +88,40 @@@ already knows, it is overhead. `SKILL.m
[`docs/case-studies/reference-coding/`](../../docs/case-studies/reference-coding/)
carry the measured numbers, including the cases where this loses.
+## Passage windows
+
+When no matching symbol is available, or with `--no-symbol`, `xc.py` selects a
+window of source around query terms, capped at 800 characters by default.
+It aligns the window to whole lines only when doing so preserves
+the selected window's term score; a partial line is preferable to dropping the
+match. Window offsets refer to the original source, including when Unicode
+lowercasing changes its length. `--full N` changes the character limit, and
+`--full 0` keeps the explicit file-head mode.
+
+ ## Hybrid retrieval fallback
+
+ With `--mode hybrid`, the optional vector arm falls back to the already
+ retrieved BM25 passages if its mapping lookup or search has a transport error:
+ connection failures, read timeouts, and interrupted HTTP responses. Prose and
+ MEATL output label the fallback as `BM25 only`; `--json` keeps its existing hits
+ format. A failed primary BM25 search still exits with an error. In standalone
+ `--mode semantic`, mapping and search HTTP/transport failures exit `2`, rather
+ than reporting no match; there are no BM25 results to fall back to.
+
+ ## Machine-readable retrieval
+
+ Pass `--json` to return the search result as JSON, including when no passages
+ match. Check both stdout and the exit status: matching results exit `0`, while
+ an empty `hits.hits` array exits `1`. A corpus with no live indices still exits
+ `3` with a diagnostic on stderr; it is not an empty search result.
+
## Tests
```sh
tools/xerj-code/tests/test_xc_corpus.sh # offline; local git fixtures over file://
+python3 tools/xerj-code/tests/test_passage_window.py # offline passage extraction regressions
++
+ python3 tools/xerj-code/tests/test_hybrid_fallback.py # offline; simulated HTTP failures
+
+ python3 tools/xerj-code/tests/test_state_ledger.py # offline; mocked HTTP responses
```