Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into fix/json-text-passage

xerj-org3fe3b4a78647
diff --cc CHANGELOG.md
index b88fade2b,60dfe3510..690564ff7
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -9,7 -9,25 +9,26 @@@ and this project adheres to [Semantic V
### Fixed
+ - **`xerj corpus index` now proves the store answers a search before it says "searchable"** (issue
+ [#1183](https://github.com/xerj-org/xerj/issues/1183)) — a resume over an index left with a
+ dangling segment by a killed finalize printed `corpus 'xerj-search' searchable: 642 records`
+ and exited 0 while the store answered every `_search` — and the salvage path's own
+ `delete_by_query` — with `store_exception: Segment … not found`. `_count` is metadata-only and
+ returned 642 over that same store, so every verification built on counts certified a corpus
+ that could not be queried. Verification now has a third leg: one `size:0` `_search` over
+ exactly the glob `xerj code` reads (`{prefix}-*`), with the node's own reason surfaced when it
+ fails. In the build arm a build whose verification search fails is not verified — nothing is
+ switched, the existing index stays what `xerj code` serves, and the unsearchable build's own
+ indices are retired by exact name (an unsearchable index left under the namespace is what
+ poisoned the live one); the #367 salvage can no longer record a store that cannot be searched.
+ The update arm refuses to rewrite the ledger over a live corpus that cannot be searched, and
+ the legacy arm — the exact live failure — refuses to print any verdict at all, because one
+ unsearchable index anywhere under the wildcard makes "the corpus is queryable" untrue whatever
+ the counts said. The probe is a `size:0` search, which the server already classifies as
+ machine traffic in the audit log (#1109), so `xerj gain` still measures only the user's
+ queries. The finalize-catalog deadlock that produces the dangling segment (item 1 of #1183)
+ remains open, tracked in the same issue.
+- **A JSON record with no prose now carries a synthesized `text` passage, and a dataset that maps no text-searchable field says so at plan time** (the build-time half of issue [#1158](https://github.com/xerj-org/xerj/issues/1158); the query-time half — falling back to an index's own text-typed fields — shipped in rc.86). Raw-JSON corpora indexed as typed records (ids, enums, numbers, dates) were invisible to `xerj code` passage search: 36,263 GHSA advisories indexed to 193k records that answered `No passage matches` while the same index returned 10,000+ hits over its typed fields. Now a record whose flattened fields carry no prose-like string (≥ 24 chars and ≥ 4 tokens — below that a string is an id, an enum or a date, and maps `keyword`) gains a `text` passage rendered from its own fields (`key: value` lines, values capped at 256 chars, the passage at 4,096, scalar arrays joined so identifiers stay matchable); a record that already carries prose is left alone — the query side already searches it, and a rendered key/value soup would only outrank the real prose — and a record that owns a field named `text` keeps it untouched. Separately, any dataset whose sampled mapping contains no text-searchable field (a csv of enums, a keyword-only XML set) now warns at plan time: `dataset 'x' maps no text-searchable field — 'xerj code' passage search will find nothing in it (sampled fields: …)`, through the progress surface so `--quiet` stays silent. Scoped to the JSON family (whole-file and JSONL) — the measured class is raw-JSON mirrors; the same helper can ride the csv/yaml/xml data sites when a real corpus needs it. Existing state dirs re-provision their JSON datasets once to gain the field; every already-built corpus keeps working, and the 100 hub corpora are unaffected (their graded snapshots map standard fields).
## [1.0.0-rc.86] - 2026-10-06