Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge branch 'fix/silent-retry-envelope' into fix/json-text-passage

xerj-orgbda0d561fbef
diff --cc CHANGELOG.md
index 690564ff7,3d87299dd..9939fef5c
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -28,8 -28,21 +28,23 @@@ and this project adheres to [Semantic V
machine traffic in the audit log (#1109), so `xerj gain` still measures only the user's
queries. The finalize-catalog deadlock that produces the dangling segment (item 1 of #1183)
remains open, tracked in the same issue.
+ - **A request retrying after a transport error or a 5xx now says so on stderr** (part of issue
+ [#1183](https://github.com/xerj-org/xerj/issues/1183)) — the autoindex client's retry envelope
+ is bounded (six attempts at a 300 s request timeout each, ≈31 min worst case) but was
+ completely silent on everything except HTTP 429, which is how the #1183 incident read as a
+ deadlock: a finalize-catalog request against a memory-pressured node spent eight minutes inside
+ that envelope with zero output, and since `reqwest::blocking` parks the caller on a oneshot,
+ every thread's `/proc` wchan read `futex_do_wait` — indistinguishable from a lock cycle at that
+ granularity. Diagnosis in the issue: it was never a deadlock; the silence was the defect. A
+ loading run (announce on) now prints `autoindex: search failed: …; retrying (attempt 2 of 6,
+ … s in this request so far)` on the first failure and at most once per 30 s after, sharing the
+ 429 notice's clock so alternating failures cannot double-announce; probes, MCP clients and
+ `--quiet` stay silent. The server-side stall root cause (the reference node running at its
+ memory breaker during large rebuilds) remains open under #1183 and the #1122 memory-ceiling
+ class.
+- **A JSON record with no prose now carries a synthesized `text` passage, and a dataset that maps no text-searchable field says so at plan time** (the build-time half of issue [#1158](https://github.com/xerj-org/xerj/issues/1158); the query-time half — falling back to an index's own text-typed fields — shipped in rc.86). Raw-JSON corpora indexed as typed records (ids, enums, numbers, dates) were invisible to `xerj code` passage search: 36,263 GHSA advisories indexed to 193k records that answered `No passage matches` while the same index returned 10,000+ hits over its typed fields. Now a record whose flattened fields carry no prose-like string (≥ 24 chars and ≥ 4 tokens — below that a string is an id, an enum or a date, and maps `keyword`) gains a `text` passage rendered from its own fields (`key: value` lines, values capped at 256 chars, the passage at 4,096, scalar arrays joined so identifiers stay matchable); a record that already carries prose is left alone — the query side already searches it, and a rendered key/value soup would only outrank the real prose — and a record that owns a field named `text` keeps it untouched. Separately, any dataset whose sampled mapping contains no text-searchable field (a csv of enums, a keyword-only XML set) now warns at plan time: `dataset 'x' maps no text-searchable field — 'xerj code' passage search will find nothing in it (sampled fields: …)`, through the progress surface so `--quiet` stays silent. Scoped to the JSON family (whole-file and JSONL) — the measured class is raw-JSON mirrors; the same helper can ride the csv/yaml/xml data sites when a real corpus needs it. Existing state dirs re-provision their JSON datasets once to gain the field; every already-built corpus keeps working, and the 100 hub corpora are unaffected (their graded snapshots map standard fields).
+
+
## [1.0.0-rc.86] - 2026-10-06
### Security