Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(xerj-autoindex): a retrying request announces itself (#1183 item 1)

xerj-org52bdc6d6e6ce
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4b621c6bf..8c160eb5a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Fixed
+
+- **A request retrying after a transport error or a 5xx now says so on stderr** (part of issue
+ [#1183](https://github.com/xerj-org/xerj/issues/1183)) — the autoindex client's retry envelope
+ is bounded (six attempts at a 300 s request timeout each, ≈31 min worst case) but was
+ completely silent on everything except HTTP 429, which is how the #1183 incident read as a
+ deadlock: a finalize-catalog request against a memory-pressured node spent eight minutes inside
+ that envelope with zero output, and since `reqwest::blocking` parks the caller on a oneshot,
+ every thread's `/proc` wchan read `futex_do_wait` — indistinguishable from a lock cycle at that
+ granularity. Diagnosis in the issue: it was never a deadlock; the silence was the defect. A
+ loading run (announce on) now prints `autoindex: search failed: …; retrying (attempt 2 of 6,
+ … s in this request so far)` on the first failure and at most once per 30 s after, sharing the
+ 429 notice's clock so alternating failures cannot double-announce; probes, MCP clients and
+ `--quiet` stay silent. The server-side stall root cause (the reference node running at its
+ memory breaker during large rebuilds) remains open under #1183 and the #1122 memory-ceiling
+ class.
+
## [1.0.0-rc.86] - 2026-10-06
### Security
diff --git a/ROADMAP.md b/ROADMAP.md
index 04c9d28dd..f42b012aa 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -181,7 +181,7 @@ and the console review's), the #1062 wrong-and-confident detection gate, the
`xerj-decide-v1` Hugging Face publication (operator credentials), and the GA
bar itself — *The road to v1.0.0 GA* below.
-**Open defects.** Eleven, all live on the tracker at rc.86 cut time — the
+**Open defects.** Ten, all live on the tracker — the
shortlist the release-notes gate checks. From the rc.80 gate runs:
[#1091](https://github.com/xerj-org/xerj/issues/1091) (hybrid first stage
13–26 s/query on the 57,638-doc FiQA index) and
@@ -189,12 +189,7 @@ shortlist the release-notes gate checks. From the rc.80 gate runs:
flywheel write-back freezes once the history index has BM25 support).
From the rc.80 console knowledge-surface review:
[#1100](https://github.com/xerj-org/xerj/issues/1100) (second-brain
-belief-time frame rendered on non-note corpora). From the corpus-hub G7
-work: [#1147](https://github.com/xerj-org/xerj/issues/1147) (corpus-index
-ingest throughput caps corpus builds at ~10⁵ records/node-day, measured —
-the osv-ecosystems corpus is deferred on exactly this number; the
-wire-side coalescing half is in flight as PR
-[#1195](https://github.com/xerj-org/xerj/pull/1195)). From the rc.81/82
+belief-time frame rendered on non-note corpora). From the rc.81/82
windows: [#1158](https://github.com/xerj-org/xerj/issues/1158) (the
build-time half — the query-time half shipped in rc.86) and
[#1122](https://github.com/xerj-org/xerj/issues/1122) (reopened
@@ -210,7 +205,10 @@ wrong-and-confident detection gate, unmeasured),
[#1110](https://github.com/xerj-org/xerj/issues/1110) (the
generalization proof: second-domain pack + fresh-machine replay) and
[#1183](https://github.com/xerj-org/xerj/issues/1183) (corpus index
-finalize-catalog deadlocking all 297 threads on a fresh build).
+finalize-catalog reported as an all-thread deadlock: diagnosed as a bounded
+but silent client retry envelope — the silence is fixed; the server-side
+stall under memory pressure stays open, tracked with the #1122
+memory-ceiling class).
**Closed inside the rc.86 window and recorded in the CHANGELOG, not
carried here:** #1109, #1189, #1190, #1186, #1146 and #1092 (PRs
[#1194](https://github.com/xerj-org/xerj/pull/1194),
@@ -218,6 +216,11 @@ carried here:** #1109, #1189, #1190, #1186, #1146 and #1092 (PRs
[#1188](https://github.com/xerj-org/xerj/pull/1188),
[#1177](https://github.com/xerj-org/xerj/pull/1177) and
[#1184](https://github.com/xerj-org/xerj/pull/1184)). **Closed inside
+the rc.87 window so far:** #1147 (corpus-index ingest throughput — the
+per-file bulk round-trip ceiling, measured at ~10⁵ records/node-day;
+fixed by PR [#1195](https://github.com/xerj-org/xerj/pull/1195), merged
+2026-10-06 just past the rc.86 tag; its CHANGELOG entry lands with the
+rc.87 notes). **Closed inside
the rc.83 window and recorded in the CHANGELOG, not
carried here:** #1136 (stale index generations surviving `--fresh` swaps —
a state file with `index_prefix=null` queried all of them; fixed by PR
diff --git a/engine/crates/xerj-autoindex/src/esclient.rs b/engine/crates/xerj-autoindex/src/esclient.rs
index 08301f8eb..72b964452 100644
--- a/engine/crates/xerj-autoindex/src/esclient.rs
+++ b/engine/crates/xerj-autoindex/src/esclient.rs
@@ -1207,6 +1207,14 @@ impl Es {
if failures >= MAX_ATTEMPTS {
return Err(e);
}
+ self.failure_notice(
+ &mut last_notice,
+ what,
+ &e,
+ failures,
+ MAX_ATTEMPTS,
+ started.elapsed(),
+ );
}
}
self.backoff(delay);
@@ -1235,6 +1243,50 @@ impl Es {
*last = Some(Instant::now());
}
+ /// Say on stderr that a request is retrying after a transport error or a
+ /// 5xx — the first failure immediately, then at most once per
+ /// [`NOTICE_EVERY`] per request, sharing [`Self::throttle_notice`]'s clock
+ /// so an alternating 429/timeout sequence cannot double-announce. Same
+ /// announce gate: loading runs speak, probes and `--quiet` stay silent.
+ ///
+ /// Why the first failure is worth a line: the retry envelope here is
+ /// bounded (six attempts at the client's 300 s request timeout ≈ 31 min
+ /// worst case) but was SILENT, and #1183 reported exactly that silence as
+ /// a deadlock — the caller parks on `reqwest::blocking`'s oneshot, which
+ /// `/proc/<pid>/task/*/wchan` reports as `futex_do_wait`, so eight
+ /// minutes of no output and zero CPU were indistinguishable from a lock
+ /// cycle. A line that names the request, the reason, and the attempt
+ /// count turns "stalled" into "retrying, bounded". Returns whether a
+ /// line was printed, so the gate is testable without capturing stderr.
+ fn failure_notice(
+ &self,
+ last: &mut Option<Instant>,
+ what: &str,
+ error: &anyhow::Error,
+ attempt: usize,
+ max_attempts: usize,
+ waited: Duration,
+ ) -> bool {
+ if !self.admission.announces() || matches!(last, Some(t) if t.elapsed() < NOTICE_EVERY) {
+ return false;
+ }
+ // One line, capped: transport error chains can run long and the point
+ // is the request and the bound, not the whole diagnostic.
+ let mut reason = format!("{error:#}").replace('\n', " ");
+ let cut = reason
+ .char_indices()
+ .nth(120)
+ .map_or(reason.len(), |(i, _)| i);
+ reason.truncate(cut);
+ eprintln!(
+ "autoindex: {what} failed: {reason}; retrying (attempt {attempt} of {max_attempts}, \
+ {} s in this request so far)",
+ waited.as_secs()
+ );
+ *last = Some(Instant::now());
+ true
+ }
+
/// Sleep between attempts, recording how long was actually slept.
///
/// One function so that moving the CALL SITE cannot separate recording from
@@ -2259,6 +2311,84 @@ mod tests {
server.join().unwrap();
}
+ /// #1183: a retrying request must say so, or its bounded envelope reads
+ /// as a deadlock. The gate is the logic — first failure announces,
+ /// the next line waits for NOTICE_EVERY, a silent client never announces
+ /// — so it is pinned directly on the returned flag; the print itself is
+ /// a one-liner the flag stands for.
+ #[test]
+ fn failure_notices_announce_immediately_then_hold_the_cadence() {
+ let mk = |announce| {
+ Es::with_bulk_policy(
+ "http://127.0.0.1:1",
+ None,
+ Duration::from_millis(100),
+ Duration::from_millis(10),
+ Duration::from_millis(20),
+ )
+ .unwrap()
+ .with_bulk_concurrency(1, announce)
+ };
+ let error = anyhow::Error::msg("connection closed before message completed");
+ // a quiet client (probe, --quiet) never announces and never marks
+ let quiet = mk(false);
+ let mut last = None;
+ assert!(!quiet.failure_notice(&mut last, "search", &error, 1, 6, Duration::from_secs(0)));
+ assert!(last.is_none(), "a refused notice must not mark the clock");
+ // a loading run announces the first failure at once…
+ let loud = mk(true);
+ assert!(loud.failure_notice(&mut last, "search", &error, 1, 6, Duration::from_secs(310)));
+ let first = last.expect("an announced notice marks the clock");
+ // …and holds the cadence for the immediate next one
+ assert!(!loud.failure_notice(&mut last, "search", &error, 2, 6, Duration::from_secs(620)));
+ assert_eq!(
+ last,
+ Some(first),
+ "a held notice must not re-mark the clock"
+ );
+ }
+
+ /// The wire-level shape of the same contract: a connection dropped before
+ /// a response is a transport failure, the retry loop announces (loading
+ /// client) and re-offers, and the second attempt succeeds — the run is
+ /// bounded AND visibly so, never a silent stall.
+ #[test]
+ fn a_dropped_connection_is_retried_after_announcing() {
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ let address = listener.local_addr().unwrap();
+ let requests = Arc::new(Mutex::new(Vec::new()));
+ let requests_server = requests.clone();
+ let server = std::thread::spawn(move || {
+ // first connection: read the request, answer nothing, close
+ let (mut first, _) = listener.accept().unwrap();
+ requests_server
+ .lock()
+ .unwrap()
+ .push(read_request(&mut first));
+ drop(first);
+ // second connection: the retried request, answered
+ let (mut second, _) = listener.accept().unwrap();
+ requests_server
+ .lock()
+ .unwrap()
+ .push(read_request(&mut second));
+ respond_json(&mut second, br#"{"deleted":0,"failures":[]}"#.as_slice());
+ });
+ let es = Es::with_bulk_policy(
+ &format!("http://{address}"),
+ None,
+ Duration::from_millis(100),
+ Duration::from_millis(10),
+ Duration::from_millis(20),
+ )
+ .unwrap()
+ .with_bulk_concurrency(1, true);
+ es.delete_by_query("data", &serde_json::json!({"term": {"ax_file": "key"}}))
+ .unwrap();
+ server.join().unwrap();
+ assert_eq!(requests.lock().unwrap().len(), 2, "one drop, one retry");
+ }
+
#[test]
fn all_timeout_retry_path_is_bounded_to_six_attempts_without_final_sleep() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();