AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(xerj-autoindex): a retrying request announces itself (#1183 item 1)
xerj-org52bdc6d6e6ce
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4b621c6bf..8c160eb5a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Fixed++- **A request retrying after a transport error or a 5xx now says so on stderr** (part of issue+ [#1183](https://github.com/xerj-org/xerj/issues/1183)) — the autoindex client's retry envelope+ is bounded (six attempts at a 300 s request timeout each, ≈31 min worst case) but was+ completely silent on everything except HTTP 429, which is how the #1183 incident read as a+ deadlock: a finalize-catalog request against a memory-pressured node spent eight minutes inside+ that envelope with zero output, and since `reqwest::blocking` parks the caller on a oneshot,+ every thread's `/proc` wchan read `futex_do_wait` — indistinguishable from a lock cycle at that+ granularity. Diagnosis in the issue: it was never a deadlock; the silence was the defect. A+ loading run (announce on) now prints `autoindex: search failed: …; retrying (attempt 2 of 6,+ … s in this request so far)` on the first failure and at most once per 30 s after, sharing the+ 429 notice's clock so alternating failures cannot double-announce; probes, MCP clients and+ `--quiet` stay silent. The server-side stall root cause (the reference node running at its+ memory breaker during large rebuilds) remains open under #1183 and the #1122 memory-ceiling+ class.+## [1.0.0-rc.86] - 2026-10-06### Securitydiff --git a/ROADMAP.md b/ROADMAP.mdindex 04c9d28dd..f42b012aa 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -181,7 +181,7 @@ and the console review's), the #1062 wrong-and-confident detection gate, the`xerj-decide-v1` Hugging Face publication (operator credentials), and the GAbar itself — *The road to v1.0.0 GA* below.-**Open defects.** Eleven, all live on the tracker at rc.86 cut time — the+**Open defects.** Ten, all live on the tracker — theshortlist the release-notes gate checks. From the rc.80 gate runs:[#1091](https://github.com/xerj-org/xerj/issues/1091) (hybrid first stage13–26 s/query on the 57,638-doc FiQA index) and@@ -189,12 +189,7 @@ shortlist the release-notes gate checks. From the rc.80 gate runs:flywheel write-back freezes once the history index has BM25 support).From the rc.80 console knowledge-surface review:[#1100](https://github.com/xerj-org/xerj/issues/1100) (second-brain-belief-time frame rendered on non-note corpora). From the corpus-hub G7-work: [#1147](https://github.com/xerj-org/xerj/issues/1147) (corpus-index-ingest throughput caps corpus builds at ~10⁵ records/node-day, measured —-the osv-ecosystems corpus is deferred on exactly this number; the-wire-side coalescing half is in flight as PR-[#1195](https://github.com/xerj-org/xerj/pull/1195)). From the rc.81/82+belief-time frame rendered on non-note corpora). From the rc.81/82windows: [#1158](https://github.com/xerj-org/xerj/issues/1158) (thebuild-time half — the query-time half shipped in rc.86) and[#1122](https://github.com/xerj-org/xerj/issues/1122) (reopened@@ -210,7 +205,10 @@ wrong-and-confident detection gate, unmeasured),[#1110](https://github.com/xerj-org/xerj/issues/1110) (thegeneralization proof: second-domain pack + fresh-machine replay) and[#1183](https://github.com/xerj-org/xerj/issues/1183) (corpus index-finalize-catalog deadlocking all 297 threads on a fresh build).+finalize-catalog reported as an all-thread deadlock: diagnosed as a bounded+but silent client retry envelope — the silence is fixed; the server-side+stall under memory pressure stays open, tracked with the #1122+memory-ceiling class).**Closed inside the rc.86 window and recorded in the CHANGELOG, notcarried here:** #1109, #1189, #1190, #1186, #1146 and #1092 (PRs[#1194](https://github.com/xerj-org/xerj/pull/1194),@@ -218,6 +216,11 @@ carried here:** #1109, #1189, #1190, #1186, #1146 and #1092 (PRs[#1188](https://github.com/xerj-org/xerj/pull/1188),[#1177](https://github.com/xerj-org/xerj/pull/1177) and[#1184](https://github.com/xerj-org/xerj/pull/1184)). **Closed inside+the rc.87 window so far:** #1147 (corpus-index ingest throughput — the+per-file bulk round-trip ceiling, measured at ~10⁵ records/node-day;+fixed by PR [#1195](https://github.com/xerj-org/xerj/pull/1195), merged+2026-10-06 just past the rc.86 tag; its CHANGELOG entry lands with the+rc.87 notes). **Closed insidethe rc.83 window and recorded in the CHANGELOG, notcarried here:** #1136 (stale index generations surviving `--fresh` swaps —a state file with `index_prefix=null` queried all of them; fixed by PRdiff --git a/engine/crates/xerj-autoindex/src/esclient.rs b/engine/crates/xerj-autoindex/src/esclient.rsindex 08301f8eb..72b964452 100644--- a/engine/crates/xerj-autoindex/src/esclient.rs+++ b/engine/crates/xerj-autoindex/src/esclient.rs@@ -1207,6 +1207,14 @@ impl Es {if failures >= MAX_ATTEMPTS {return Err(e);}+ self.failure_notice(+ &mut last_notice,+ what,+ &e,+ failures,+ MAX_ATTEMPTS,+ started.elapsed(),+ );}}self.backoff(delay);@@ -1235,6 +1243,50 @@ impl Es {*last = Some(Instant::now());}+ /// Say on stderr that a request is retrying after a transport error or a+ /// 5xx — the first failure immediately, then at most once per+ /// [`NOTICE_EVERY`] per request, sharing [`Self::throttle_notice`]'s clock+ /// so an alternating 429/timeout sequence cannot double-announce. Same+ /// announce gate: loading runs speak, probes and `--quiet` stay silent.+ ///+ /// Why the first failure is worth a line: the retry envelope here is+ /// bounded (six attempts at the client's 300 s request timeout ≈ 31 min+ /// worst case) but was SILENT, and #1183 reported exactly that silence as+ /// a deadlock — the caller parks on `reqwest::blocking`'s oneshot, which+ /// `/proc/<pid>/task/*/wchan` reports as `futex_do_wait`, so eight+ /// minutes of no output and zero CPU were indistinguishable from a lock+ /// cycle. A line that names the request, the reason, and the attempt+ /// count turns "stalled" into "retrying, bounded". Returns whether a+ /// line was printed, so the gate is testable without capturing stderr.+ fn failure_notice(+ &self,+ last: &mut Option<Instant>,+ what: &str,+ error: &anyhow::Error,+ attempt: usize,+ max_attempts: usize,+ waited: Duration,+ ) -> bool {+ if !self.admission.announces() || matches!(last, Some(t) if t.elapsed() < NOTICE_EVERY) {+ return false;+ }+ // One line, capped: transport error chains can run long and the point+ // is the request and the bound, not the whole diagnostic.+ let mut reason = format!("{error:#}").replace('\n', " ");+ let cut = reason+ .char_indices()+ .nth(120)+ .map_or(reason.len(), |(i, _)| i);+ reason.truncate(cut);+ eprintln!(+ "autoindex: {what} failed: {reason}; retrying (attempt {attempt} of {max_attempts}, \+ {} s in this request so far)",+ waited.as_secs()+ );+ *last = Some(Instant::now());+ true+ }+/// Sleep between attempts, recording how long was actually slept.////// One function so that moving the CALL SITE cannot separate recording from@@ -2259,6 +2311,84 @@ mod tests {server.join().unwrap();}+ /// #1183: a retrying request must say so, or its bounded envelope reads+ /// as a deadlock. The gate is the logic — first failure announces,+ /// the next line waits for NOTICE_EVERY, a silent client never announces+ /// — so it is pinned directly on the returned flag; the print itself is+ /// a one-liner the flag stands for.+ #[test]+ fn failure_notices_announce_immediately_then_hold_the_cadence() {+ let mk = |announce| {+ Es::with_bulk_policy(+ "http://127.0.0.1:1",+ None,+ Duration::from_millis(100),+ Duration::from_millis(10),+ Duration::from_millis(20),+ )+ .unwrap()+ .with_bulk_concurrency(1, announce)+ };+ let error = anyhow::Error::msg("connection closed before message completed");+ // a quiet client (probe, --quiet) never announces and never marks+ let quiet = mk(false);+ let mut last = None;+ assert!(!quiet.failure_notice(&mut last, "search", &error, 1, 6, Duration::from_secs(0)));+ assert!(last.is_none(), "a refused notice must not mark the clock");+ // a loading run announces the first failure at once…+ let loud = mk(true);+ assert!(loud.failure_notice(&mut last, "search", &error, 1, 6, Duration::from_secs(310)));+ let first = last.expect("an announced notice marks the clock");+ // …and holds the cadence for the immediate next one+ assert!(!loud.failure_notice(&mut last, "search", &error, 2, 6, Duration::from_secs(620)));+ assert_eq!(+ last,+ Some(first),+ "a held notice must not re-mark the clock"+ );+ }++ /// The wire-level shape of the same contract: a connection dropped before+ /// a response is a transport failure, the retry loop announces (loading+ /// client) and re-offers, and the second attempt succeeds — the run is+ /// bounded AND visibly so, never a silent stall.+ #[test]+ fn a_dropped_connection_is_retried_after_announcing() {+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();+ let address = listener.local_addr().unwrap();+ let requests = Arc::new(Mutex::new(Vec::new()));+ let requests_server = requests.clone();+ let server = std::thread::spawn(move || {+ // first connection: read the request, answer nothing, close+ let (mut first, _) = listener.accept().unwrap();+ requests_server+ .lock()+ .unwrap()+ .push(read_request(&mut first));+ drop(first);+ // second connection: the retried request, answered+ let (mut second, _) = listener.accept().unwrap();+ requests_server+ .lock()+ .unwrap()+ .push(read_request(&mut second));+ respond_json(&mut second, br#"{"deleted":0,"failures":[]}"#.as_slice());+ });+ let es = Es::with_bulk_policy(+ &format!("http://{address}"),+ None,+ Duration::from_millis(100),+ Duration::from_millis(10),+ Duration::from_millis(20),+ )+ .unwrap()+ .with_bulk_concurrency(1, true);+ es.delete_by_query("data", &serde_json::json!({"term": {"ax_file": "key"}}))+ .unwrap();+ server.join().unwrap();+ assert_eq!(requests.lock().unwrap().len(), 2, "one drop, one retry");+ }+#[test]fn all_timeout_retry_path_is_bounded_to_six_attempts_without_final_sleep() {let listener = TcpListener::bind("127.0.0.1:0").unwrap();