AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(#202): correct two overclaims, and pin the snapshot recovery door
xerj-org2677014db7c2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f172f4453..76e5feed1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -52,8 +52,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0now boots **red** with that index unserved, where it previously came up greenwith an empty mapping. It joins the failed set[#206](https://github.com/xerj-org/xerj/issues/206) introduced, so every- surface that set feeds already reports it — measured on a two-index node with- one truncated `schema.json`:+ surface that set feeds already reports it — measured on a node holding two+ user indices, `victim` with a truncated `schema.json`:```GET /_cluster/health -> red, unassigned_primary_shards: 1@@ -66,15 +66,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0-> 503 no_shard_available_action_exceptioncarrying that same reasonGET /health/ready -> 200 "ready (degraded): 1 of 2 indices- serving, 1 failed to open"+ serving, 1 failed to open; see GET+ /_cluster/indices/failed"POST /healthy/_search -> 200```- Recovery is the three doors #206 added, both verified end to end here: repair- the file and `POST /_cluster/indices/failed/{index}/_retry` (503 while it is- still torn, `200 {"reopened": true}` once it is not, health back to green), or- restore the index from a snapshot, or `DELETE /{index}`. The node stays in- kubelet rotation as long as any index is still serving.+ The readiness counts above are the two-index test router's. A running node+ also holds its internal `.xerj_*` indices, so the same condition on the shipped+ binary reads `ready (degraded): 15 of 16 indices serving, 1 failed to open` —+ the 200 and its meaning are unchanged, only the totals move with what else the+ node holds.++ Recovery is the three doors #206 added. Two were measured end to end on the+ shipped binary: repair the file and+ `POST /_cluster/indices/failed/{index}/_retry` (503 while it is still torn,+ `200 {"reopened": true}` once it is not, health back to green), and+ `DELETE /{index}` (200, directory removed, health back to green). The third —+ restore the index from a snapshot — is the path this change touches (a+ successful restore now clears the recorded failure, which it has to or health+ would stay red after the repair worked); it is covered by an engine-level+ test rather than measured over HTTP. The node stays in kubelet rotation as+ long as any index is still serving.- **Concurrent sidecar writes can no longer manufacture the torn file the readernow refuses.** `write_file_atomic` staged every write indiff --git a/engine/crates/xerj-api/tests/corrupt_index_http.rs b/engine/crates/xerj-api/tests/corrupt_index_http.rsindex 5c615592c..090990609 100644--- a/engine/crates/xerj-api/tests/corrupt_index_http.rs+++ b/engine/crates/xerj-api/tests/corrupt_index_http.rs@@ -19,9 +19,13 @@//! side that stops a corrupt-metadata index from being visible, deletable and//! survivable on `:9200` fails here rather than in production.//!-//! Every case below quarantines an index by truncating its `schema.json` —-//! the #202 trigger — and then asserts the #206 contract over-//! `build_es_compat_router`.+//! Nine of the eleven cases below quarantine an index by truncating its+//! `schema.json` — the #202 trigger — and then assert the #206 contract over+//! `build_es_compat_router`. The other two are controls that pin the negative+//! half of that contract, and deliberately corrupt nothing:+//! `cluster_health_is_not_red_without_a_failed_index` (an intact node must not+//! be dragged red by the code that reports failures) and `an_empty_node_is_ready`+//! (a fresh pod with no indices at all must still join the Service).use axum::body::Body;use axum::http::{Request, StatusCode};diff --git a/engine/crates/xerj-engine/tests/corrupt_index_metadata.rs b/engine/crates/xerj-engine/tests/corrupt_index_metadata.rsindex aa2a45b7e..657038131 100644--- a/engine/crates/xerj-engine/tests/corrupt_index_metadata.rs+++ b/engine/crates/xerj-engine/tests/corrupt_index_metadata.rs@@ -328,6 +328,71 @@ async fn a_failed_index_can_still_be_deleted() {assert!(engine.get_index("removable").is_ok());}+/// The third recovery door, and the only one where #202 had to change the+/// engine: restoring the index from a snapshot must also *clear* the recorded+/// failure.+///+/// The restore loop reopens the index and puts it back in `indices`, but the+/// entry this change writes into `failed_indices` at boot is what cluster+/// health reads. Leave it behind and the repair works while `/_cluster/health`+/// stays red and `/health/ready` keeps reporting a failed index — a node that+/// is fine but says it is not, with no way to correct it short of a restart.+#[tokio::test]+async fn restoring_from_a_snapshot_clears_the_recorded_failure() {+ let dir = TempDir::new().unwrap();+ // The repository has to live under `data_dir` — an external snapshot root+ // is refused unless it is in `limits.snapshot_repo_allowlist`. It has no+ // `wal/` child, so the boot scan skips it rather than reading it as an+ // index.+ let repo_path = dir.path().join("snap_repo").to_str().unwrap().to_string();+ create_mapped_index(&dir, "snap_back").await;++ // Snapshot the index while it is still intact.+ {+ let engine = Engine::new(config_for(&dir)).expect("engine::new");+ engine+ .create_snapshot(&repo_path, "s1", Some(vec!["snap_back".to_string()]))+ .await+ .expect("create_snapshot");+ }++ // Now tear the sidecar: the node boots red with the index unserved.+ let path = dir.path().join("snap_back").join("schema.json");+ let good = std::fs::read(&path).unwrap();+ std::fs::write(&path, &good[..good.len() / 2]).unwrap();++ let engine = Engine::new(config_for(&dir)).expect("engine::new");+ assert_eq!(engine.health().await.status, "red");+ assert!(engine.failed_indices.contains_key("snap_back"));++ engine+ .restore_snapshot(&repo_path, "s1", Some(vec!["snap_back".to_string()]))+ .await+ .expect("restoring over a failed index must succeed — it is the repair");++ assert!(+ !engine.failed_indices.contains_key("snap_back"),+ "a successful restore must clear the recorded failure, or health stays \+ red after the repair actually worked"+ );+ assert_eq!(+ engine.health().await.status,+ "green",+ "health must follow the repair"+ );++ // And it is the snapshot's mapping that is being served, not a re-inferred+ // one — restoring must not be a second route to the #202 mapping loss.+ let index = engine+ .get_index("snap_back")+ .expect("the restored index must serve");+ assert_eq!(+ index.schema().await.field("code").map(|f| f.field_type),+ Some(FieldType::Keyword),+ "the restored index must carry the mapping the snapshot was taken with"+ );+}+// ── the writer must not manufacture what the reader now refuses ───────────────/// Refusing a torn sidecar is only safe if our own writer cannot produce one.