Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(#202): correct two overclaims, and pin the snapshot recovery door

xerj-org2677014db7c2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f172f4453..76e5feed1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -52,8 +52,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
now boots **red** with that index unserved, where it previously came up green
with an empty mapping. It joins the failed set
[#206](https://github.com/xerj-org/xerj/issues/206) introduced, so every
- surface that set feeds already reports it — measured on a two-index node with
- one truncated `schema.json`:
+ surface that set feeds already reports it — measured on a node holding two
+ user indices, `victim` with a truncated `schema.json`:
```
GET /_cluster/health -> red, unassigned_primary_shards: 1
@@ -66,15 +66,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
-> 503 no_shard_available_action_exception
carrying that same reason
GET /health/ready -> 200 "ready (degraded): 1 of 2 indices
- serving, 1 failed to open"
+ serving, 1 failed to open; see GET
+ /_cluster/indices/failed"
POST /healthy/_search -> 200
```
- Recovery is the three doors #206 added, both verified end to end here: repair
- the file and `POST /_cluster/indices/failed/{index}/_retry` (503 while it is
- still torn, `200 {"reopened": true}` once it is not, health back to green), or
- restore the index from a snapshot, or `DELETE /{index}`. The node stays in
- kubelet rotation as long as any index is still serving.
+ The readiness counts above are the two-index test router's. A running node
+ also holds its internal `.xerj_*` indices, so the same condition on the shipped
+ binary reads `ready (degraded): 15 of 16 indices serving, 1 failed to open` —
+ the 200 and its meaning are unchanged, only the totals move with what else the
+ node holds.
+
+ Recovery is the three doors #206 added. Two were measured end to end on the
+ shipped binary: repair the file and
+ `POST /_cluster/indices/failed/{index}/_retry` (503 while it is still torn,
+ `200 {"reopened": true}` once it is not, health back to green), and
+ `DELETE /{index}` (200, directory removed, health back to green). The third —
+ restore the index from a snapshot — is the path this change touches (a
+ successful restore now clears the recorded failure, which it has to or health
+ would stay red after the repair worked); it is covered by an engine-level
+ test rather than measured over HTTP. The node stays in kubelet rotation as
+ long as any index is still serving.
- **Concurrent sidecar writes can no longer manufacture the torn file the reader
now refuses.** `write_file_atomic` staged every write in
diff --git a/engine/crates/xerj-api/tests/corrupt_index_http.rs b/engine/crates/xerj-api/tests/corrupt_index_http.rs
index 5c615592c..090990609 100644
--- a/engine/crates/xerj-api/tests/corrupt_index_http.rs
+++ b/engine/crates/xerj-api/tests/corrupt_index_http.rs
@@ -19,9 +19,13 @@
//! side that stops a corrupt-metadata index from being visible, deletable and
//! survivable on `:9200` fails here rather than in production.
//!
-//! Every case below quarantines an index by truncating its `schema.json` —
-//! the #202 trigger — and then asserts the #206 contract over
-//! `build_es_compat_router`.
+//! Nine of the eleven cases below quarantine an index by truncating its
+//! `schema.json` — the #202 trigger — and then assert the #206 contract over
+//! `build_es_compat_router`. The other two are controls that pin the negative
+//! half of that contract, and deliberately corrupt nothing:
+//! `cluster_health_is_not_red_without_a_failed_index` (an intact node must not
+//! be dragged red by the code that reports failures) and `an_empty_node_is_ready`
+//! (a fresh pod with no indices at all must still join the Service).
use axum::body::Body;
use axum::http::{Request, StatusCode};
diff --git a/engine/crates/xerj-engine/tests/corrupt_index_metadata.rs b/engine/crates/xerj-engine/tests/corrupt_index_metadata.rs
index aa2a45b7e..657038131 100644
--- a/engine/crates/xerj-engine/tests/corrupt_index_metadata.rs
+++ b/engine/crates/xerj-engine/tests/corrupt_index_metadata.rs
@@ -328,6 +328,71 @@ async fn a_failed_index_can_still_be_deleted() {
assert!(engine.get_index("removable").is_ok());
}
+/// The third recovery door, and the only one where #202 had to change the
+/// engine: restoring the index from a snapshot must also *clear* the recorded
+/// failure.
+///
+/// The restore loop reopens the index and puts it back in `indices`, but the
+/// entry this change writes into `failed_indices` at boot is what cluster
+/// health reads. Leave it behind and the repair works while `/_cluster/health`
+/// stays red and `/health/ready` keeps reporting a failed index — a node that
+/// is fine but says it is not, with no way to correct it short of a restart.
+#[tokio::test]
+async fn restoring_from_a_snapshot_clears_the_recorded_failure() {
+ let dir = TempDir::new().unwrap();
+ // The repository has to live under `data_dir` — an external snapshot root
+ // is refused unless it is in `limits.snapshot_repo_allowlist`. It has no
+ // `wal/` child, so the boot scan skips it rather than reading it as an
+ // index.
+ let repo_path = dir.path().join("snap_repo").to_str().unwrap().to_string();
+ create_mapped_index(&dir, "snap_back").await;
+
+ // Snapshot the index while it is still intact.
+ {
+ let engine = Engine::new(config_for(&dir)).expect("engine::new");
+ engine
+ .create_snapshot(&repo_path, "s1", Some(vec!["snap_back".to_string()]))
+ .await
+ .expect("create_snapshot");
+ }
+
+ // Now tear the sidecar: the node boots red with the index unserved.
+ let path = dir.path().join("snap_back").join("schema.json");
+ let good = std::fs::read(&path).unwrap();
+ std::fs::write(&path, &good[..good.len() / 2]).unwrap();
+
+ let engine = Engine::new(config_for(&dir)).expect("engine::new");
+ assert_eq!(engine.health().await.status, "red");
+ assert!(engine.failed_indices.contains_key("snap_back"));
+
+ engine
+ .restore_snapshot(&repo_path, "s1", Some(vec!["snap_back".to_string()]))
+ .await
+ .expect("restoring over a failed index must succeed — it is the repair");
+
+ assert!(
+ !engine.failed_indices.contains_key("snap_back"),
+ "a successful restore must clear the recorded failure, or health stays \
+ red after the repair actually worked"
+ );
+ assert_eq!(
+ engine.health().await.status,
+ "green",
+ "health must follow the repair"
+ );
+
+ // And it is the snapshot's mapping that is being served, not a re-inferred
+ // one — restoring must not be a second route to the #202 mapping loss.
+ let index = engine
+ .get_index("snap_back")
+ .expect("the restored index must serve");
+ assert_eq!(
+ index.schema().await.field("code").map(|f| f.field_type),
+ Some(FieldType::Keyword),
+ "the restored index must carry the mapping the snapshot was taken with"
+ );
+}
+
// ── the writer must not manufacture what the reader now refuses ───────────────
/// Refusing a torn sidecar is only safe if our own writer cannot produce one.