Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): make a failed index inspectable, deletable and retryable (#206)

xerj-orgd78860eacede
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 00ff4fc62..970f2f6ad 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **A failed index can now be inspected, deleted and retried without stopping
+ the server** ([#206](https://github.com/xerj-org/xerj/issues/206)). An index
+ directory that refused to open at boot was recorded in a map that only
+ `Engine::health` ever read. Live-reproduced with a corrupt `snapshot.json`:
+ it was absent from `_cat/indices` and `_cluster/state`, `DELETE /{index}`
+ answered `404 index_not_found` and left the bytes on disk, and the only
+ recovery was to stop the server and edit the data directory by hand. A
+ failed index is now a real state: listed as `red` in `_cat/indices`, present
+ in `_cluster/state` as an `UNASSIGNED` primary with `ALLOCATION_FAILED` and
+ the verbatim open error, enumerated with its reason by
+ `GET /_cluster/indices/failed`, reopenable via
+ `POST /_cluster/indices/failed/{name}/_retry` once the cause is fixed, and
+ deletable through the ordinary `DELETE /{index}`. Reads, writes and creates
+ against it return `503 no_shard_available_action_exception` carrying the
+ reason instead of a `404` that claimed it did not exist.
+
+ Two related defects in the same report are fixed with it. **Readiness no
+ longer hard-fails on a partly degraded node** — `/health/ready` returned
+ `503` for any red status, so one broken index pulled a pod holding 200
+ healthy ones out of service permanently; it now reports `200 ready
+ (degraded)` while the node can still serve something and `503` only when
+ every index it holds failed to open. And **`_cat/health` no longer prints a
+ hardcoded `green`** — the one health surface an existing ES dashboard points
+ at was the one that could not report a broken node; it and `_cluster/health`
+ now count an unopenable index as an unassigned primary, agreeing with
+ `/v1/health` and `/v1/cluster/health`.
+
- **`autoindex` no longer leaves immortal catalog entries for skipped files**
([#238](https://github.com/xerj-org/xerj/issues/238)). A file added after the
resume plan was frozen is skipped and reported in the catalog — but that
diff --git a/docs/recipes/production-deployment.md b/docs/recipes/production-deployment.md
index 9a3dd865c..a13182e58 100644
--- a/docs/recipes/production-deployment.md
+++ b/docs/recipes/production-deployment.md
@@ -183,6 +183,57 @@ container. If you enable in-process TLS, change the probe to
your container is marked unhealthy forever. Same for a Kubernetes probe: set
`scheme: HTTPS` on the `httpGet`.
+### What readiness actually means
+
+`/health/ready` answers **"can this node serve?"**, not "is every index
+perfect". A node with 200 healthy indices and one that will not open stays
+`200` and keeps taking traffic, with the degradation named in the body:
+
+```text
+ready (degraded): 200 of 201 indices serving, 1 failed to open; see GET /_cluster/indices/failed
+```
+
+The body carries counts only — this path is exempt from authentication, so it
+must not enumerate index names to anything that can reach the port. The names
+and reasons live behind auth on `GET /_cluster/indices/failed`.
+
+It returns `503` only when there is genuinely nothing to serve — every index
+the node holds failed to open. This is deliberate: readiness used to hard-fail
+on any red status, so one bad index directory removed the whole pod from
+service permanently and took the 200 working indices down with it. The red
+status itself is untouched — `/v1/health`, `/v1/cluster/health`,
+`_cluster/health` and `_cat/health` all still report `red`, so alert on those,
+not on the probe.
+
+### Recovering an index that will not open
+
+An index directory that fails to open at boot is quarantined rather than
+silently dropped, and it is a state you can act on without stopping the
+server. All four moves work on a live node:
+
+```bash
+# 1. What failed, and why — the verbatim open error, not just a name
+curl -s localhost:9200/_cluster/indices/failed | jq .
+
+# 2. It is visible in the surfaces you already watch
+curl -s localhost:9200/_cat/indices # → red open broken <uuid> 1 0 0 0 0b 0b 0b
+curl -s "localhost:9200/_cluster/health?level=indices" | jq '.indices.broken'
+
+# 3. Fix the cause the error names (e.g. restore snapshot.json from a
+# backup or from an interrupted-write `.tmp.*` sibling), then retry
+curl -s -XPOST localhost:9200/_cluster/indices/failed/broken/_retry
+
+# 4. Or give up on it and remove it
+curl -s -XDELETE localhost:9200/broken
+```
+
+A retry that does not work returns `503` with the *current* reason, not the
+one recorded at boot. While an index is in this state, reads and writes to it
+return `503 no_shard_available_action_exception` naming the open error —
+**not** `404 index_not_found`, which would be a lie: the name is taken and the
+data is still on disk. Creating an index over that name is refused for the
+same reason.
+
### gRPC posture
The gRPC `XerjSearch` service (default `:8081`) always speaks **plaintext
diff --git a/engine/crates/xerj-api/src/authz.rs b/engine/crates/xerj-api/src/authz.rs
index b6a3df603..af6ce235d 100644
--- a/engine/crates/xerj-api/src/authz.rs
+++ b/engine/crates/xerj-api/src/authz.rs
@@ -1471,7 +1471,18 @@ async fn prune_response(
// other structural key would be dropped for a scoped principal, which
// would corrupt the very responses this is protecting. Read outside the
// request's visibility scope on purpose: this needs the unfiltered truth.
- let known: HashSet<String> = state.engine.index_name_list().into_iter().collect();
+ // Failed indices count as real names here (issue #206). They are now
+ // listed by `_cat/indices`, `_cat/shards` and `_cluster/state`, and a
+ // name that is not in this set is never a pruning candidate — so leaving
+ // them out would hand a scoped principal the name of a brain that
+ // happened to fail to open. The engine-side guard already filters them at
+ // the handler; this is the same second pass the open indices get.
+ let known: HashSet<String> = state
+ .engine
+ .index_name_list()
+ .into_iter()
+ .chain(state.engine.failed_indices.iter().map(|e| e.key().clone()))
+ .collect();
let pruned: Vec<u8> = if is_json {
match serde_json::from_slice::<Value>(&bytes) {
diff --git a/engine/crates/xerj-api/src/error.rs b/engine/crates/xerj-api/src/error.rs
index cd02ff37f..29fa420a0 100644
--- a/engine/crates/xerj-api/src/error.rs
+++ b/engine/crates/xerj-api/src/error.rs
@@ -264,6 +264,7 @@ impl IntoResponse for ApiError {
/// |---|---|---|
/// | `IndexNotFound` | `index_not_found_exception` | 404 |
/// | `IndexAlreadyExists` | `resource_already_exists_exception` | 409 |
+/// | `IndexUnavailable` | `no_shard_available_action_exception` | 503 |
/// | `DocumentNotFound` | `document_missing_exception` | 404 |
/// | `InvalidMapping` | `mapper_parsing_exception` | 400 |
/// | `InvalidQuery` | `search_phase_execution_exception` | 400 |
@@ -284,6 +285,10 @@ fn xerj_error_type(e: &XerjError) -> String {
// ── Index lifecycle ───────────────────────────────────────────────
XerjError::IndexNotFound { .. } => "index_not_found_exception",
XerjError::IndexAlreadyExists { .. } => "resource_already_exists_exception",
+ // The index exists; its only copy cannot be opened. ES reports an
+ // existing-but-unallocated index this way, so dashboards that already
+ // understand a red shard understand this too.
+ XerjError::IndexUnavailable { .. } => "no_shard_available_action_exception",
// ── Document operations ───────────────────────────────────────────
XerjError::DocumentNotFound { .. } => "document_missing_exception",
// ── Mapping / schema ──────────────────────────────────────────────
@@ -328,7 +333,9 @@ fn extract_index_name(e: &XerjError) -> Option<String> {
Some(name.clone())
}
XerjError::DocumentNotFound { index, .. } => Some(index.clone()),
- XerjError::IndexBlocked { index, .. } => Some(index.clone()),
+ XerjError::IndexBlocked { index, .. } | XerjError::IndexUnavailable { index, .. } => {
+ Some(index.clone())
+ }
_ => None,
}
}
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 91c93f51d..dc1fc533d 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -184,6 +184,36 @@ pub async fn cluster_health_for_index(
cluster_health_inner(state, Some(index), params).await
}
+/// The `unassigned_info` block for an index whose only primary could not be
+/// opened (issue #206).
+///
+/// Field names and shape follow Elasticsearch's own `UnassignedInfo`
+/// serialisation — `reason`, `at` as an ISO-8601 instant, `failed_attempts`
+/// only when there has been one, `delayed`, `details`, `allocation_status`
+/// (`elasticsearch/server/src/main/java/org/elasticsearch/cluster/routing/UnassignedInfo.java:483-503`,
+/// read for semantics only; Elasticsearch is licence-incompatible with this
+/// project and nothing was copied). Matching the names is the point of the
+/// exercise: the operator's existing dashboard already knows how to render
+/// this block, and `details` is where the verbatim open error goes.
+fn unassigned_info_json(f: &xerj_engine::engine::FailedIndex) -> Value {
+ let mut info = json!({
+ "reason": "ALLOCATION_FAILED",
+ "at": Utc.timestamp_millis_opt(f.failed_at_ms)
+ .single()
+ .map(|t| t.format("%Y-%m-%dT%H:%M:%S%.3fZ").to_string())
+ .unwrap_or_default(),
+ "delayed": false,
+ "details": f.reason,
+ // The primary's on-disk copy exists but cannot be opened, so there is
+ // no copy this node can allocate — ES's `no_valid_shard_copy`.
+ "allocation_status": "no_valid_shard_copy",
+ });
+ if f.retries > 0 {
+ info["failed_attempts"] = json!(f.retries);
+ }
+ info
+}
+
async fn cluster_health_inner(
state: AppState,
index_filter: Option<String>,
@@ -248,6 +278,24 @@ async fn cluster_health_inner(
selected
};
+ // Indices that exist on disk but refused to open. Their primary has no
+ // serving copy, which is the textbook definition of a red cluster — and
+ // the one condition this endpoint used to ignore entirely, so an operator
+ // watching the ES-compatible surface never saw the failure that
+ // `/v1/health` was already reporting (issue #206).
+ let selected_failed: Vec<xerj_engine::engine::FailedIndex> = state
+ .engine
+ .list_failed_indices()
+ .into_iter()
+ .filter(|f| match &index_filter {
+ None => true,
+ Some(sel) => sel
+ .split(',')
+ .map(str::trim)
+ .any(|s| s == "_all" || s == "*" || s == f.name || glob_match_simple(s, &f.name)),
+ })
+ .collect();
+
// Per-index shard count helper — defaults to 1 when unset.
let shard_count = |name: &str| -> u32 {
state
@@ -267,7 +315,7 @@ async fn cluster_health_inner(
.unwrap_or(1) as u32
};
- let idx_count = selected.len() as u32;
+ let idx_count = (selected.len() + selected_failed.len()) as u32;
let mut closed_count = 0u32;
let mut unassigned_replicas: u32 = 0;
for info in &selected {
@@ -314,12 +362,17 @@ async fn cluster_health_inner(
}
}
- // Any unassigned replica forces yellow; closed indices surface as yellow
- // in the legacy pre-7.2 path, but post-7.2 replicated-closed semantics
- // keep the *cluster* status driven by replicas only. Our tests cover
- // both — we currently only track a single "closed" flag per index and
- // don't differentiate the closed-replication mode.
- let status = if unassigned_replicas > 0 {
+ // An unopenable index is an unassigned PRIMARY — red, and it outranks
+ // every yellow condition. Any unassigned replica forces yellow; closed
+ // indices surface as yellow in the legacy pre-7.2 path, but post-7.2
+ // replicated-closed semantics keep the *cluster* status driven by
+ // replicas only. Our tests cover both — we currently only track a single
+ // "closed" flag per index and don't differentiate the closed-replication
+ // mode.
+ let unassigned_primaries = selected_failed.len() as u32;
+ let status = if unassigned_primaries > 0 {
+ "red"
+ } else if unassigned_replicas > 0 {
"yellow"
} else {
"green"
@@ -360,12 +413,15 @@ async fn cluster_health_inner(
})
.unwrap_or(Some(1))
.unwrap_or(1);
- // wait_for_active_shards unmet: `all` when there are any unassigned
- // replicas, or a numeric count greater than the currently active
- // shards. The HTTP `timeout` has already elapsed by the time this
- // check runs (we don't block), so we set timed_out accordingly.
+ // wait_for_active_shards unmet: `all` when any shard is unassigned —
+ // a replica, or (now that a failed index is a reachable state, issue
+ // #206) an unopenable primary, which is the more serious of the two and
+ // would otherwise have satisfied "all shards active" on a red cluster.
+ // A numeric count is unmet when it exceeds the currently active shards.
+ // The HTTP `timeout` has already elapsed by the time this check runs (we
+ // don't block), so we set timed_out accordingly.
let wait_for_active_shards_unmet = match params.wait_for_active_shards.as_deref() {
- Some("all") => unassigned_replicas > 0,
+ Some("all") => unassigned_replicas > 0 || unassigned_primaries > 0,
Some(s) => s
.parse::<u64>()
.ok()
@@ -412,8 +468,8 @@ async fn cluster_health_inner(
"active_shards": active,
"relocating_shards": 0,
"initializing_shards": 0,
- "unassigned_shards": unassigned_replicas,
- "unassigned_primary_shards": 0,
+ "unassigned_shards": unassigned_replicas + unassigned_primaries,
+ "unassigned_primary_shards": unassigned_primaries,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
@@ -475,6 +531,36 @@ async fn cluster_health_inner(
}
indices_map.insert(info.name.clone(), idx_obj);
}
+ // Failed indices carry the open error verbatim in `unassigned_info`,
+ // so `?level=indices` answers "which index, and why" in one call.
+ for f in &selected_failed {
+ let mut idx_obj = json!({
+ "status": "red",
+ "number_of_shards": 1,
+ "number_of_replicas": 0,
+ "active_primary_shards": 0,
+ "active_shards": 0,
+ "relocating_shards": 0,
+ "initializing_shards": 0,
+ "unassigned_shards": 1,
+ "unassigned_primary_shards": 1,
+ "unassigned_info": unassigned_info_json(f),
+ });
+ if level == "shards" {
+ idx_obj["shards"] = json!({
+ "0": {
+ "status": "red",
+ "primary_active": false,
+ "active_shards": 0,
+ "relocating_shards": 0,
+ "initializing_shards": 0,
+ "unassigned_shards": 1,
+ "unassigned_primary_shards": 1,
+ }
+ });
+ }
+ indices_map.insert(f.name.clone(), idx_obj);
+ }
resp["indices"] = Value::Object(indices_map);
}
@@ -510,6 +596,11 @@ pub async fn cat_indices_pattern(
async fn cat_indices_inner(state: AppState, pattern: Option<String>) -> axum::response::Response {
let indices = state.engine.list_indices().await;
+ // Indices whose directory exists but refused to open. They are listed as
+ // `red` rather than omitted: an index the operator can see is an index the
+ // operator can delete or retry, and omitting them is what made a failed
+ // index invisible to every dashboard (issue #206).
+ let failed = state.engine.list_failed_indices();
// Narrow to the path pattern when present (`/_cat/indices/{pattern}`):
// a comma-separated list of concrete names and/or `*` globs. ES rules:
@@ -526,7 +617,10 @@ async fn cat_indices_inner(state: AppState, pattern: Option<String>) -> axum::re
.collect();
for &p in &parts {
let is_wildcard = p == "_all" || p == "*" || p.contains('*');
- if !is_wildcard && !indices.iter().any(|i| i.name == p) {
+ if !is_wildcard
+ && !indices.iter().any(|i| i.name == p)
+ && !failed.iter().any(|f| f.name == p)
+ {
return ApiError::new(xerj_common::XerjError::index_not_found(p))
.into_response();
}
@@ -569,6 +663,33 @@ async fn cat_indices_inner(state: AppState, pattern: Option<String>) -> axum::re
hsize,
));
}
+
+ // Failed indices, after the healthy ones. `red` is the status column and
+ // the shard is unassigned, so `pri` is 1 and every stat column is 0 —
+ // nothing about the contents can be read, and guessing a doc count here
+ // would be a claim we cannot back.
+ let selected_failed: Vec<&xerj_engine::engine::FailedIndex> = match pattern.as_deref() {
+ None => failed.iter().collect(),
+ Some(pat) => failed
+ .iter()
+ .filter(|f| {
+ pat.split(',')
+ .map(str::trim)
+ .filter(|s| !s.is_empty())
+ .any(|p| {
+ p == "_all" || p == "*" || f.name == p || glob_match_simple(p, &f.name)
+ })
+ })
+ .collect(),
+ };
+ for f in selected_failed {
+ lines.push(format!(
+ "red open {} {} 1 0 0 0 0b 0b 0b",
+ f.name,
+ stable_index_uuid(&f.name),
+ ));
+ }
+
// ES returns an empty body (not a bare newline) when nothing matches.
let body = if lines.is_empty() {
String::new()
@@ -1334,7 +1455,17 @@ pub async fn delete_index(
.unwrap_or(true);
let all = state.engine.list_indices().await;
- let all_names: Vec<String> = all.iter().map(|i| i.name.clone()).collect();
+ let mut all_names: Vec<String> = all.iter().map(|i| i.name.clone()).collect();
+ // A failed index is deletable. Its name has to take part in selector
+ // resolution or a literal `DELETE /broken` 404s and a wildcard silently
+ // skips it — which is exactly the stuck state issue #206 describes.
+ all_names.extend(
+ state
+ .engine
+ .list_failed_indices()
+ .into_iter()
+ .map(|f| f.name),
+ );
let mut to_delete: Vec<String> = Vec::new();
let parts: Vec<&str> = index
@@ -1396,7 +1527,21 @@ pub async fn delete_index(
}
for name in &to_delete {
- let _ = state.engine.delete_index(name).await;
+ // Report a delete that did not happen. The result used to be dropped
+ // on the floor, so an index whose bytes could not be removed (a failed
+ // index on a read-only mount, an fs error) still answered
+ // `acknowledged: true` and the operator had no way to know the name
+ // was still taken. A concurrent delete that already removed it is the
+ // one benign outcome and stays silent.
+ match state.engine.delete_index(name).await {
+ Ok(()) => {}
+ Err(e) => {
+ let inner: xerj_common::XerjError = e.into();
+ if !matches!(inner, xerj_common::XerjError::IndexNotFound { .. }) {
+ return ApiError::new(inner).into_response();
+ }
+ }
+ }
// RC4-W4 item 5: drop the index's per-index metric label series so it
// doesn't linger for the process lifetime after the index is gone.
state.metrics.prune_index_labels(name);
@@ -15863,6 +16008,14 @@ async fn cat_ann_inner(
// GET /_cat/health
// ─────────────────────────────────────────────────────────────────────────────
+/// `GET /_cat/health`.
+///
+/// The status column used to be the literal string `green`, so the one
+/// endpoint an operator's existing dashboard points at was the one endpoint
+/// that could not report a broken node — `/v1/health`, `/v1/cluster/health`
+/// and `/health/ready` all knew, and `_cat/health` said everything was fine
+/// (issue #206). It now reports the same status `_cluster/health` computes,
+/// from the same inputs.
pub async fn cat_health(State(state): State<AppState>) -> impl IntoResponse {
let health = state.engine.health().await;
// epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
@@ -15873,7 +16026,21 @@ pub async fn cat_health(State(state): State<AppState>) -> impl IntoResponse {
let now = Utc::now();
let ts = now.format("%H:%M:%S").to_string();
let shards = health.index_count as u32;
- let body = format!("{epoch} {ts} xerj green 1 1 {shards} {shards} 0 0 0 0 - 100.0%\n");
+ let unassigned = state.engine.list_failed_indices().len() as u32;
+ // Mirror `_cluster/health`: an unopenable index is an unassigned primary
+ // (red); an unassigned replica is yellow. The engine's own "yellow" means
+ // "unflushed memtable", which is a durability nuance rather than an ES
+ // shard state, so it deliberately does not colour this column.
+ let status = if unassigned > 0 { "red" } else { "green" };
+ let total = shards + unassigned;
+ let pct = if total == 0 {
+ 100.0
+ } else {
+ (shards as f64) / (total as f64) * 100.0
+ };
+ let body = format!(
+ "{epoch} {ts} xerj {status} 1 1 {shards} {shards} 0 0 {unassigned} 0 - {pct:.1}%\n"
+ );
(
StatusCode::OK,
[(
@@ -21247,6 +21414,12 @@ pub async fn cat_shards(State(state): State<AppState>) -> impl IntoResponse {
info.name, info.doc_count, store_bytes,
));
}
+ // `_cat/shards` is the call an operator makes straight after seeing a red
+ // cluster, so an index whose primary could not be opened has to be in it.
+ // ES prints UNASSIGNED rows with no node and empty stats.
+ for f in state.engine.list_failed_indices() {
+ lines.push(format!("{} 0 p UNASSIGNED 0 0b - -", f.name));
+ }
let body = if lines.is_empty() {
String::new()
} else {
@@ -23340,6 +23513,45 @@ pub async fn cluster_state(State(state): State<AppState>) -> impl IntoResponse {
);
}
+ // An index that would not open still exists — its metadata belongs in
+ // cluster state, and its shard is UNASSIGNED with the open error as the
+ // allocation-failure detail. Leaving it out is what made a failed index
+ // invisible to every tool that reads cluster state (issue #206).
+ let mut unassigned_shards: Vec<Value> = Vec::new();
+ for f in state.engine.list_failed_indices() {
+ metadata_indices.insert(
+ f.name.clone(),
+ json!({
+ "state": "open",
+ "settings": {
+ "index": {
+ "number_of_shards": "1",
+ "number_of_replicas": "0",
+ "uuid": stable_index_uuid(&f.name),
+ "version": { "created": "8130099" },
+ "provided_name": f.name,
+ }
+ },
+ "mappings": {},
+ "aliases": [],
+ }),
+ );
+ let shard = json!({
+ "state": "UNASSIGNED",
+ "primary": true,
+ "node": null,
+ "relocating_node": null,
+ "shard": 0,
+ "index": f.name,
+ "unassigned_info": unassigned_info_json(&f),
+ });
+ routing_table.insert(
+ f.name.clone(),
+ json!({ "shards": { "0": [shard.clone()] } }),
+ );
+ unassigned_shards.push(shard);
+ }
+
Json(json!({
"cluster_name": "xerj",
"cluster_uuid": "xerj-cluster-1",
@@ -23363,7 +23575,7 @@ pub async fn cluster_state(State(state): State<AppState>) -> impl IntoResponse {
"indices": routing_table,
},
"routing_nodes": {
- "unassigned": [],
+ "unassigned": unassigned_shards,
"nodes": {
node_id: []
}
diff --git a/engine/crates/xerj-api/src/native.rs b/engine/crates/xerj-api/src/native.rs
index 824dc13b3..c008077bc 100644
--- a/engine/crates/xerj-api/src/native.rs
+++ b/engine/crates/xerj-api/src/native.rs
@@ -708,12 +708,22 @@ pub async fn embedding_execution_identity(State(state): State<AppState>) -> impl
// be a no-op — checking the engine here would create a feedback loop
// where a slow engine flapping flush causes constant pod restarts.
//
-// - **readiness**: returns 200 only when the engine reports a non-`red`
-// cluster status (i.e. at least one index is queryable). kubelet uses
-// this to decide whether to send traffic. Until ready, the Service
-// removes the pod from rotation but doesn't restart it — appropriate
-// for "still replaying WAL" or "still loading 50 K segments" startup
-// states that are transient but visible.
+// - **readiness**: returns 200 when the node can serve. kubelet uses this
+// to decide whether to send traffic. Until ready, the Service removes the
+// pod from rotation but doesn't restart it — appropriate for "still
+// replaying WAL" or "still loading 50 K segments" startup states that are
+// transient but visible.
+//
+// Readiness is deliberately NOT "cluster status != red" (issue #206). One
+// index that will not open turns the engine red forever, and a pod with 200
+// healthy indices and one broken one was therefore pulled out of service
+// permanently and never came back — a single-index problem escalated into a
+// total outage, with the 199 working indices unreachable. A node is ready
+// when it has at least one index it can serve, or nothing to serve at all;
+// it is unready only when every index it holds is broken. The red status
+// itself is untouched and still reported by `/v1/health`,
+// `/v1/cluster/health`, `_cluster/health` and `_cat/health`, and the
+// degraded state is named in this body so a probe log says which it is.
// ─────────────────────────────────────────────────────────────────────────────
pub async fn liveness() -> impl IntoResponse {
@@ -722,14 +732,95 @@ pub async fn liveness() -> impl IntoResponse {
pub async fn readiness(State(state): State<AppState>) -> impl IntoResponse {
let h = state.engine.health().await;
- if h.status == "red" {
- (
+ // Read the node's own state, not a principal's filtered view: this path is
+ // exempt from authentication so a kubelet can probe a hardened node, and
+ // the probe has to answer for the whole node. For the same reason the body
+ // carries COUNTS ONLY — naming the failed indices here would hand every
+ // index name on the node to anyone who can reach the port. The names live
+ // behind auth on `GET /_cluster/indices/failed`.
+ let failed = state.engine.failed_indices.len();
+ if failed == 0 {
+ return (axum::http::StatusCode::OK, "ready").into_response();
+ }
+ if h.index_count == 0 {
+ // Nothing on this node can be served: every index it holds failed to
+ // open. Removing it from rotation is correct here.
+ return (
axum::http::StatusCode::SERVICE_UNAVAILABLE,
- format!("not ready: cluster status = {}", h.status),
+ format!(
+ "not ready: no index could be opened ({failed} failed); \
+ see GET /_cluster/indices/failed"
+ ),
)
- .into_response()
- } else {
- (axum::http::StatusCode::OK, "ready").into_response()
+ .into_response();
+ }
+ (
+ axum::http::StatusCode::OK,
+ format!(
+ "ready (degraded): {} of {} indices serving, {failed} failed to open; \
+ see GET /_cluster/indices/failed",
+ h.index_count,
+ h.index_count + failed
+ ),
+ )
+ .into_response()
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Handler: GET /_cluster/indices/failed — list failed indices
+// Handler: POST /_cluster/indices/failed/{name}/_retry — retry one open
+//
+// Issue #206: an index whose directory refused to open had no operator
+// surface at all. These two, plus `DELETE /{index}` (which now accepts a
+// failed index), are the three moves the issue asks for: inspect with the
+// real reason, retry after fixing the cause, delete.
+// ─────────────────────────────────────────────────────────────────────────────
+
+pub async fn failed_indices(State(state): State<AppState>) -> impl IntoResponse {
+ let started = Instant::now();
+ let request_id = Uuid::new_v4().to_string();
+ let failed = state.engine.list_failed_indices();
+ let items: Vec<serde_json::Value> = failed
+ .iter()
+ .map(|f| {
+ serde_json::json!({
+ "index": f.name,
+ "reason": f.reason,
+ "failed_at_millis": f.failed_at_ms,
+ "retries": f.retries,
+ "retry": format!("POST /_cluster/indices/failed/{}/_retry", f.name),
+ "delete": format!("DELETE /{}", f.name),
+ })
+ })
+ .collect();
+ let resp = NativeResponse::new(
+ serde_json::json!({ "count": items.len(), "failed_indices": items }),
+ started.elapsed().as_millis() as u64,
+ &request_id,
+ );
+ Json(resp).into_response()
+}
+
+pub async fn retry_failed_index(
+ State(state): State<AppState>,
+ axum::extract::Path(name): axum::extract::Path<String>,
+) -> impl IntoResponse {
+ let started = Instant::now();
+ let request_id = Uuid::new_v4().to_string();
+ match state.engine.retry_failed_index(&name) {
+ Ok(()) => {
+ let resp = NativeResponse::new(
+ serde_json::json!({ "index": name, "reopened": true }),
+ started.elapsed().as_millis() as u64,
+ &request_id,
+ );
+ Json(resp).into_response()
+ }
+ // The retry did not work. Answering 200 with `reopened: false` would
+ // be exactly the accepted-and-ignored shape this repo is trying to
+ // stop shipping — the caller gets the live failure instead (503 for a
+ // still-broken index, 404 for a name that is not a failed index).
+ Err(e) => crate::error::ApiError::new(e.into()).into_response(),
}
}
diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rs
index c2fb51f1e..28bc98596 100644
--- a/engine/crates/xerj-api/src/router.rs
+++ b/engine/crates/xerj-api/src/router.rs
@@ -51,8 +51,10 @@ use crate::{
/// POST /v1/indices/:name/search search
/// POST /v1/indices/:name/_flush flush_index
/// GET /v1/health health
-/// GET /v1/health/ready readiness probe (200 unless red)
+/// GET /v1/health/ready readiness probe (200 unless nothing can be served)
/// GET /v1/cluster/health cluster_health
+/// GET /_cluster/indices/failed failed_indices (name + open error)
+/// POST /_cluster/indices/failed/:name/_retry retry_failed_index
/// POST /v1/admin/flush admin_flush (flush all indices)
/// POST /v1/admin/backup admin_backup (snapshot to disk)
/// GET /v1/metrics metrics (Prometheus text)
@@ -104,6 +106,15 @@ pub fn build_native_router(state: AppState) -> Router {
.route("/v1/health", get(native::health))
.route("/v1/health/ready", get(native::readiness))
.route("/v1/cluster/health", get(native::cluster_health))
+ // Failed-index recovery (issue #206) — inspect / retry. Delete goes
+ // through the ordinary `DELETE /{index}`. Mounted on both routers
+ // under the same path so an operator does not have to know which
+ // listener they reached.
+ .route("/_cluster/indices/failed", get(native::failed_indices))
+ .route(
+ "/_cluster/indices/failed/:name/_retry",
+ post(native::retry_failed_index),
+ )
.route(
"/v1/embedding/identity",
get(native::embedding_execution_identity),
@@ -515,6 +526,17 @@ pub fn build_es_compat_router(state: AppState) -> Router {
get(es_compat::get_cluster_settings).put(es_compat::put_cluster_settings),
)
.route("/_cluster/reroute", post(es_compat::cluster_reroute))
+ // ── Failed-index recovery (issue #206) ─────────────────────────────
+ // XERJ extension: no ES equivalent (ES recovers a red index through
+ // shard reallocation, which a single-binary node has no analogue for),
+ // so it cannot collide with the parity surface. Mounted on the
+ // ES-compat port because that is the port an operator already has open
+ // when they discover the red index there.
+ .route("/_cluster/indices/failed", get(native::failed_indices))
+ .route(
+ "/_cluster/indices/failed/:name/_retry",
+ post(native::retry_failed_index),
+ )
.route(
"/_cluster/pending_tasks",
get(es_compat::cluster_pending_tasks),
diff --git a/engine/crates/xerj-api/tests/failed_index_is_visible_and_actionable.rs b/engine/crates/xerj-api/tests/failed_index_is_visible_and_actionable.rs
new file mode 100644
index 000000000..bcbad450b
--- /dev/null
+++ b/engine/crates/xerj-api/tests/failed_index_is_visible_and_actionable.rs
@@ -0,0 +1,447 @@
+//! Issue #206, over HTTP: a failed index must be visible on the surfaces an
+//! operator actually watches, and actionable from them.
+//!
+//! Three separate failures were reported and all three are pinned here.
+//!
+//! 1. **The index was invisible.** `_cat/indices` and `_cluster/state` list
+//! only indices that opened, so the broken one appeared nowhere; `DELETE`
+//! answered 404 `index_not_found`. The only recovery was to stop the server
+//! and remove the directory by hand.
+//! 2. **Readiness went hard-fail.** `/health/ready` returned 503 for any red
+//! status, so one broken index removed a pod holding 199 healthy ones from
+//! service permanently — a single-index problem escalated to a total
+//! outage.
+//! 3. **The health surfaces disagreed.** `/v1/health`, `/v1/cluster/health`
+//! and `/health/ready` used real engine health while `_cat/health` printed
+//! a hardcoded `green` — so the ES-compatible endpoint, the one existing
+//! dashboards point at, was the one that could not report a broken node.
+//! `_cluster/health` did not count an unopenable primary either.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use http_body_util::BodyExt;
+use serde_json::Value;
+use tower::ServiceExt;
+use xerj_api::{
+ router::{build_es_compat_router, build_native_router},
+ state::AppState,
+};
+use xerj_common::types::Schema;
+use xerj_common::{config::Config, metrics::Metrics};
+use xerj_engine::Engine;
+
+/// A node whose data dir holds one healthy index and one whose manifest is
+/// corrupt, so the boot that produces `state` quarantines exactly one index.
+async fn node_with_one_broken_index() -> (AppState, tempfile::TempDir) {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ {
+ let engine = Engine::new(config.clone()).expect("first boot");
+ for name in ["broken", "healthy"] {
+ engine.create_index(name, Schema::empty()).unwrap();
+ let idx = engine.get_index(name).unwrap();
+ idx.index_document(Some("1".into()), serde_json::json!({"t": "hello"}))
+ .await
+ .unwrap();
+ engine.flush_index(name).await.unwrap();
+ }
+ }
+ std::fs::write(
+ dir.path().join("broken").join("snapshot.json"),
+ b"{not json",
+ )
+ .unwrap();
+
+ let metrics = Metrics::new().expect("metrics");
+ let engine = Engine::new(config.clone()).expect("boot with one broken index");
+ assert_eq!(
+ engine.list_failed_indices().len(),
+ 1,
+ "fixture must produce exactly one failed index"
+ );
+ (AppState::new(config, engine, metrics), dir)
+}
+
+async fn send(app: &axum::Router, method: &str, uri: &str) -> (StatusCode, String) {
+ let req = Request::builder()
+ .method(method)
+ .uri(uri)
+ .header("content-type", "application/json")
+ .body(Body::empty())
+ .expect("request");
+ let resp = app.clone().oneshot(req).await.expect("response");
+ let status = resp.status();
+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();
+ (status, String::from_utf8_lossy(&bytes).into_owned())
+}
+
+async fn send_json(app: &axum::Router, method: &str, uri: &str) -> (StatusCode, Value) {
+ let (status, text) = send(app, method, uri).await;
+ (status, serde_json::from_str(&text).unwrap_or(Value::Null))
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn cat_indices_lists_the_failed_index_as_red() {
+ let (state, _dir) = node_with_one_broken_index().await;
+ let app = build_es_compat_router(state);
+
+ let (status, body) = send(&app, "GET", "/_cat/indices").await;
+ assert_eq!(status, StatusCode::OK);
+ let broken: Vec<&str> = body.lines().filter(|l| l.contains("broken")).collect();
+ assert_eq!(
+ broken.len(),
+ 1,
+ "failed index missing from _cat/indices: {body:?}"
+ );
+ assert!(
+ broken[0].starts_with("red open broken "),
+ "a failed index must be listed red: {:?}",
+ broken[0]
+ );
+ assert!(
+ body.lines().any(|l| l.starts_with("green open healthy ")),
+ "the healthy index must be untouched: {body:?}"
+ );
+
+ // A concrete-name request for it resolves instead of 404ing.
+ let (status, body) = send(&app, "GET", "/_cat/indices/broken").await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert!(body.starts_with("red open broken "), "{body:?}");
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn all_four_health_surfaces_agree_that_the_node_is_red() {
+ let (state, _dir) = node_with_one_broken_index().await;
+ let es = build_es_compat_router(state.clone());
+ let native = build_native_router(state);
+
+ // `_cat/health` used to print a hardcoded `green` here.
+ let (status, body) = send(&es, "GET", "/_cat/health").await;
+ assert_eq!(status, StatusCode::OK);
+ let cols: Vec<&str> = body.split_whitespace().collect();
+ assert_eq!(cols[3], "red", "_cat/health status column: {body:?}");
+ assert_eq!(cols[10], "1", "_cat/health unassign column: {body:?}");
+
+ // `_cluster/health` counted unassigned replicas only, never an
+ // unopenable primary.
+ let (status, health) = send_json(&es, "GET", "/_cluster/health").await;
+ assert_eq!(status, StatusCode::OK);
+ assert_eq!(health["status"], "red", "{health}");
+ assert_eq!(health["unassigned_primary_shards"], 1, "{health}");
+ assert_eq!(health["unassigned_shards"], 1, "{health}");
+
+ // `?level=indices` answers "which index, and why" in one call.
+ let (_, detail) = send_json(&es, "GET", "/_cluster/health?level=indices").await;
+ let broken = &detail["indices"]["broken"];
+ assert_eq!(broken["status"], "red", "{detail}");
+ assert!(
+ broken["unassigned_info"]["details"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("snapshot.json"),
+ "the open error must reach the operator: {detail}"
+ );
+
+ // The two native surfaces already agreed; they must keep agreeing.
+ let (_, native_health) = send_json(&native, "GET", "/v1/health").await;
+ assert_eq!(native_health["data"]["status"], "red", "{native_health}");
+ let (_, native_cluster) = send_json(&native, "GET", "/v1/cluster/health").await;
+ assert_eq!(native_cluster["data"]["status"], "red", "{native_cluster}");
+
+ // `wait_for_active_shards=all` must not report success on a cluster whose
+ // primary cannot be opened — "all shards active" is exactly what is false.
+ let (status, waited) =
+ send_json(&es, "GET", "/_cluster/health?wait_for_active_shards=all").await;
+ assert_eq!(status, StatusCode::REQUEST_TIMEOUT, "{waited}");
+ assert_eq!(waited["timed_out"], true, "{waited}");
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn cluster_state_carries_the_failed_index_as_an_unassigned_primary() {
+ let (state, _dir) = node_with_one_broken_index().await;
+ let app = build_es_compat_router(state);
+
+ let (status, cs) = send_json(&app, "GET", "/_cluster/state").await;
+ assert_eq!(status, StatusCode::OK);
+ assert!(
+ cs["metadata"]["indices"]["broken"].is_object(),
+ "a failed index still exists and belongs in cluster state: {cs}"
+ );
+ let shard = &cs["routing_table"]["indices"]["broken"]["shards"]["0"][0];
+ assert_eq!(shard["state"], "UNASSIGNED", "{cs}");
+ assert_eq!(shard["primary"], true, "{cs}");
+ assert_eq!(
+ shard["unassigned_info"]["reason"], "ALLOCATION_FAILED",
+ "{cs}"
+ );
+ assert!(
+ shard["unassigned_info"]["details"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("could not be parsed"),
+ "{cs}"
+ );
+ // ES's own `unassigned_info` field names, so an operator's existing
+ // dashboard renders this block without knowing anything about XERJ.
+ assert_eq!(
+ shard["unassigned_info"]["allocation_status"], "no_valid_shard_copy",
+ "{cs}"
+ );
+ assert!(
+ shard["unassigned_info"]["at"]
+ .as_str()
+ .unwrap_or_default()
+ .ends_with('Z'),
+ "`at` must be an ISO-8601 instant like ES: {cs}"
+ );
+ assert_eq!(
+ cs["routing_nodes"]["unassigned"].as_array().map(Vec::len),
+ Some(1),
+ "{cs}"
+ );
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn readiness_keeps_a_partly_degraded_node_in_service() {
+ let (state, _dir) = node_with_one_broken_index().await;
+ let app = build_es_compat_router(state);
+
+ // One broken index out of two: the node can still serve `healthy`, so
+ // pulling it out of rotation forever would be the outage, not the fix.
+ let (status, body) = send(&app, "GET", "/health/ready").await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert!(
+ body.contains("degraded"),
+ "the degradation must be named: {body:?}"
+ );
+ assert!(body.contains("1 of 2 indices serving"), "{body:?}");
+ // Counts only — this path is auth-exempt, so it must not enumerate index
+ // names to an unauthenticated prober.
+ assert!(
+ !body.contains("broken"),
+ "readiness leaked an index name: {body:?}"
+ );
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn readiness_still_fails_when_nothing_can_be_served() {
+ // Every index in the data dir is broken — there is genuinely nothing to
+ // send traffic to, so 503 is correct and must survive the fix above.
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ {
+ let engine = Engine::new(config.clone()).expect("first boot");
+ engine.create_index("only", Schema::empty()).unwrap();
+ let idx = engine.get_index("only").unwrap();
+ idx.index_document(Some("1".into()), serde_json::json!({"t": "x"}))
+ .await
+ .unwrap();
+ engine.flush_index("only").await.unwrap();
+ }
+ std::fs::write(dir.path().join("only").join("snapshot.json"), b"{not json").unwrap();
+ let engine = Engine::new(config.clone()).expect("reboot");
+ let state = AppState::new(config, engine, Metrics::new().expect("metrics"));
+ let app = build_es_compat_router(state);
+
+ let (status, body) = send(&app, "GET", "/health/ready").await;
+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{body}");
+ assert!(body.contains("no index could be opened"), "{body:?}");
+ assert!(
+ !body.contains("only"),
+ "readiness leaked an index name: {body:?}"
+ );
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn the_failed_index_can_be_inspected_retried_and_deleted_over_http() {
+ let (state, dir) = node_with_one_broken_index().await;
+ let app = build_es_compat_router(state);
+
+ // Inspect: name AND reason, plus the two commands that act on it.
+ let (status, listing) = send_json(&app, "GET", "/_cluster/indices/failed").await;
+ assert_eq!(status, StatusCode::OK);
+ let items = listing["data"]["failed_indices"].as_array().expect("array");
+ assert_eq!(items.len(), 1, "{listing}");
+ assert_eq!(items[0]["index"], "broken");
+ assert!(
+ items[0]["reason"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("snapshot.json"),
+ "{listing}"
+ );
+
+ // Reading it names the real problem instead of claiming it does not exist.
+ let (status, err) = send_json(&app, "GET", "/broken/_search").await;
+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{err}");
+ assert_eq!(
+ err["error"]["type"], "no_shard_available_action_exception",
+ "{err}"
+ );
+
+ // Retry while still broken: 503 with the live reason, not a cheerful 200.
+ let (status, err) = send_json(&app, "POST", "/_cluster/indices/failed/broken/_retry").await;
+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{err}");
+ assert!(
+ err["error"]["reason"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("snapshot.json"),
+ "{err}"
+ );
+
+ // Delete without a restart.
+ let (status, body) = send_json(&app, "DELETE", "/broken").await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["acknowledged"], true, "{body}");
+ assert!(
+ !dir.path().join("broken").exists(),
+ "DELETE must remove the bytes on disk"
+ );
+
+ // The node is green again and nothing is left to inspect.
+ let (_, listing) = send_json(&app, "GET", "/_cluster/indices/failed").await;
+ assert_eq!(listing["data"]["count"], 0, "{listing}");
+ let (_, health) = send_json(&app, "GET", "/_cluster/health").await;
+ assert_eq!(health["status"], "green", "{health}");
+ let (status, body) = send(&app, "GET", "/_cat/indices").await;
+ assert_eq!(status, StatusCode::OK);
+ assert!(!body.contains("broken"), "{body:?}");
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn retry_over_http_reopens_the_index_once_the_cause_is_fixed() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ {
+ let engine = Engine::new(config.clone()).expect("first boot");
+ engine.create_index("broken", Schema::empty()).unwrap();
+ let idx = engine.get_index("broken").unwrap();
+ idx.index_document(Some("1".into()), serde_json::json!({"t": "hello"}))
+ .await
+ .unwrap();
+ engine.flush_index("broken").await.unwrap();
+ }
+ let manifest = dir.path().join("broken").join("snapshot.json");
+ let good = std::fs::read(&manifest).unwrap();
+ std::fs::write(&manifest, b"{not json").unwrap();
+
+ let engine = Engine::new(config.clone()).expect("reboot");
+ let state = AppState::new(config, engine, Metrics::new().expect("metrics"));
+ let app = build_es_compat_router(state);
+
+ // Operator restores the manifest the storage error pointed them at, then
+ // retries — no restart.
+ std::fs::write(&manifest, &good).unwrap();
+ let (status, body) = send_json(&app, "POST", "/_cluster/indices/failed/broken/_retry").await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["data"]["reopened"], true, "{body}");
+
+ let (_, health) = send_json(&app, "GET", "/_cluster/health").await;
+ assert_eq!(health["status"], "green", "{health}");
+ let (status, count) = send_json(&app, "GET", "/broken/_count").await;
+ assert_eq!(status, StatusCode::OK, "{count}");
+ assert_eq!(count["count"], 1, "the reopened index must serve its data");
+
+ // Retrying a name that is not a failed index is a 404, not a silent ok.
+ let (status, err) = send_json(&app, "POST", "/_cluster/indices/failed/broken/_retry").await;
+ assert_eq!(status, StatusCode::NOT_FOUND, "{err}");
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The boundary half: making a failed index visible must not make it visible to
+// everyone. A brain a scoped key may not read stays unreadable when it breaks.
+// ─────────────────────────────────────────────────────────────────────────────
+
+const ADMIN_KEY: &str = "admin-secret-key-for-failed-index-test";
+
+async fn send_auth(
+ app: &axum::Router,
+ method: &str,
+ uri: &str,
+ auth: &str,
+) -> (StatusCode, String) {
+ let req = Request::builder()
+ .method(method)
+ .uri(uri)
+ .header("content-type", "application/json")
+ .header("authorization", auth)
+ .body(Body::empty())
+ .expect("request");
+ let resp = app.clone().oneshot(req).await.expect("response");
+ let status = resp.status();
+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();
+ (status, String::from_utf8_lossy(&bytes).into_owned())
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_failed_brain_is_not_leaked_to_a_scoped_key() {
+ // A brain and an ordinary index; the brain's manifest is corrupted so it
+ // is a FAILED index on the boot that serves the requests below.
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ config.auth.enabled = true;
+ config.auth.admin_api_key = ADMIN_KEY.to_string();
+ {
+ let engine = Engine::new(config.clone()).expect("first boot");
+ for name in [".xerj-memory-bob-edges", "logs-2026"] {
+ engine.create_index(name, Schema::empty()).unwrap();
+ let idx = engine.get_index(name).unwrap();
+ idx.index_document(Some("1".into()), serde_json::json!({"t": "x"}))
+ .await
+ .unwrap();
+ engine.flush_index(name).await.unwrap();
+ }
+ }
+ std::fs::write(
+ dir.path()
+ .join(".xerj-memory-bob-edges")
+ .join("snapshot.json"),
+ b"{not json",
+ )
+ .unwrap();
+ let engine = Engine::new(config.clone()).expect("reboot");
+ let state = AppState::new(config, engine, Metrics::new().expect("metrics"));
+ let app = build_es_compat_router(state);
+
+ let admin = format!("ApiKey {ADMIN_KEY}");
+ let mint = Request::builder()
+ .method("POST")
+ .uri("/_security/api_key")
+ .header("content-type", "application/json")
+ .header("authorization", &admin)
+ .body(Body::from(
+ r#"{"name":"logs-agent","role_descriptors":{"logs":{"indices":[
+ {"names":["logs-2026"],"privileges":["read","write"]}]}}}"#,
+ ))
+ .expect("request");
+ let resp = app.clone().oneshot(mint).await.expect("response");
+ assert_eq!(resp.status(), StatusCode::OK);
+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();
+ let minted: Value = serde_json::from_slice(&bytes).expect("json");
+ let agent = format!("ApiKey {}", minted["encoded"].as_str().expect("encoded"));
+
+ for uri in [
+ "/_cat/indices",
+ "/_cat/shards",
+ "/_cluster/state",
+ "/_cluster/health?level=indices",
+ "/_cluster/indices/failed",
+ ] {
+ let (status, body) = send_auth(&app, "GET", uri, &agent).await;
+ assert!(status.is_success(), "{uri}: {status} {body}");
+ assert!(
+ !body.contains("bob"),
+ "{uri} leaked a failed brain to a scoped key: {body}"
+ );
+ }
+
+ // The admin still sees it — "prune everything" would pass the test above.
+ let (status, body) = send_auth(&app, "GET", "/_cluster/indices/failed", &admin).await;
+ assert!(status.is_success(), "{body}");
+ assert!(body.contains("bob"), "the admin must still see it: {body}");
+}
diff --git a/engine/crates/xerj-common/src/error.rs b/engine/crates/xerj-common/src/error.rs
index 86d57fb6e..aecb43bd8 100644
--- a/engine/crates/xerj-common/src/error.rs
+++ b/engine/crates/xerj-common/src/error.rs
@@ -22,6 +22,20 @@ pub enum XerjError {
#[error("index already exists: {name}")]
IndexAlreadyExists { name: String },
+ /// The index directory exists but could not be opened, so the index
+ /// cannot serve. Deliberately distinct from [`XerjError::IndexNotFound`]:
+ /// the name is real, the data is still on disk, and the operator's move is
+ /// to fix the cause and retry — or to delete it. Reporting these as 404
+ /// "not found" is what made a failed index un-inspectable and
+ /// un-deletable (issue #206).
+ #[error(
+ "index [{index}] failed to open and is unavailable: {reason}. \
+ Inspect it with GET /_cluster/indices/failed, retry it with \
+ POST /_cluster/indices/failed/{index}/_retry once the cause is fixed, \
+ or remove it with DELETE /{index}"
+ )]
+ IndexUnavailable { index: String, reason: String },
+
// ── Document operations ────────────────────────────────────────────────
/// A document with the given ID was not found in the index.
#[error("document not found: id={id} in index={index}")]
@@ -137,6 +151,13 @@ impl XerjError {
Self::IndexAlreadyExists { name: name.into() }
}
+ pub fn index_unavailable(index: impl Into<String>, reason: impl Into<String>) -> Self {
+ Self::IndexUnavailable {
+ index: index.into(),
+ reason: reason.into(),
+ }
+ }
+
pub fn document_not_found(id: impl Into<String>, index: impl Into<String>) -> Self {
Self::DocumentNotFound {
id: id.into(),
@@ -266,6 +287,9 @@ impl XerjError {
| Self::ConfigError { .. }
| Self::ResultWindowTooLarge { .. } => 400,
Self::IndexBlocked { .. } => 403,
+ // The index exists but has no serving copy — ES answers the same
+ // shape (no_shard_available_action_exception) with 503.
+ Self::IndexUnavailable { .. } => 503,
Self::AuthError { .. } => 401,
Self::ResourceExhausted { .. } | Self::CircuitBreaking { .. } => 429,
Self::StorageError { .. }
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index 136c30deb..6e0f19cfa 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -98,6 +98,28 @@ pub struct IndexInfo {
pub schema_version: u64,
}
+/// An index directory that is present on disk but could not be opened.
+///
+/// Before issue #206 a failed open left only `name → reason` in a private map
+/// that nothing but [`Engine::health`] ever read: the index was invisible to
+/// `_cat/indices` and `_cluster/state`, `DELETE` answered 404, and the only
+/// recovery was to stop the server and edit the data directory by hand. A
+/// failed index is now a real, addressable state — it is listed, it carries
+/// the open error verbatim, it can be deleted, and it can be retried once the
+/// operator has fixed the cause.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct FailedIndex {
+ /// Index name, i.e. the directory name under `data_dir`.
+ pub name: String,
+ /// The verbatim error from the last open attempt. Never summarised —
+ /// the storage layer's message already names the file and the fix.
+ pub reason: String,
+ /// Wall-clock ms of the first failure (boot, restore, or a failed retry).
+ pub failed_at_ms: i64,
+ /// How many explicit retries have been attempted since (0 at first boot).
+ pub retries: u32,
+}
+
/// Overall engine health.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct HealthStatus {
@@ -297,8 +319,11 @@ pub struct Engine {
pub search_templates: Arc<DashMap<String, Value>>,
/// async search id → stored result JSON
pub async_searches: Arc<DashMap<String, Value>>,
- /// Names of index directories that failed to open on startup (health = red).
- pub failed_indices: Arc<DashMap<String, String>>,
+ /// Index directories that are present but could not be opened, keyed by
+ /// index name (health = red). See [`FailedIndex`] — these are inspectable
+ /// (`GET /_cluster/indices/failed`), deletable (`DELETE /{index}`) and
+ /// retryable (`POST /_cluster/indices/failed/{index}/_retry`).
+ pub failed_indices: Arc<DashMap<String, FailedIndex>>,
/// transform id → transform definition JSON
pub transforms: Arc<DashMap<String, Value>>,
/// index_name → frozen state (true = frozen / read-only)
@@ -517,7 +542,7 @@ impl Engine {
}
Err(e) => {
warn!(name = name_str.as_str(), error = %e, "failed to open index");
- engine.failed_indices.insert(name_str, e.to_string());
+ engine.record_failed_index(&name_str, e.to_string());
}
}
}
@@ -831,6 +856,16 @@ impl Engine {
));
}
+ // The name is taken by an index that exists on disk but would not
+ // open. Creating over it would run the store open again and surface a
+ // storage error that says nothing about the operator's options, so
+ // refuse here with the recorded reason instead (issue #206).
+ if let Some(f) = self.failed_indices.get(name) {
+ return Err(EngineError::Common(
+ xerj_common::XerjError::index_unavailable(name, f.reason.clone()),
+ ));
+ }
+
// Apply matching template (highest priority wins) on every create path.
let effective_schema = self.apply_index_template(name, schema);
let idx = Index::create_with_settings(
@@ -1177,6 +1212,11 @@ impl Engine {
/// Also drops any aliases that pointed only at this index (matching ES
/// semantics) and clears the `closed_indices` flag so the name is
/// truly gone when another test recreates it.
+ ///
+ /// A **failed** index (present on disk, refused at open — see
+ /// [`FailedIndex`]) is deletable through this same door. It used to answer
+ /// 404 `index_not_found`, which left an operator with no lever but
+ /// stopping the server and removing the directory by hand (issue #206).
pub async fn delete_index(&self, name: &str) -> Result<()> {
// Per-index authorization backstop (issue #79) — "destroy the brain"
// is the loudest door, so it is checked before the index is removed
@@ -1186,13 +1226,37 @@ impl Engine {
xerj_common::XerjError::index_not_found(name),
));
}
- let idx =
- self.indices.remove(name).map(|(_, v)| v).ok_or_else(|| {
- EngineError::Common(xerj_common::XerjError::index_not_found(name))
- })?;
+ match self.indices.remove(name).map(|(_, v)| v) {
+ Some(idx) => idx.delete_all_data().await?,
+ None => {
+ // Not open. If it is a known failed index, its bytes are still
+ // on disk and removing them is exactly what the operator asked
+ // for; anything else is a genuine 404.
+ if !self.failed_indices.contains_key(name) {
+ return Err(EngineError::Common(
+ xerj_common::XerjError::index_not_found(name),
+ ));
+ }
+ // Drop the bookkeeping only after the bytes are gone. Removing
+ // it first would take a delete that failed (read-only mount,
+ // fs error) out of the failed list while its directory
+ // survives — the operator would believe the name was freed and
+ // the index would reappear on the next boot.
+ self.remove_index_dir(name)?;
+ self.failed_indices.remove(name);
+ }
+ }
- idx.delete_all_data().await?;
+ self.forget_index_metadata(name);
+ info!(name, "index deleted");
+ Ok(())
+ }
+ /// Drop every piece of engine-side metadata that names `index` — aliases
+ /// that pointed only at it, the closed flag, settings, mappings, alias
+ /// metadata. Shared by the open-index and failed-index delete paths so the
+ /// two cannot drift.
+ fn forget_index_metadata(&self, name: &str) {
// Remove this index from every alias that references it; drop the
// alias entirely when its backing list becomes empty.
let empty_aliases: Vec<String> = self
@@ -1216,11 +1280,111 @@ impl Engine {
self.index_settings.remove(name);
self.index_mappings.remove(name);
self.index_alias_metadata.remove(name);
+ }
- info!(name, "index deleted");
+ /// Remove `<data_dir>/<name>` from disk, refusing anything that does not
+ /// resolve inside `data_dir`.
+ ///
+ /// The open-index path deletes through `Index::delete_all_data`, which can
+ /// only ever point at a directory the engine itself built. The failed-index
+ /// path has no `Index` to ask, so the name is re-validated here and the
+ /// resolved path is proven to be under `data_dir` before anything is
+ /// removed — the same canonicalisation rule the snapshot-restore path
+ /// applies before it writes.
+ fn remove_index_dir(&self, name: &str) -> Result<()> {
+ // Reject traversal/absolute forms up front: only a legal index name
+ // can name a directory we own.
+ IndexName::new(name).map_err(EngineError::Common)?;
+ let dir = self.data_dir.join(name);
+ if !dir.exists() {
+ return Ok(());
+ }
+ let dir_canon = dir.canonicalize().map_err(EngineError::Io)?;
+ let root_canon = self.data_dir.canonicalize().map_err(EngineError::Io)?;
+ if !dir_canon.starts_with(&root_canon) {
+ return Err(EngineError::Common(xerj_common::XerjError::storage(
+ format!("refusing to delete index [{name}] outside data_dir (canonical)"),
+ )));
+ }
+ std::fs::remove_dir_all(&dir_canon).map_err(EngineError::Io)?;
Ok(())
}
+ // ── Failed-index recovery (issue #206) ────────────────────────────────────
+
+ /// Record (or re-record) an index that could not be opened.
+ ///
+ /// Preserves `failed_at_ms` across repeated failures so "since when" stays
+ /// truthful, and counts retries so a flapping directory is visible as
+ /// such.
+ fn record_failed_index(&self, name: &str, reason: String) {
+ match self.failed_indices.get_mut(name) {
+ Some(mut existing) => {
+ existing.reason = reason;
+ existing.retries = existing.retries.saturating_add(1);
+ }
+ None => {
+ self.failed_indices.insert(
+ name.to_string(),
+ FailedIndex {
+ name: name.to_string(),
+ reason,
+ failed_at_ms: now_millis(),
+ retries: 0,
+ },
+ );
+ }
+ }
+ }
+
+ /// Every failed index the caller is allowed to see, sorted by name.
+ pub fn list_failed_indices(&self) -> Vec<FailedIndex> {
+ let mut out: Vec<FailedIndex> = self
+ .failed_indices
+ .iter()
+ .filter(|e| crate::index_guard::visible(e.key()))
+ .map(|e| e.value().clone())
+ .collect();
+ out.sort_by(|a, b| a.name.cmp(&b.name));
+ out
+ }
+
+ /// Re-attempt the open of a failed index.
+ ///
+ /// On success the index becomes a normal, serving index (mapping and date
+ /// flags restored exactly as at boot) and leaves the failed set. On failure
+ /// the recorded reason is refreshed, the retry counter advances, and the
+ /// new error is returned — the operator gets the *current* reason, not the
+ /// one from boot.
+ pub fn retry_failed_index(&self, name: &str) -> Result<()> {
+ if !crate::index_guard::visible(name) || !self.failed_indices.contains_key(name) {
+ return Err(EngineError::Common(
+ xerj_common::XerjError::index_not_found(name),
+ ));
+ }
+ let index_name = IndexName::new(name).map_err(EngineError::Common)?;
+ match Index::open(index_name, &self.config, &self.data_dir) {
+ Ok(idx) => {
+ self.load_persisted_es_mapping(name);
+ if let Some(m) = self.index_mappings.get(name) {
+ Engine::apply_date_mapping_flags(&idx, m.value());
+ }
+ self.indices.insert(name.to_string(), idx);
+ self.failed_indices.remove(name);
+ info!(name, "failed index reopened");
+ Ok(())
+ }
+ Err(e) => {
+ let reason = e.to_string();
+ self.record_failed_index(name, reason.clone());
+ warn!(name, error = %reason, "retry of failed index did not succeed");
+ Err(EngineError::Common(
+ xerj_common::XerjError::index_unavailable(name, reason),
+ ))
+ }
+ }
+ }
+
/// Get a reference to an index by name, resolving aliases first.
/// If the name is an alias pointing to multiple indices, returns the first one.
///
@@ -1245,9 +1409,7 @@ impl Engine {
.indices
.get(real_name.as_str())
.map(|r| Arc::clone(r.value()))
- .ok_or_else(|| {
- EngineError::Common(xerj_common::XerjError::index_not_found(real_name))
- });
+ .ok_or_else(|| EngineError::Common(self.missing_index_error(real_name)));
}
}
if !crate::index_guard::visible(name) {
@@ -1258,7 +1420,21 @@ impl Engine {
self.indices
.get(name)
.map(|r| Arc::clone(r.value()))
- .ok_or_else(|| EngineError::Common(xerj_common::XerjError::index_not_found(name)))
+ .ok_or_else(|| EngineError::Common(self.missing_index_error(name)))
+ }
+
+ /// The error for a name that is not in `indices`.
+ ///
+ /// A name that failed to open is **not** "not found": the directory is
+ /// still there and the operator has a lever. Reporting a 404 for it is how
+ /// issue #206's stuck state stayed invisible, so a failed index answers
+ /// 503 `no_shard_available_action_exception` carrying the open error and
+ /// the three commands that act on it.
+ fn missing_index_error(&self, name: &str) -> xerj_common::XerjError {
+ match self.failed_indices.get(name) {
+ Some(f) => xerj_common::XerjError::index_unavailable(name, f.reason.clone()),
+ None => xerj_common::XerjError::index_not_found(name),
+ }
}
/// Return an index by name, creating it if it doesn't exist (ES behaviour).
@@ -1278,6 +1454,15 @@ impl Engine {
if let Ok(idx) = self.get_index(name) {
return Ok(idx);
}
+ // A failed index already occupies this name and its bytes are still on
+ // disk. Auto-creating over it would either destroy recoverable data or
+ // fail deep inside the store with an opaque message — refuse loudly
+ // with the open reason and the operator's options instead (issue #206).
+ if let Some(f) = self.failed_indices.get(name) {
+ return Err(EngineError::Common(
+ xerj_common::XerjError::index_unavailable(name, f.reason.clone()),
+ ));
+ }
// Auto-create with empty schema.
self.create_index(name, Schema::empty())?;
self.get_index(name)
@@ -1974,7 +2159,7 @@ impl Engine {
}
Err(e) => {
warn!(index = idx_name, error = %e, "failed to reopen restored index");
- self.failed_indices.insert(idx_name.clone(), e.to_string());
+ self.record_failed_index(idx_name, e.to_string());
}
}
}
diff --git a/engine/crates/xerj-engine/tests/failed_index_recovery.rs b/engine/crates/xerj-engine/tests/failed_index_recovery.rs
new file mode 100644
index 000000000..e7257800a
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/failed_index_recovery.rs
@@ -0,0 +1,195 @@
+//! Issue #206: a failed index is a **state**, not a dead entry.
+//!
+//! Before this, an index whose directory refused to open at boot was recorded
+//! in a private map that only `Engine::health` ever read. Live-reproduced at
+//! `12d5cc32` with a corrupt `snapshot.json` (the storage layer's documented
+//! "present but unparseable manifest" refusal):
+//!
+//! ```text
+//! failed_indices = [("broken", "storage error: … could not be parsed …")]
+//! indices = [] ← invisible
+//! health = "red"
+//! delete = Some(IndexNotFound { name: "broken" }) ← undeletable
+//! ```
+//!
+//! So the index could not be listed, could not be deleted, and could not be
+//! retried: the only lever was stopping the server and editing the data
+//! directory by hand. Every assertion below fails on that build.
+//!
+//! The corruption used here is deliberate and realistic — `IndexStore::open`
+//! refuses a manifest it cannot parse rather than treating it as empty,
+//! because treating it as empty would orphan and then delete every segment.
+//! That refusal is correct; having no way to act on it afterwards was not.
+
+use tempfile::TempDir;
+use xerj_common::config::Config;
+use xerj_common::types::Schema;
+use xerj_common::XerjError;
+use xerj_engine::{Engine, EngineError};
+
+fn config_for(dir: &TempDir) -> Config {
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ config
+}
+
+/// Build a data dir holding `names`, each with one flushed document, then
+/// corrupt `break_me`'s manifest so the next boot refuses to open it.
+/// Returns the good manifest bytes so a test can "fix the cause" and retry.
+async fn data_dir_with_one_broken_index(names: &[&str], break_me: &str) -> (TempDir, Vec<u8>) {
+ let dir = TempDir::new().unwrap();
+ {
+ let engine = Engine::new(config_for(&dir)).expect("first boot");
+ for name in names {
+ engine.create_index(name, Schema::empty()).unwrap();
+ let idx = engine.get_index(name).unwrap();
+ idx.index_document(Some("1".to_string()), serde_json::json!({"t": "hello"}))
+ .await
+ .unwrap();
+ engine.flush_index(name).await.unwrap();
+ }
+ }
+ let manifest = dir.path().join(break_me).join("snapshot.json");
+ let good = std::fs::read(&manifest).expect("a flushed index must have a manifest");
+ std::fs::write(&manifest, b"{not json").unwrap();
+ (dir, good)
+}
+
+fn xerj_error(e: EngineError) -> XerjError {
+ e.into()
+}
+
+/// `Index` is not `Debug`, so `unwrap_err()` is unavailable on the
+/// index-returning calls. Take the error explicitly instead.
+fn err_of<T>(r: Result<T, EngineError>, what: &str) -> XerjError {
+ match r {
+ Ok(_) => panic!("{what} must not succeed"),
+ Err(e) => xerj_error(e),
+ }
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_failed_index_is_listed_with_its_reason() {
+ let (dir, _good) = data_dir_with_one_broken_index(&["broken", "healthy"], "broken").await;
+ let engine = Engine::new(config_for(&dir)).expect("boot with one broken index");
+
+ let failed = engine.list_failed_indices();
+ assert_eq!(
+ failed.len(),
+ 1,
+ "expected exactly one failed index: {failed:?}"
+ );
+ let f = &failed[0];
+ assert_eq!(f.name, "broken");
+ assert!(
+ f.reason.contains("snapshot.json") && f.reason.contains("could not be parsed"),
+ "the reason must be the verbatim open error, not a summary: {}",
+ f.reason
+ );
+ assert!(f.failed_at_ms > 0, "a failed index records when it failed");
+ assert_eq!(f.retries, 0, "no retry has been attempted yet");
+
+ // The healthy index in the same data dir is unaffected — one broken index
+ // must not cost the node the others.
+ assert_eq!(engine.list_indices().await.len(), 1);
+ assert!(engine.get_index("healthy").is_ok());
+ assert_eq!(engine.health().await.status, "red");
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn reads_and_writes_to_a_failed_index_say_unavailable_not_not_found() {
+ let (dir, _good) = data_dir_with_one_broken_index(&["broken"], "broken").await;
+ let engine = Engine::new(config_for(&dir)).expect("boot");
+
+ // `index_not_found` (404) was a lie: the name is taken and the bytes are
+ // on disk. It is now 503 `no_shard_available_action_exception` carrying
+ // the open error.
+ let e = err_of(engine.get_index("broken"), "get_index on a failed index");
+ assert!(
+ matches!(e, XerjError::IndexUnavailable { .. }),
+ "get_index on a failed index must not report not-found: {e:?}"
+ );
+ assert_eq!(e.http_status(), 503);
+ assert!(e.to_string().contains("snapshot.json"), "{e}");
+
+ // Auto-create must not run over a failed index — that would either destroy
+ // recoverable segments or fail deep in the store with an opaque message.
+ let e = err_of(
+ engine.get_or_create_index("broken"),
+ "auto-create over a failed index",
+ );
+ assert!(
+ matches!(e, XerjError::IndexUnavailable { .. }),
+ "auto-create must refuse a failed index: {e:?}"
+ );
+ // …and the refusal must not have created anything.
+ assert!(engine.list_indices().await.is_empty());
+ assert_eq!(engine.list_failed_indices().len(), 1);
+
+ // Explicit create is refused for the same reason, with the same message.
+ let e = xerj_error(engine.create_index("broken", Schema::empty()).unwrap_err());
+ assert!(
+ matches!(e, XerjError::IndexUnavailable { .. }),
+ "create over a failed index must name the real cause: {e:?}"
+ );
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_failed_index_can_be_deleted_without_a_restart() {
+ let (dir, _good) = data_dir_with_one_broken_index(&["broken", "healthy"], "broken").await;
+ let engine = Engine::new(config_for(&dir)).expect("boot");
+ let index_dir = dir.path().join("broken");
+ assert!(index_dir.exists());
+
+ engine
+ .delete_index("broken")
+ .await
+ .expect("a failed index must be deletable");
+
+ assert!(!index_dir.exists(), "delete must remove the bytes on disk");
+ assert!(engine.list_failed_indices().is_empty());
+ // Health recovers without a restart, and the name is reusable.
+ assert_ne!(engine.health().await.status, "red");
+ engine
+ .create_index("broken", Schema::empty())
+ .expect("the freed name must be reusable");
+
+ // A name that was never an index is still a plain 404.
+ let e = xerj_error(engine.delete_index("never-existed").await.unwrap_err());
+ assert!(matches!(e, XerjError::IndexNotFound { .. }), "{e:?}");
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn retry_reports_the_live_reason_and_succeeds_once_the_cause_is_fixed() {
+ let (dir, good) = data_dir_with_one_broken_index(&["broken"], "broken").await;
+ let engine = Engine::new(config_for(&dir)).expect("boot");
+
+ // Retry while still broken: the error is returned, not swallowed, and the
+ // attempt is counted so a flapping directory is visible as one.
+ let e = xerj_error(engine.retry_failed_index("broken").unwrap_err());
+ assert!(
+ matches!(e, XerjError::IndexUnavailable { .. }),
+ "a retry that did not work must fail loudly: {e:?}"
+ );
+ let after = engine.list_failed_indices();
+ assert_eq!(after.len(), 1);
+ assert_eq!(after[0].retries, 1, "the retry must be counted");
+
+ // Operator fixes the cause (here: restores the manifest from the backup
+ // the storage error told them to look for) and retries again.
+ std::fs::write(dir.path().join("broken").join("snapshot.json"), &good).unwrap();
+ engine
+ .retry_failed_index("broken")
+ .expect("retry after the fix must reopen the index");
+
+ assert!(engine.list_failed_indices().is_empty());
+ assert_eq!(engine.list_indices().await.len(), 1);
+ assert_ne!(engine.health().await.status, "red");
+ // The index actually serves: the document survived the whole round trip.
+ let idx = engine.get_index("broken").expect("reopened index");
+ assert_eq!(idx.stats().await.doc_count, 1);
+
+ // Retrying a name that is not a failed index is a 404, not a silent ok.
+ let e = xerj_error(engine.retry_failed_index("broken").unwrap_err());
+ assert!(matches!(e, XerjError::IndexNotFound { .. }), "{e:?}");
+}