AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): make a failed index inspectable, deletable and retryable (#206)
xerj-orgd78860eacede
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 00ff4fc62..970f2f6ad 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **A failed index can now be inspected, deleted and retried without stopping+ the server** ([#206](https://github.com/xerj-org/xerj/issues/206)). An index+ directory that refused to open at boot was recorded in a map that only+ `Engine::health` ever read. Live-reproduced with a corrupt `snapshot.json`:+ it was absent from `_cat/indices` and `_cluster/state`, `DELETE /{index}`+ answered `404 index_not_found` and left the bytes on disk, and the only+ recovery was to stop the server and edit the data directory by hand. A+ failed index is now a real state: listed as `red` in `_cat/indices`, present+ in `_cluster/state` as an `UNASSIGNED` primary with `ALLOCATION_FAILED` and+ the verbatim open error, enumerated with its reason by+ `GET /_cluster/indices/failed`, reopenable via+ `POST /_cluster/indices/failed/{name}/_retry` once the cause is fixed, and+ deletable through the ordinary `DELETE /{index}`. Reads, writes and creates+ against it return `503 no_shard_available_action_exception` carrying the+ reason instead of a `404` that claimed it did not exist.++ Two related defects in the same report are fixed with it. **Readiness no+ longer hard-fails on a partly degraded node** — `/health/ready` returned+ `503` for any red status, so one broken index pulled a pod holding 200+ healthy ones out of service permanently; it now reports `200 ready+ (degraded)` while the node can still serve something and `503` only when+ every index it holds failed to open. And **`_cat/health` no longer prints a+ hardcoded `green`** — the one health surface an existing ES dashboard points+ at was the one that could not report a broken node; it and `_cluster/health`+ now count an unopenable index as an unassigned primary, agreeing with+ `/v1/health` and `/v1/cluster/health`.+- **`autoindex` no longer leaves immortal catalog entries for skipped files**([#238](https://github.com/xerj-org/xerj/issues/238)). A file added after theresume plan was frozen is skipped and reported in the catalog — but thatdiff --git a/docs/recipes/production-deployment.md b/docs/recipes/production-deployment.mdindex 9a3dd865c..a13182e58 100644--- a/docs/recipes/production-deployment.md+++ b/docs/recipes/production-deployment.md@@ -183,6 +183,57 @@ container. If you enable in-process TLS, change the probe toyour container is marked unhealthy forever. Same for a Kubernetes probe: set`scheme: HTTPS` on the `httpGet`.+### What readiness actually means++`/health/ready` answers **"can this node serve?"**, not "is every index+perfect". A node with 200 healthy indices and one that will not open stays+`200` and keeps taking traffic, with the degradation named in the body:++```text+ready (degraded): 200 of 201 indices serving, 1 failed to open; see GET /_cluster/indices/failed+```++The body carries counts only — this path is exempt from authentication, so it+must not enumerate index names to anything that can reach the port. The names+and reasons live behind auth on `GET /_cluster/indices/failed`.++It returns `503` only when there is genuinely nothing to serve — every index+the node holds failed to open. This is deliberate: readiness used to hard-fail+on any red status, so one bad index directory removed the whole pod from+service permanently and took the 200 working indices down with it. The red+status itself is untouched — `/v1/health`, `/v1/cluster/health`,+`_cluster/health` and `_cat/health` all still report `red`, so alert on those,+not on the probe.++### Recovering an index that will not open++An index directory that fails to open at boot is quarantined rather than+silently dropped, and it is a state you can act on without stopping the+server. All four moves work on a live node:++```bash+# 1. What failed, and why — the verbatim open error, not just a name+curl -s localhost:9200/_cluster/indices/failed | jq .++# 2. It is visible in the surfaces you already watch+curl -s localhost:9200/_cat/indices # → red open broken <uuid> 1 0 0 0 0b 0b 0b+curl -s "localhost:9200/_cluster/health?level=indices" | jq '.indices.broken'++# 3. Fix the cause the error names (e.g. restore snapshot.json from a+# backup or from an interrupted-write `.tmp.*` sibling), then retry+curl -s -XPOST localhost:9200/_cluster/indices/failed/broken/_retry++# 4. Or give up on it and remove it+curl -s -XDELETE localhost:9200/broken+```++A retry that does not work returns `503` with the *current* reason, not the+one recorded at boot. While an index is in this state, reads and writes to it+return `503 no_shard_available_action_exception` naming the open error —+**not** `404 index_not_found`, which would be a lie: the name is taken and the+data is still on disk. Creating an index over that name is refused for the+same reason.+### gRPC postureThe gRPC `XerjSearch` service (default `:8081`) always speaks **plaintextdiff --git a/engine/crates/xerj-api/src/authz.rs b/engine/crates/xerj-api/src/authz.rsindex b6a3df603..af6ce235d 100644--- a/engine/crates/xerj-api/src/authz.rs+++ b/engine/crates/xerj-api/src/authz.rs@@ -1471,7 +1471,18 @@ async fn prune_response(// other structural key would be dropped for a scoped principal, which// would corrupt the very responses this is protecting. Read outside the// request's visibility scope on purpose: this needs the unfiltered truth.- let known: HashSet<String> = state.engine.index_name_list().into_iter().collect();+ // Failed indices count as real names here (issue #206). They are now+ // listed by `_cat/indices`, `_cat/shards` and `_cluster/state`, and a+ // name that is not in this set is never a pruning candidate — so leaving+ // them out would hand a scoped principal the name of a brain that+ // happened to fail to open. The engine-side guard already filters them at+ // the handler; this is the same second pass the open indices get.+ let known: HashSet<String> = state+ .engine+ .index_name_list()+ .into_iter()+ .chain(state.engine.failed_indices.iter().map(|e| e.key().clone()))+ .collect();let pruned: Vec<u8> = if is_json {match serde_json::from_slice::<Value>(&bytes) {diff --git a/engine/crates/xerj-api/src/error.rs b/engine/crates/xerj-api/src/error.rsindex cd02ff37f..29fa420a0 100644--- a/engine/crates/xerj-api/src/error.rs+++ b/engine/crates/xerj-api/src/error.rs@@ -264,6 +264,7 @@ impl IntoResponse for ApiError {/// |---|---|---|/// | `IndexNotFound` | `index_not_found_exception` | 404 |/// | `IndexAlreadyExists` | `resource_already_exists_exception` | 409 |+/// | `IndexUnavailable` | `no_shard_available_action_exception` | 503 |/// | `DocumentNotFound` | `document_missing_exception` | 404 |/// | `InvalidMapping` | `mapper_parsing_exception` | 400 |/// | `InvalidQuery` | `search_phase_execution_exception` | 400 |@@ -284,6 +285,10 @@ fn xerj_error_type(e: &XerjError) -> String {// ── Index lifecycle ───────────────────────────────────────────────XerjError::IndexNotFound { .. } => "index_not_found_exception",XerjError::IndexAlreadyExists { .. } => "resource_already_exists_exception",+ // The index exists; its only copy cannot be opened. ES reports an+ // existing-but-unallocated index this way, so dashboards that already+ // understand a red shard understand this too.+ XerjError::IndexUnavailable { .. } => "no_shard_available_action_exception",// ── Document operations ───────────────────────────────────────────XerjError::DocumentNotFound { .. } => "document_missing_exception",// ── Mapping / schema ──────────────────────────────────────────────@@ -328,7 +333,9 @@ fn extract_index_name(e: &XerjError) -> Option<String> {Some(name.clone())}XerjError::DocumentNotFound { index, .. } => Some(index.clone()),- XerjError::IndexBlocked { index, .. } => Some(index.clone()),+ XerjError::IndexBlocked { index, .. } | XerjError::IndexUnavailable { index, .. } => {+ Some(index.clone())+ }_ => None,}}diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 91c93f51d..dc1fc533d 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -184,6 +184,36 @@ pub async fn cluster_health_for_index(cluster_health_inner(state, Some(index), params).await}+/// The `unassigned_info` block for an index whose only primary could not be+/// opened (issue #206).+///+/// Field names and shape follow Elasticsearch's own `UnassignedInfo`+/// serialisation — `reason`, `at` as an ISO-8601 instant, `failed_attempts`+/// only when there has been one, `delayed`, `details`, `allocation_status`+/// (`elasticsearch/server/src/main/java/org/elasticsearch/cluster/routing/UnassignedInfo.java:483-503`,+/// read for semantics only; Elasticsearch is licence-incompatible with this+/// project and nothing was copied). Matching the names is the point of the+/// exercise: the operator's existing dashboard already knows how to render+/// this block, and `details` is where the verbatim open error goes.+fn unassigned_info_json(f: &xerj_engine::engine::FailedIndex) -> Value {+ let mut info = json!({+ "reason": "ALLOCATION_FAILED",+ "at": Utc.timestamp_millis_opt(f.failed_at_ms)+ .single()+ .map(|t| t.format("%Y-%m-%dT%H:%M:%S%.3fZ").to_string())+ .unwrap_or_default(),+ "delayed": false,+ "details": f.reason,+ // The primary's on-disk copy exists but cannot be opened, so there is+ // no copy this node can allocate — ES's `no_valid_shard_copy`.+ "allocation_status": "no_valid_shard_copy",+ });+ if f.retries > 0 {+ info["failed_attempts"] = json!(f.retries);+ }+ info+}+async fn cluster_health_inner(state: AppState,index_filter: Option<String>,@@ -248,6 +278,24 @@ async fn cluster_health_inner(selected};+ // Indices that exist on disk but refused to open. Their primary has no+ // serving copy, which is the textbook definition of a red cluster — and+ // the one condition this endpoint used to ignore entirely, so an operator+ // watching the ES-compatible surface never saw the failure that+ // `/v1/health` was already reporting (issue #206).+ let selected_failed: Vec<xerj_engine::engine::FailedIndex> = state+ .engine+ .list_failed_indices()+ .into_iter()+ .filter(|f| match &index_filter {+ None => true,+ Some(sel) => sel+ .split(',')+ .map(str::trim)+ .any(|s| s == "_all" || s == "*" || s == f.name || glob_match_simple(s, &f.name)),+ })+ .collect();+// Per-index shard count helper — defaults to 1 when unset.let shard_count = |name: &str| -> u32 {state@@ -267,7 +315,7 @@ async fn cluster_health_inner(.unwrap_or(1) as u32};- let idx_count = selected.len() as u32;+ let idx_count = (selected.len() + selected_failed.len()) as u32;let mut closed_count = 0u32;let mut unassigned_replicas: u32 = 0;for info in &selected {@@ -314,12 +362,17 @@ async fn cluster_health_inner(}}- // Any unassigned replica forces yellow; closed indices surface as yellow- // in the legacy pre-7.2 path, but post-7.2 replicated-closed semantics- // keep the *cluster* status driven by replicas only. Our tests cover- // both — we currently only track a single "closed" flag per index and- // don't differentiate the closed-replication mode.- let status = if unassigned_replicas > 0 {+ // An unopenable index is an unassigned PRIMARY — red, and it outranks+ // every yellow condition. Any unassigned replica forces yellow; closed+ // indices surface as yellow in the legacy pre-7.2 path, but post-7.2+ // replicated-closed semantics keep the *cluster* status driven by+ // replicas only. Our tests cover both — we currently only track a single+ // "closed" flag per index and don't differentiate the closed-replication+ // mode.+ let unassigned_primaries = selected_failed.len() as u32;+ let status = if unassigned_primaries > 0 {+ "red"+ } else if unassigned_replicas > 0 {"yellow"} else {"green"@@ -360,12 +413,15 @@ async fn cluster_health_inner(}).unwrap_or(Some(1)).unwrap_or(1);- // wait_for_active_shards unmet: `all` when there are any unassigned- // replicas, or a numeric count greater than the currently active- // shards. The HTTP `timeout` has already elapsed by the time this- // check runs (we don't block), so we set timed_out accordingly.+ // wait_for_active_shards unmet: `all` when any shard is unassigned —+ // a replica, or (now that a failed index is a reachable state, issue+ // #206) an unopenable primary, which is the more serious of the two and+ // would otherwise have satisfied "all shards active" on a red cluster.+ // A numeric count is unmet when it exceeds the currently active shards.+ // The HTTP `timeout` has already elapsed by the time this check runs (we+ // don't block), so we set timed_out accordingly.let wait_for_active_shards_unmet = match params.wait_for_active_shards.as_deref() {- Some("all") => unassigned_replicas > 0,+ Some("all") => unassigned_replicas > 0 || unassigned_primaries > 0,Some(s) => s.parse::<u64>().ok()@@ -412,8 +468,8 @@ async fn cluster_health_inner("active_shards": active,"relocating_shards": 0,"initializing_shards": 0,- "unassigned_shards": unassigned_replicas,- "unassigned_primary_shards": 0,+ "unassigned_shards": unassigned_replicas + unassigned_primaries,+ "unassigned_primary_shards": unassigned_primaries,"delayed_unassigned_shards": 0,"number_of_pending_tasks": 0,"number_of_in_flight_fetch": 0,@@ -475,6 +531,36 @@ async fn cluster_health_inner(}indices_map.insert(info.name.clone(), idx_obj);}+ // Failed indices carry the open error verbatim in `unassigned_info`,+ // so `?level=indices` answers "which index, and why" in one call.+ for f in &selected_failed {+ let mut idx_obj = json!({+ "status": "red",+ "number_of_shards": 1,+ "number_of_replicas": 0,+ "active_primary_shards": 0,+ "active_shards": 0,+ "relocating_shards": 0,+ "initializing_shards": 0,+ "unassigned_shards": 1,+ "unassigned_primary_shards": 1,+ "unassigned_info": unassigned_info_json(f),+ });+ if level == "shards" {+ idx_obj["shards"] = json!({+ "0": {+ "status": "red",+ "primary_active": false,+ "active_shards": 0,+ "relocating_shards": 0,+ "initializing_shards": 0,+ "unassigned_shards": 1,+ "unassigned_primary_shards": 1,+ }+ });+ }+ indices_map.insert(f.name.clone(), idx_obj);+ }resp["indices"] = Value::Object(indices_map);}@@ -510,6 +596,11 @@ pub async fn cat_indices_pattern(async fn cat_indices_inner(state: AppState, pattern: Option<String>) -> axum::response::Response {let indices = state.engine.list_indices().await;+ // Indices whose directory exists but refused to open. They are listed as+ // `red` rather than omitted: an index the operator can see is an index the+ // operator can delete or retry, and omitting them is what made a failed+ // index invisible to every dashboard (issue #206).+ let failed = state.engine.list_failed_indices();// Narrow to the path pattern when present (`/_cat/indices/{pattern}`):// a comma-separated list of concrete names and/or `*` globs. ES rules:@@ -526,7 +617,10 @@ async fn cat_indices_inner(state: AppState, pattern: Option<String>) -> axum::re.collect();for &p in &parts {let is_wildcard = p == "_all" || p == "*" || p.contains('*');- if !is_wildcard && !indices.iter().any(|i| i.name == p) {+ if !is_wildcard+ && !indices.iter().any(|i| i.name == p)+ && !failed.iter().any(|f| f.name == p)+ {return ApiError::new(xerj_common::XerjError::index_not_found(p)).into_response();}@@ -569,6 +663,33 @@ async fn cat_indices_inner(state: AppState, pattern: Option<String>) -> axum::rehsize,));}++ // Failed indices, after the healthy ones. `red` is the status column and+ // the shard is unassigned, so `pri` is 1 and every stat column is 0 —+ // nothing about the contents can be read, and guessing a doc count here+ // would be a claim we cannot back.+ let selected_failed: Vec<&xerj_engine::engine::FailedIndex> = match pattern.as_deref() {+ None => failed.iter().collect(),+ Some(pat) => failed+ .iter()+ .filter(|f| {+ pat.split(',')+ .map(str::trim)+ .filter(|s| !s.is_empty())+ .any(|p| {+ p == "_all" || p == "*" || f.name == p || glob_match_simple(p, &f.name)+ })+ })+ .collect(),+ };+ for f in selected_failed {+ lines.push(format!(+ "red open {} {} 1 0 0 0 0b 0b 0b",+ f.name,+ stable_index_uuid(&f.name),+ ));+ }+// ES returns an empty body (not a bare newline) when nothing matches.let body = if lines.is_empty() {String::new()@@ -1334,7 +1455,17 @@ pub async fn delete_index(.unwrap_or(true);let all = state.engine.list_indices().await;- let all_names: Vec<String> = all.iter().map(|i| i.name.clone()).collect();+ let mut all_names: Vec<String> = all.iter().map(|i| i.name.clone()).collect();+ // A failed index is deletable. Its name has to take part in selector+ // resolution or a literal `DELETE /broken` 404s and a wildcard silently+ // skips it — which is exactly the stuck state issue #206 describes.+ all_names.extend(+ state+ .engine+ .list_failed_indices()+ .into_iter()+ .map(|f| f.name),+ );let mut to_delete: Vec<String> = Vec::new();let parts: Vec<&str> = index@@ -1396,7 +1527,21 @@ pub async fn delete_index(}for name in &to_delete {- let _ = state.engine.delete_index(name).await;+ // Report a delete that did not happen. The result used to be dropped+ // on the floor, so an index whose bytes could not be removed (a failed+ // index on a read-only mount, an fs error) still answered+ // `acknowledged: true` and the operator had no way to know the name+ // was still taken. A concurrent delete that already removed it is the+ // one benign outcome and stays silent.+ match state.engine.delete_index(name).await {+ Ok(()) => {}+ Err(e) => {+ let inner: xerj_common::XerjError = e.into();+ if !matches!(inner, xerj_common::XerjError::IndexNotFound { .. }) {+ return ApiError::new(inner).into_response();+ }+ }+ }// RC4-W4 item 5: drop the index's per-index metric label series so it// doesn't linger for the process lifetime after the index is gone.state.metrics.prune_index_labels(name);@@ -15863,6 +16008,14 @@ async fn cat_ann_inner(// GET /_cat/health// ─────────────────────────────────────────────────────────────────────────────+/// `GET /_cat/health`.+///+/// The status column used to be the literal string `green`, so the one+/// endpoint an operator's existing dashboard points at was the one endpoint+/// that could not report a broken node — `/v1/health`, `/v1/cluster/health`+/// and `/health/ready` all knew, and `_cat/health` said everything was fine+/// (issue #206). It now reports the same status `_cluster/health` computes,+/// from the same inputs.pub async fn cat_health(State(state): State<AppState>) -> impl IntoResponse {let health = state.engine.health().await;// epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent@@ -15873,7 +16026,21 @@ pub async fn cat_health(State(state): State<AppState>) -> impl IntoResponse {let now = Utc::now();let ts = now.format("%H:%M:%S").to_string();let shards = health.index_count as u32;- let body = format!("{epoch} {ts} xerj green 1 1 {shards} {shards} 0 0 0 0 - 100.0%\n");+ let unassigned = state.engine.list_failed_indices().len() as u32;+ // Mirror `_cluster/health`: an unopenable index is an unassigned primary+ // (red); an unassigned replica is yellow. The engine's own "yellow" means+ // "unflushed memtable", which is a durability nuance rather than an ES+ // shard state, so it deliberately does not colour this column.+ let status = if unassigned > 0 { "red" } else { "green" };+ let total = shards + unassigned;+ let pct = if total == 0 {+ 100.0+ } else {+ (shards as f64) / (total as f64) * 100.0+ };+ let body = format!(+ "{epoch} {ts} xerj {status} 1 1 {shards} {shards} 0 0 {unassigned} 0 - {pct:.1}%\n"+ );(StatusCode::OK,[(@@ -21247,6 +21414,12 @@ pub async fn cat_shards(State(state): State<AppState>) -> impl IntoResponse {info.name, info.doc_count, store_bytes,));}+ // `_cat/shards` is the call an operator makes straight after seeing a red+ // cluster, so an index whose primary could not be opened has to be in it.+ // ES prints UNASSIGNED rows with no node and empty stats.+ for f in state.engine.list_failed_indices() {+ lines.push(format!("{} 0 p UNASSIGNED 0 0b - -", f.name));+ }let body = if lines.is_empty() {String::new()} else {@@ -23340,6 +23513,45 @@ pub async fn cluster_state(State(state): State<AppState>) -> impl IntoResponse {);}+ // An index that would not open still exists — its metadata belongs in+ // cluster state, and its shard is UNASSIGNED with the open error as the+ // allocation-failure detail. Leaving it out is what made a failed index+ // invisible to every tool that reads cluster state (issue #206).+ let mut unassigned_shards: Vec<Value> = Vec::new();+ for f in state.engine.list_failed_indices() {+ metadata_indices.insert(+ f.name.clone(),+ json!({+ "state": "open",+ "settings": {+ "index": {+ "number_of_shards": "1",+ "number_of_replicas": "0",+ "uuid": stable_index_uuid(&f.name),+ "version": { "created": "8130099" },+ "provided_name": f.name,+ }+ },+ "mappings": {},+ "aliases": [],+ }),+ );+ let shard = json!({+ "state": "UNASSIGNED",+ "primary": true,+ "node": null,+ "relocating_node": null,+ "shard": 0,+ "index": f.name,+ "unassigned_info": unassigned_info_json(&f),+ });+ routing_table.insert(+ f.name.clone(),+ json!({ "shards": { "0": [shard.clone()] } }),+ );+ unassigned_shards.push(shard);+ }+Json(json!({"cluster_name": "xerj","cluster_uuid": "xerj-cluster-1",@@ -23363,7 +23575,7 @@ pub async fn cluster_state(State(state): State<AppState>) -> impl IntoResponse {"indices": routing_table,},"routing_nodes": {- "unassigned": [],+ "unassigned": unassigned_shards,"nodes": {node_id: []}diff --git a/engine/crates/xerj-api/src/native.rs b/engine/crates/xerj-api/src/native.rsindex 824dc13b3..c008077bc 100644--- a/engine/crates/xerj-api/src/native.rs+++ b/engine/crates/xerj-api/src/native.rs@@ -708,12 +708,22 @@ pub async fn embedding_execution_identity(State(state): State<AppState>) -> impl// be a no-op — checking the engine here would create a feedback loop// where a slow engine flapping flush causes constant pod restarts.//-// - **readiness**: returns 200 only when the engine reports a non-`red`-// cluster status (i.e. at least one index is queryable). kubelet uses-// this to decide whether to send traffic. Until ready, the Service-// removes the pod from rotation but doesn't restart it — appropriate-// for "still replaying WAL" or "still loading 50 K segments" startup-// states that are transient but visible.+// - **readiness**: returns 200 when the node can serve. kubelet uses this+// to decide whether to send traffic. Until ready, the Service removes the+// pod from rotation but doesn't restart it — appropriate for "still+// replaying WAL" or "still loading 50 K segments" startup states that are+// transient but visible.+//+// Readiness is deliberately NOT "cluster status != red" (issue #206). One+// index that will not open turns the engine red forever, and a pod with 200+// healthy indices and one broken one was therefore pulled out of service+// permanently and never came back — a single-index problem escalated into a+// total outage, with the 199 working indices unreachable. A node is ready+// when it has at least one index it can serve, or nothing to serve at all;+// it is unready only when every index it holds is broken. The red status+// itself is untouched and still reported by `/v1/health`,+// `/v1/cluster/health`, `_cluster/health` and `_cat/health`, and the+// degraded state is named in this body so a probe log says which it is.// ─────────────────────────────────────────────────────────────────────────────pub async fn liveness() -> impl IntoResponse {@@ -722,14 +732,95 @@ pub async fn liveness() -> impl IntoResponse {pub async fn readiness(State(state): State<AppState>) -> impl IntoResponse {let h = state.engine.health().await;- if h.status == "red" {- (+ // Read the node's own state, not a principal's filtered view: this path is+ // exempt from authentication so a kubelet can probe a hardened node, and+ // the probe has to answer for the whole node. For the same reason the body+ // carries COUNTS ONLY — naming the failed indices here would hand every+ // index name on the node to anyone who can reach the port. The names live+ // behind auth on `GET /_cluster/indices/failed`.+ let failed = state.engine.failed_indices.len();+ if failed == 0 {+ return (axum::http::StatusCode::OK, "ready").into_response();+ }+ if h.index_count == 0 {+ // Nothing on this node can be served: every index it holds failed to+ // open. Removing it from rotation is correct here.+ return (axum::http::StatusCode::SERVICE_UNAVAILABLE,- format!("not ready: cluster status = {}", h.status),+ format!(+ "not ready: no index could be opened ({failed} failed); \+ see GET /_cluster/indices/failed"+ ),)- .into_response()- } else {- (axum::http::StatusCode::OK, "ready").into_response()+ .into_response();+ }+ (+ axum::http::StatusCode::OK,+ format!(+ "ready (degraded): {} of {} indices serving, {failed} failed to open; \+ see GET /_cluster/indices/failed",+ h.index_count,+ h.index_count + failed+ ),+ )+ .into_response()+}++// ─────────────────────────────────────────────────────────────────────────────+// Handler: GET /_cluster/indices/failed — list failed indices+// Handler: POST /_cluster/indices/failed/{name}/_retry — retry one open+//+// Issue #206: an index whose directory refused to open had no operator+// surface at all. These two, plus `DELETE /{index}` (which now accepts a+// failed index), are the three moves the issue asks for: inspect with the+// real reason, retry after fixing the cause, delete.+// ─────────────────────────────────────────────────────────────────────────────++pub async fn failed_indices(State(state): State<AppState>) -> impl IntoResponse {+ let started = Instant::now();+ let request_id = Uuid::new_v4().to_string();+ let failed = state.engine.list_failed_indices();+ let items: Vec<serde_json::Value> = failed+ .iter()+ .map(|f| {+ serde_json::json!({+ "index": f.name,+ "reason": f.reason,+ "failed_at_millis": f.failed_at_ms,+ "retries": f.retries,+ "retry": format!("POST /_cluster/indices/failed/{}/_retry", f.name),+ "delete": format!("DELETE /{}", f.name),+ })+ })+ .collect();+ let resp = NativeResponse::new(+ serde_json::json!({ "count": items.len(), "failed_indices": items }),+ started.elapsed().as_millis() as u64,+ &request_id,+ );+ Json(resp).into_response()+}++pub async fn retry_failed_index(+ State(state): State<AppState>,+ axum::extract::Path(name): axum::extract::Path<String>,+) -> impl IntoResponse {+ let started = Instant::now();+ let request_id = Uuid::new_v4().to_string();+ match state.engine.retry_failed_index(&name) {+ Ok(()) => {+ let resp = NativeResponse::new(+ serde_json::json!({ "index": name, "reopened": true }),+ started.elapsed().as_millis() as u64,+ &request_id,+ );+ Json(resp).into_response()+ }+ // The retry did not work. Answering 200 with `reopened: false` would+ // be exactly the accepted-and-ignored shape this repo is trying to+ // stop shipping — the caller gets the live failure instead (503 for a+ // still-broken index, 404 for a name that is not a failed index).+ Err(e) => crate::error::ApiError::new(e.into()).into_response(),}}diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rsindex c2fb51f1e..28bc98596 100644--- a/engine/crates/xerj-api/src/router.rs+++ b/engine/crates/xerj-api/src/router.rs@@ -51,8 +51,10 @@ use crate::{/// POST /v1/indices/:name/search search/// POST /v1/indices/:name/_flush flush_index/// GET /v1/health health-/// GET /v1/health/ready readiness probe (200 unless red)+/// GET /v1/health/ready readiness probe (200 unless nothing can be served)/// GET /v1/cluster/health cluster_health+/// GET /_cluster/indices/failed failed_indices (name + open error)+/// POST /_cluster/indices/failed/:name/_retry retry_failed_index/// POST /v1/admin/flush admin_flush (flush all indices)/// POST /v1/admin/backup admin_backup (snapshot to disk)/// GET /v1/metrics metrics (Prometheus text)@@ -104,6 +106,15 @@ pub fn build_native_router(state: AppState) -> Router {.route("/v1/health", get(native::health)).route("/v1/health/ready", get(native::readiness)).route("/v1/cluster/health", get(native::cluster_health))+ // Failed-index recovery (issue #206) — inspect / retry. Delete goes+ // through the ordinary `DELETE /{index}`. Mounted on both routers+ // under the same path so an operator does not have to know which+ // listener they reached.+ .route("/_cluster/indices/failed", get(native::failed_indices))+ .route(+ "/_cluster/indices/failed/:name/_retry",+ post(native::retry_failed_index),+ ).route("/v1/embedding/identity",get(native::embedding_execution_identity),@@ -515,6 +526,17 @@ pub fn build_es_compat_router(state: AppState) -> Router {get(es_compat::get_cluster_settings).put(es_compat::put_cluster_settings),).route("/_cluster/reroute", post(es_compat::cluster_reroute))+ // ── Failed-index recovery (issue #206) ─────────────────────────────+ // XERJ extension: no ES equivalent (ES recovers a red index through+ // shard reallocation, which a single-binary node has no analogue for),+ // so it cannot collide with the parity surface. Mounted on the+ // ES-compat port because that is the port an operator already has open+ // when they discover the red index there.+ .route("/_cluster/indices/failed", get(native::failed_indices))+ .route(+ "/_cluster/indices/failed/:name/_retry",+ post(native::retry_failed_index),+ ).route("/_cluster/pending_tasks",get(es_compat::cluster_pending_tasks),diff --git a/engine/crates/xerj-api/tests/failed_index_is_visible_and_actionable.rs b/engine/crates/xerj-api/tests/failed_index_is_visible_and_actionable.rsnew file mode 100644index 000000000..bcbad450b--- /dev/null+++ b/engine/crates/xerj-api/tests/failed_index_is_visible_and_actionable.rs@@ -0,0 +1,447 @@+//! Issue #206, over HTTP: a failed index must be visible on the surfaces an+//! operator actually watches, and actionable from them.+//!+//! Three separate failures were reported and all three are pinned here.+//!+//! 1. **The index was invisible.** `_cat/indices` and `_cluster/state` list+//! only indices that opened, so the broken one appeared nowhere; `DELETE`+//! answered 404 `index_not_found`. The only recovery was to stop the server+//! and remove the directory by hand.+//! 2. **Readiness went hard-fail.** `/health/ready` returned 503 for any red+//! status, so one broken index removed a pod holding 199 healthy ones from+//! service permanently — a single-index problem escalated to a total+//! outage.+//! 3. **The health surfaces disagreed.** `/v1/health`, `/v1/cluster/health`+//! and `/health/ready` used real engine health while `_cat/health` printed+//! a hardcoded `green` — so the ES-compatible endpoint, the one existing+//! dashboards point at, was the one that could not report a broken node.+//! `_cluster/health` did not count an unopenable primary either.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use http_body_util::BodyExt;+use serde_json::Value;+use tower::ServiceExt;+use xerj_api::{+ router::{build_es_compat_router, build_native_router},+ state::AppState,+};+use xerj_common::types::Schema;+use xerj_common::{config::Config, metrics::Metrics};+use xerj_engine::Engine;++/// A node whose data dir holds one healthy index and one whose manifest is+/// corrupt, so the boot that produces `state` quarantines exactly one index.+async fn node_with_one_broken_index() -> (AppState, tempfile::TempDir) {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ {+ let engine = Engine::new(config.clone()).expect("first boot");+ for name in ["broken", "healthy"] {+ engine.create_index(name, Schema::empty()).unwrap();+ let idx = engine.get_index(name).unwrap();+ idx.index_document(Some("1".into()), serde_json::json!({"t": "hello"}))+ .await+ .unwrap();+ engine.flush_index(name).await.unwrap();+ }+ }+ std::fs::write(+ dir.path().join("broken").join("snapshot.json"),+ b"{not json",+ )+ .unwrap();++ let metrics = Metrics::new().expect("metrics");+ let engine = Engine::new(config.clone()).expect("boot with one broken index");+ assert_eq!(+ engine.list_failed_indices().len(),+ 1,+ "fixture must produce exactly one failed index"+ );+ (AppState::new(config, engine, metrics), dir)+}++async fn send(app: &axum::Router, method: &str, uri: &str) -> (StatusCode, String) {+ let req = Request::builder()+ .method(method)+ .uri(uri)+ .header("content-type", "application/json")+ .body(Body::empty())+ .expect("request");+ let resp = app.clone().oneshot(req).await.expect("response");+ let status = resp.status();+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();+ (status, String::from_utf8_lossy(&bytes).into_owned())+}++async fn send_json(app: &axum::Router, method: &str, uri: &str) -> (StatusCode, Value) {+ let (status, text) = send(app, method, uri).await;+ (status, serde_json::from_str(&text).unwrap_or(Value::Null))+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn cat_indices_lists_the_failed_index_as_red() {+ let (state, _dir) = node_with_one_broken_index().await;+ let app = build_es_compat_router(state);++ let (status, body) = send(&app, "GET", "/_cat/indices").await;+ assert_eq!(status, StatusCode::OK);+ let broken: Vec<&str> = body.lines().filter(|l| l.contains("broken")).collect();+ assert_eq!(+ broken.len(),+ 1,+ "failed index missing from _cat/indices: {body:?}"+ );+ assert!(+ broken[0].starts_with("red open broken "),+ "a failed index must be listed red: {:?}",+ broken[0]+ );+ assert!(+ body.lines().any(|l| l.starts_with("green open healthy ")),+ "the healthy index must be untouched: {body:?}"+ );++ // A concrete-name request for it resolves instead of 404ing.+ let (status, body) = send(&app, "GET", "/_cat/indices/broken").await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert!(body.starts_with("red open broken "), "{body:?}");+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn all_four_health_surfaces_agree_that_the_node_is_red() {+ let (state, _dir) = node_with_one_broken_index().await;+ let es = build_es_compat_router(state.clone());+ let native = build_native_router(state);++ // `_cat/health` used to print a hardcoded `green` here.+ let (status, body) = send(&es, "GET", "/_cat/health").await;+ assert_eq!(status, StatusCode::OK);+ let cols: Vec<&str> = body.split_whitespace().collect();+ assert_eq!(cols[3], "red", "_cat/health status column: {body:?}");+ assert_eq!(cols[10], "1", "_cat/health unassign column: {body:?}");++ // `_cluster/health` counted unassigned replicas only, never an+ // unopenable primary.+ let (status, health) = send_json(&es, "GET", "/_cluster/health").await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(health["status"], "red", "{health}");+ assert_eq!(health["unassigned_primary_shards"], 1, "{health}");+ assert_eq!(health["unassigned_shards"], 1, "{health}");++ // `?level=indices` answers "which index, and why" in one call.+ let (_, detail) = send_json(&es, "GET", "/_cluster/health?level=indices").await;+ let broken = &detail["indices"]["broken"];+ assert_eq!(broken["status"], "red", "{detail}");+ assert!(+ broken["unassigned_info"]["details"]+ .as_str()+ .unwrap_or_default()+ .contains("snapshot.json"),+ "the open error must reach the operator: {detail}"+ );++ // The two native surfaces already agreed; they must keep agreeing.+ let (_, native_health) = send_json(&native, "GET", "/v1/health").await;+ assert_eq!(native_health["data"]["status"], "red", "{native_health}");+ let (_, native_cluster) = send_json(&native, "GET", "/v1/cluster/health").await;+ assert_eq!(native_cluster["data"]["status"], "red", "{native_cluster}");++ // `wait_for_active_shards=all` must not report success on a cluster whose+ // primary cannot be opened — "all shards active" is exactly what is false.+ let (status, waited) =+ send_json(&es, "GET", "/_cluster/health?wait_for_active_shards=all").await;+ assert_eq!(status, StatusCode::REQUEST_TIMEOUT, "{waited}");+ assert_eq!(waited["timed_out"], true, "{waited}");+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn cluster_state_carries_the_failed_index_as_an_unassigned_primary() {+ let (state, _dir) = node_with_one_broken_index().await;+ let app = build_es_compat_router(state);++ let (status, cs) = send_json(&app, "GET", "/_cluster/state").await;+ assert_eq!(status, StatusCode::OK);+ assert!(+ cs["metadata"]["indices"]["broken"].is_object(),+ "a failed index still exists and belongs in cluster state: {cs}"+ );+ let shard = &cs["routing_table"]["indices"]["broken"]["shards"]["0"][0];+ assert_eq!(shard["state"], "UNASSIGNED", "{cs}");+ assert_eq!(shard["primary"], true, "{cs}");+ assert_eq!(+ shard["unassigned_info"]["reason"], "ALLOCATION_FAILED",+ "{cs}"+ );+ assert!(+ shard["unassigned_info"]["details"]+ .as_str()+ .unwrap_or_default()+ .contains("could not be parsed"),+ "{cs}"+ );+ // ES's own `unassigned_info` field names, so an operator's existing+ // dashboard renders this block without knowing anything about XERJ.+ assert_eq!(+ shard["unassigned_info"]["allocation_status"], "no_valid_shard_copy",+ "{cs}"+ );+ assert!(+ shard["unassigned_info"]["at"]+ .as_str()+ .unwrap_or_default()+ .ends_with('Z'),+ "`at` must be an ISO-8601 instant like ES: {cs}"+ );+ assert_eq!(+ cs["routing_nodes"]["unassigned"].as_array().map(Vec::len),+ Some(1),+ "{cs}"+ );+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn readiness_keeps_a_partly_degraded_node_in_service() {+ let (state, _dir) = node_with_one_broken_index().await;+ let app = build_es_compat_router(state);++ // One broken index out of two: the node can still serve `healthy`, so+ // pulling it out of rotation forever would be the outage, not the fix.+ let (status, body) = send(&app, "GET", "/health/ready").await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert!(+ body.contains("degraded"),+ "the degradation must be named: {body:?}"+ );+ assert!(body.contains("1 of 2 indices serving"), "{body:?}");+ // Counts only — this path is auth-exempt, so it must not enumerate index+ // names to an unauthenticated prober.+ assert!(+ !body.contains("broken"),+ "readiness leaked an index name: {body:?}"+ );+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn readiness_still_fails_when_nothing_can_be_served() {+ // Every index in the data dir is broken — there is genuinely nothing to+ // send traffic to, so 503 is correct and must survive the fix above.+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ {+ let engine = Engine::new(config.clone()).expect("first boot");+ engine.create_index("only", Schema::empty()).unwrap();+ let idx = engine.get_index("only").unwrap();+ idx.index_document(Some("1".into()), serde_json::json!({"t": "x"}))+ .await+ .unwrap();+ engine.flush_index("only").await.unwrap();+ }+ std::fs::write(dir.path().join("only").join("snapshot.json"), b"{not json").unwrap();+ let engine = Engine::new(config.clone()).expect("reboot");+ let state = AppState::new(config, engine, Metrics::new().expect("metrics"));+ let app = build_es_compat_router(state);++ let (status, body) = send(&app, "GET", "/health/ready").await;+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{body}");+ assert!(body.contains("no index could be opened"), "{body:?}");+ assert!(+ !body.contains("only"),+ "readiness leaked an index name: {body:?}"+ );+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn the_failed_index_can_be_inspected_retried_and_deleted_over_http() {+ let (state, dir) = node_with_one_broken_index().await;+ let app = build_es_compat_router(state);++ // Inspect: name AND reason, plus the two commands that act on it.+ let (status, listing) = send_json(&app, "GET", "/_cluster/indices/failed").await;+ assert_eq!(status, StatusCode::OK);+ let items = listing["data"]["failed_indices"].as_array().expect("array");+ assert_eq!(items.len(), 1, "{listing}");+ assert_eq!(items[0]["index"], "broken");+ assert!(+ items[0]["reason"]+ .as_str()+ .unwrap_or_default()+ .contains("snapshot.json"),+ "{listing}"+ );++ // Reading it names the real problem instead of claiming it does not exist.+ let (status, err) = send_json(&app, "GET", "/broken/_search").await;+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{err}");+ assert_eq!(+ err["error"]["type"], "no_shard_available_action_exception",+ "{err}"+ );++ // Retry while still broken: 503 with the live reason, not a cheerful 200.+ let (status, err) = send_json(&app, "POST", "/_cluster/indices/failed/broken/_retry").await;+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{err}");+ assert!(+ err["error"]["reason"]+ .as_str()+ .unwrap_or_default()+ .contains("snapshot.json"),+ "{err}"+ );++ // Delete without a restart.+ let (status, body) = send_json(&app, "DELETE", "/broken").await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["acknowledged"], true, "{body}");+ assert!(+ !dir.path().join("broken").exists(),+ "DELETE must remove the bytes on disk"+ );++ // The node is green again and nothing is left to inspect.+ let (_, listing) = send_json(&app, "GET", "/_cluster/indices/failed").await;+ assert_eq!(listing["data"]["count"], 0, "{listing}");+ let (_, health) = send_json(&app, "GET", "/_cluster/health").await;+ assert_eq!(health["status"], "green", "{health}");+ let (status, body) = send(&app, "GET", "/_cat/indices").await;+ assert_eq!(status, StatusCode::OK);+ assert!(!body.contains("broken"), "{body:?}");+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn retry_over_http_reopens_the_index_once_the_cause_is_fixed() {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ {+ let engine = Engine::new(config.clone()).expect("first boot");+ engine.create_index("broken", Schema::empty()).unwrap();+ let idx = engine.get_index("broken").unwrap();+ idx.index_document(Some("1".into()), serde_json::json!({"t": "hello"}))+ .await+ .unwrap();+ engine.flush_index("broken").await.unwrap();+ }+ let manifest = dir.path().join("broken").join("snapshot.json");+ let good = std::fs::read(&manifest).unwrap();+ std::fs::write(&manifest, b"{not json").unwrap();++ let engine = Engine::new(config.clone()).expect("reboot");+ let state = AppState::new(config, engine, Metrics::new().expect("metrics"));+ let app = build_es_compat_router(state);++ // Operator restores the manifest the storage error pointed them at, then+ // retries — no restart.+ std::fs::write(&manifest, &good).unwrap();+ let (status, body) = send_json(&app, "POST", "/_cluster/indices/failed/broken/_retry").await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["data"]["reopened"], true, "{body}");++ let (_, health) = send_json(&app, "GET", "/_cluster/health").await;+ assert_eq!(health["status"], "green", "{health}");+ let (status, count) = send_json(&app, "GET", "/broken/_count").await;+ assert_eq!(status, StatusCode::OK, "{count}");+ assert_eq!(count["count"], 1, "the reopened index must serve its data");++ // Retrying a name that is not a failed index is a 404, not a silent ok.+ let (status, err) = send_json(&app, "POST", "/_cluster/indices/failed/broken/_retry").await;+ assert_eq!(status, StatusCode::NOT_FOUND, "{err}");+}++// ─────────────────────────────────────────────────────────────────────────────+// The boundary half: making a failed index visible must not make it visible to+// everyone. A brain a scoped key may not read stays unreadable when it breaks.+// ─────────────────────────────────────────────────────────────────────────────++const ADMIN_KEY: &str = "admin-secret-key-for-failed-index-test";++async fn send_auth(+ app: &axum::Router,+ method: &str,+ uri: &str,+ auth: &str,+) -> (StatusCode, String) {+ let req = Request::builder()+ .method(method)+ .uri(uri)+ .header("content-type", "application/json")+ .header("authorization", auth)+ .body(Body::empty())+ .expect("request");+ let resp = app.clone().oneshot(req).await.expect("response");+ let status = resp.status();+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();+ (status, String::from_utf8_lossy(&bytes).into_owned())+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn a_failed_brain_is_not_leaked_to_a_scoped_key() {+ // A brain and an ordinary index; the brain's manifest is corrupted so it+ // is a FAILED index on the boot that serves the requests below.+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ config.auth.enabled = true;+ config.auth.admin_api_key = ADMIN_KEY.to_string();+ {+ let engine = Engine::new(config.clone()).expect("first boot");+ for name in [".xerj-memory-bob-edges", "logs-2026"] {+ engine.create_index(name, Schema::empty()).unwrap();+ let idx = engine.get_index(name).unwrap();+ idx.index_document(Some("1".into()), serde_json::json!({"t": "x"}))+ .await+ .unwrap();+ engine.flush_index(name).await.unwrap();+ }+ }+ std::fs::write(+ dir.path()+ .join(".xerj-memory-bob-edges")+ .join("snapshot.json"),+ b"{not json",+ )+ .unwrap();+ let engine = Engine::new(config.clone()).expect("reboot");+ let state = AppState::new(config, engine, Metrics::new().expect("metrics"));+ let app = build_es_compat_router(state);++ let admin = format!("ApiKey {ADMIN_KEY}");+ let mint = Request::builder()+ .method("POST")+ .uri("/_security/api_key")+ .header("content-type", "application/json")+ .header("authorization", &admin)+ .body(Body::from(+ r#"{"name":"logs-agent","role_descriptors":{"logs":{"indices":[+ {"names":["logs-2026"],"privileges":["read","write"]}]}}}"#,+ ))+ .expect("request");+ let resp = app.clone().oneshot(mint).await.expect("response");+ assert_eq!(resp.status(), StatusCode::OK);+ let bytes = resp.into_body().collect().await.expect("body").to_bytes();+ let minted: Value = serde_json::from_slice(&bytes).expect("json");+ let agent = format!("ApiKey {}", minted["encoded"].as_str().expect("encoded"));++ for uri in [+ "/_cat/indices",+ "/_cat/shards",+ "/_cluster/state",+ "/_cluster/health?level=indices",+ "/_cluster/indices/failed",+ ] {+ let (status, body) = send_auth(&app, "GET", uri, &agent).await;+ assert!(status.is_success(), "{uri}: {status} {body}");+ assert!(+ !body.contains("bob"),+ "{uri} leaked a failed brain to a scoped key: {body}"+ );+ }++ // The admin still sees it — "prune everything" would pass the test above.+ let (status, body) = send_auth(&app, "GET", "/_cluster/indices/failed", &admin).await;+ assert!(status.is_success(), "{body}");+ assert!(body.contains("bob"), "the admin must still see it: {body}");+}diff --git a/engine/crates/xerj-common/src/error.rs b/engine/crates/xerj-common/src/error.rsindex 86d57fb6e..aecb43bd8 100644--- a/engine/crates/xerj-common/src/error.rs+++ b/engine/crates/xerj-common/src/error.rs@@ -22,6 +22,20 @@ pub enum XerjError {#[error("index already exists: {name}")]IndexAlreadyExists { name: String },+ /// The index directory exists but could not be opened, so the index+ /// cannot serve. Deliberately distinct from [`XerjError::IndexNotFound`]:+ /// the name is real, the data is still on disk, and the operator's move is+ /// to fix the cause and retry — or to delete it. Reporting these as 404+ /// "not found" is what made a failed index un-inspectable and+ /// un-deletable (issue #206).+ #[error(+ "index [{index}] failed to open and is unavailable: {reason}. \+ Inspect it with GET /_cluster/indices/failed, retry it with \+ POST /_cluster/indices/failed/{index}/_retry once the cause is fixed, \+ or remove it with DELETE /{index}"+ )]+ IndexUnavailable { index: String, reason: String },+// ── Document operations ────────────────────────────────────────────────/// A document with the given ID was not found in the index.#[error("document not found: id={id} in index={index}")]@@ -137,6 +151,13 @@ impl XerjError {Self::IndexAlreadyExists { name: name.into() }}+ pub fn index_unavailable(index: impl Into<String>, reason: impl Into<String>) -> Self {+ Self::IndexUnavailable {+ index: index.into(),+ reason: reason.into(),+ }+ }+pub fn document_not_found(id: impl Into<String>, index: impl Into<String>) -> Self {Self::DocumentNotFound {id: id.into(),@@ -266,6 +287,9 @@ impl XerjError {| Self::ConfigError { .. }| Self::ResultWindowTooLarge { .. } => 400,Self::IndexBlocked { .. } => 403,+ // The index exists but has no serving copy — ES answers the same+ // shape (no_shard_available_action_exception) with 503.+ Self::IndexUnavailable { .. } => 503,Self::AuthError { .. } => 401,Self::ResourceExhausted { .. } | Self::CircuitBreaking { .. } => 429,Self::StorageError { .. }diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex 136c30deb..6e0f19cfa 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -98,6 +98,28 @@ pub struct IndexInfo {pub schema_version: u64,}+/// An index directory that is present on disk but could not be opened.+///+/// Before issue #206 a failed open left only `name → reason` in a private map+/// that nothing but [`Engine::health`] ever read: the index was invisible to+/// `_cat/indices` and `_cluster/state`, `DELETE` answered 404, and the only+/// recovery was to stop the server and edit the data directory by hand. A+/// failed index is now a real, addressable state — it is listed, it carries+/// the open error verbatim, it can be deleted, and it can be retried once the+/// operator has fixed the cause.+#[derive(Debug, Clone, Serialize, Deserialize)]+pub struct FailedIndex {+ /// Index name, i.e. the directory name under `data_dir`.+ pub name: String,+ /// The verbatim error from the last open attempt. Never summarised —+ /// the storage layer's message already names the file and the fix.+ pub reason: String,+ /// Wall-clock ms of the first failure (boot, restore, or a failed retry).+ pub failed_at_ms: i64,+ /// How many explicit retries have been attempted since (0 at first boot).+ pub retries: u32,+}+/// Overall engine health.#[derive(Debug, Clone, Serialize, Deserialize)]pub struct HealthStatus {@@ -297,8 +319,11 @@ pub struct Engine {pub search_templates: Arc<DashMap<String, Value>>,/// async search id → stored result JSONpub async_searches: Arc<DashMap<String, Value>>,- /// Names of index directories that failed to open on startup (health = red).- pub failed_indices: Arc<DashMap<String, String>>,+ /// Index directories that are present but could not be opened, keyed by+ /// index name (health = red). See [`FailedIndex`] — these are inspectable+ /// (`GET /_cluster/indices/failed`), deletable (`DELETE /{index}`) and+ /// retryable (`POST /_cluster/indices/failed/{index}/_retry`).+ pub failed_indices: Arc<DashMap<String, FailedIndex>>,/// transform id → transform definition JSONpub transforms: Arc<DashMap<String, Value>>,/// index_name → frozen state (true = frozen / read-only)@@ -517,7 +542,7 @@ impl Engine {}Err(e) => {warn!(name = name_str.as_str(), error = %e, "failed to open index");- engine.failed_indices.insert(name_str, e.to_string());+ engine.record_failed_index(&name_str, e.to_string());}}}@@ -831,6 +856,16 @@ impl Engine {));}+ // The name is taken by an index that exists on disk but would not+ // open. Creating over it would run the store open again and surface a+ // storage error that says nothing about the operator's options, so+ // refuse here with the recorded reason instead (issue #206).+ if let Some(f) = self.failed_indices.get(name) {+ return Err(EngineError::Common(+ xerj_common::XerjError::index_unavailable(name, f.reason.clone()),+ ));+ }+// Apply matching template (highest priority wins) on every create path.let effective_schema = self.apply_index_template(name, schema);let idx = Index::create_with_settings(@@ -1177,6 +1212,11 @@ impl Engine {/// Also drops any aliases that pointed only at this index (matching ES/// semantics) and clears the `closed_indices` flag so the name is/// truly gone when another test recreates it.+ ///+ /// A **failed** index (present on disk, refused at open — see+ /// [`FailedIndex`]) is deletable through this same door. It used to answer+ /// 404 `index_not_found`, which left an operator with no lever but+ /// stopping the server and removing the directory by hand (issue #206).pub async fn delete_index(&self, name: &str) -> Result<()> {// Per-index authorization backstop (issue #79) — "destroy the brain"// is the loudest door, so it is checked before the index is removed@@ -1186,13 +1226,37 @@ impl Engine {xerj_common::XerjError::index_not_found(name),));}- let idx =- self.indices.remove(name).map(|(_, v)| v).ok_or_else(|| {- EngineError::Common(xerj_common::XerjError::index_not_found(name))- })?;+ match self.indices.remove(name).map(|(_, v)| v) {+ Some(idx) => idx.delete_all_data().await?,+ None => {+ // Not open. If it is a known failed index, its bytes are still+ // on disk and removing them is exactly what the operator asked+ // for; anything else is a genuine 404.+ if !self.failed_indices.contains_key(name) {+ return Err(EngineError::Common(+ xerj_common::XerjError::index_not_found(name),+ ));+ }+ // Drop the bookkeeping only after the bytes are gone. Removing+ // it first would take a delete that failed (read-only mount,+ // fs error) out of the failed list while its directory+ // survives — the operator would believe the name was freed and+ // the index would reappear on the next boot.+ self.remove_index_dir(name)?;+ self.failed_indices.remove(name);+ }+ }- idx.delete_all_data().await?;+ self.forget_index_metadata(name);+ info!(name, "index deleted");+ Ok(())+ }+ /// Drop every piece of engine-side metadata that names `index` — aliases+ /// that pointed only at it, the closed flag, settings, mappings, alias+ /// metadata. Shared by the open-index and failed-index delete paths so the+ /// two cannot drift.+ fn forget_index_metadata(&self, name: &str) {// Remove this index from every alias that references it; drop the// alias entirely when its backing list becomes empty.let empty_aliases: Vec<String> = self@@ -1216,11 +1280,111 @@ impl Engine {self.index_settings.remove(name);self.index_mappings.remove(name);self.index_alias_metadata.remove(name);+ }- info!(name, "index deleted");+ /// Remove `<data_dir>/<name>` from disk, refusing anything that does not+ /// resolve inside `data_dir`.+ ///+ /// The open-index path deletes through `Index::delete_all_data`, which can+ /// only ever point at a directory the engine itself built. The failed-index+ /// path has no `Index` to ask, so the name is re-validated here and the+ /// resolved path is proven to be under `data_dir` before anything is+ /// removed — the same canonicalisation rule the snapshot-restore path+ /// applies before it writes.+ fn remove_index_dir(&self, name: &str) -> Result<()> {+ // Reject traversal/absolute forms up front: only a legal index name+ // can name a directory we own.+ IndexName::new(name).map_err(EngineError::Common)?;+ let dir = self.data_dir.join(name);+ if !dir.exists() {+ return Ok(());+ }+ let dir_canon = dir.canonicalize().map_err(EngineError::Io)?;+ let root_canon = self.data_dir.canonicalize().map_err(EngineError::Io)?;+ if !dir_canon.starts_with(&root_canon) {+ return Err(EngineError::Common(xerj_common::XerjError::storage(+ format!("refusing to delete index [{name}] outside data_dir (canonical)"),+ )));+ }+ std::fs::remove_dir_all(&dir_canon).map_err(EngineError::Io)?;Ok(())}+ // ── Failed-index recovery (issue #206) ────────────────────────────────────++ /// Record (or re-record) an index that could not be opened.+ ///+ /// Preserves `failed_at_ms` across repeated failures so "since when" stays+ /// truthful, and counts retries so a flapping directory is visible as+ /// such.+ fn record_failed_index(&self, name: &str, reason: String) {+ match self.failed_indices.get_mut(name) {+ Some(mut existing) => {+ existing.reason = reason;+ existing.retries = existing.retries.saturating_add(1);+ }+ None => {+ self.failed_indices.insert(+ name.to_string(),+ FailedIndex {+ name: name.to_string(),+ reason,+ failed_at_ms: now_millis(),+ retries: 0,+ },+ );+ }+ }+ }++ /// Every failed index the caller is allowed to see, sorted by name.+ pub fn list_failed_indices(&self) -> Vec<FailedIndex> {+ let mut out: Vec<FailedIndex> = self+ .failed_indices+ .iter()+ .filter(|e| crate::index_guard::visible(e.key()))+ .map(|e| e.value().clone())+ .collect();+ out.sort_by(|a, b| a.name.cmp(&b.name));+ out+ }++ /// Re-attempt the open of a failed index.+ ///+ /// On success the index becomes a normal, serving index (mapping and date+ /// flags restored exactly as at boot) and leaves the failed set. On failure+ /// the recorded reason is refreshed, the retry counter advances, and the+ /// new error is returned — the operator gets the *current* reason, not the+ /// one from boot.+ pub fn retry_failed_index(&self, name: &str) -> Result<()> {+ if !crate::index_guard::visible(name) || !self.failed_indices.contains_key(name) {+ return Err(EngineError::Common(+ xerj_common::XerjError::index_not_found(name),+ ));+ }+ let index_name = IndexName::new(name).map_err(EngineError::Common)?;+ match Index::open(index_name, &self.config, &self.data_dir) {+ Ok(idx) => {+ self.load_persisted_es_mapping(name);+ if let Some(m) = self.index_mappings.get(name) {+ Engine::apply_date_mapping_flags(&idx, m.value());+ }+ self.indices.insert(name.to_string(), idx);+ self.failed_indices.remove(name);+ info!(name, "failed index reopened");+ Ok(())+ }+ Err(e) => {+ let reason = e.to_string();+ self.record_failed_index(name, reason.clone());+ warn!(name, error = %reason, "retry of failed index did not succeed");+ Err(EngineError::Common(+ xerj_common::XerjError::index_unavailable(name, reason),+ ))+ }+ }+ }+/// Get a reference to an index by name, resolving aliases first./// If the name is an alias pointing to multiple indices, returns the first one.///@@ -1245,9 +1409,7 @@ impl Engine {.indices.get(real_name.as_str()).map(|r| Arc::clone(r.value()))- .ok_or_else(|| {- EngineError::Common(xerj_common::XerjError::index_not_found(real_name))- });+ .ok_or_else(|| EngineError::Common(self.missing_index_error(real_name)));}}if !crate::index_guard::visible(name) {@@ -1258,7 +1420,21 @@ impl Engine {self.indices.get(name).map(|r| Arc::clone(r.value()))- .ok_or_else(|| EngineError::Common(xerj_common::XerjError::index_not_found(name)))+ .ok_or_else(|| EngineError::Common(self.missing_index_error(name)))+ }++ /// The error for a name that is not in `indices`.+ ///+ /// A name that failed to open is **not** "not found": the directory is+ /// still there and the operator has a lever. Reporting a 404 for it is how+ /// issue #206's stuck state stayed invisible, so a failed index answers+ /// 503 `no_shard_available_action_exception` carrying the open error and+ /// the three commands that act on it.+ fn missing_index_error(&self, name: &str) -> xerj_common::XerjError {+ match self.failed_indices.get(name) {+ Some(f) => xerj_common::XerjError::index_unavailable(name, f.reason.clone()),+ None => xerj_common::XerjError::index_not_found(name),+ }}/// Return an index by name, creating it if it doesn't exist (ES behaviour).@@ -1278,6 +1454,15 @@ impl Engine {if let Ok(idx) = self.get_index(name) {return Ok(idx);}+ // A failed index already occupies this name and its bytes are still on+ // disk. Auto-creating over it would either destroy recoverable data or+ // fail deep inside the store with an opaque message — refuse loudly+ // with the open reason and the operator's options instead (issue #206).+ if let Some(f) = self.failed_indices.get(name) {+ return Err(EngineError::Common(+ xerj_common::XerjError::index_unavailable(name, f.reason.clone()),+ ));+ }// Auto-create with empty schema.self.create_index(name, Schema::empty())?;self.get_index(name)@@ -1974,7 +2159,7 @@ impl Engine {}Err(e) => {warn!(index = idx_name, error = %e, "failed to reopen restored index");- self.failed_indices.insert(idx_name.clone(), e.to_string());+ self.record_failed_index(idx_name, e.to_string());}}}diff --git a/engine/crates/xerj-engine/tests/failed_index_recovery.rs b/engine/crates/xerj-engine/tests/failed_index_recovery.rsnew file mode 100644index 000000000..e7257800a--- /dev/null+++ b/engine/crates/xerj-engine/tests/failed_index_recovery.rs@@ -0,0 +1,195 @@+//! Issue #206: a failed index is a **state**, not a dead entry.+//!+//! Before this, an index whose directory refused to open at boot was recorded+//! in a private map that only `Engine::health` ever read. Live-reproduced at+//! `12d5cc32` with a corrupt `snapshot.json` (the storage layer's documented+//! "present but unparseable manifest" refusal):+//!+//! ```text+//! failed_indices = [("broken", "storage error: … could not be parsed …")]+//! indices = [] ← invisible+//! health = "red"+//! delete = Some(IndexNotFound { name: "broken" }) ← undeletable+//! ```+//!+//! So the index could not be listed, could not be deleted, and could not be+//! retried: the only lever was stopping the server and editing the data+//! directory by hand. Every assertion below fails on that build.+//!+//! The corruption used here is deliberate and realistic — `IndexStore::open`+//! refuses a manifest it cannot parse rather than treating it as empty,+//! because treating it as empty would orphan and then delete every segment.+//! That refusal is correct; having no way to act on it afterwards was not.++use tempfile::TempDir;+use xerj_common::config::Config;+use xerj_common::types::Schema;+use xerj_common::XerjError;+use xerj_engine::{Engine, EngineError};++fn config_for(dir: &TempDir) -> Config {+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ config+}++/// Build a data dir holding `names`, each with one flushed document, then+/// corrupt `break_me`'s manifest so the next boot refuses to open it.+/// Returns the good manifest bytes so a test can "fix the cause" and retry.+async fn data_dir_with_one_broken_index(names: &[&str], break_me: &str) -> (TempDir, Vec<u8>) {+ let dir = TempDir::new().unwrap();+ {+ let engine = Engine::new(config_for(&dir)).expect("first boot");+ for name in names {+ engine.create_index(name, Schema::empty()).unwrap();+ let idx = engine.get_index(name).unwrap();+ idx.index_document(Some("1".to_string()), serde_json::json!({"t": "hello"}))+ .await+ .unwrap();+ engine.flush_index(name).await.unwrap();+ }+ }+ let manifest = dir.path().join(break_me).join("snapshot.json");+ let good = std::fs::read(&manifest).expect("a flushed index must have a manifest");+ std::fs::write(&manifest, b"{not json").unwrap();+ (dir, good)+}++fn xerj_error(e: EngineError) -> XerjError {+ e.into()+}++/// `Index` is not `Debug`, so `unwrap_err()` is unavailable on the+/// index-returning calls. Take the error explicitly instead.+fn err_of<T>(r: Result<T, EngineError>, what: &str) -> XerjError {+ match r {+ Ok(_) => panic!("{what} must not succeed"),+ Err(e) => xerj_error(e),+ }+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn a_failed_index_is_listed_with_its_reason() {+ let (dir, _good) = data_dir_with_one_broken_index(&["broken", "healthy"], "broken").await;+ let engine = Engine::new(config_for(&dir)).expect("boot with one broken index");++ let failed = engine.list_failed_indices();+ assert_eq!(+ failed.len(),+ 1,+ "expected exactly one failed index: {failed:?}"+ );+ let f = &failed[0];+ assert_eq!(f.name, "broken");+ assert!(+ f.reason.contains("snapshot.json") && f.reason.contains("could not be parsed"),+ "the reason must be the verbatim open error, not a summary: {}",+ f.reason+ );+ assert!(f.failed_at_ms > 0, "a failed index records when it failed");+ assert_eq!(f.retries, 0, "no retry has been attempted yet");++ // The healthy index in the same data dir is unaffected — one broken index+ // must not cost the node the others.+ assert_eq!(engine.list_indices().await.len(), 1);+ assert!(engine.get_index("healthy").is_ok());+ assert_eq!(engine.health().await.status, "red");+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn reads_and_writes_to_a_failed_index_say_unavailable_not_not_found() {+ let (dir, _good) = data_dir_with_one_broken_index(&["broken"], "broken").await;+ let engine = Engine::new(config_for(&dir)).expect("boot");++ // `index_not_found` (404) was a lie: the name is taken and the bytes are+ // on disk. It is now 503 `no_shard_available_action_exception` carrying+ // the open error.+ let e = err_of(engine.get_index("broken"), "get_index on a failed index");+ assert!(+ matches!(e, XerjError::IndexUnavailable { .. }),+ "get_index on a failed index must not report not-found: {e:?}"+ );+ assert_eq!(e.http_status(), 503);+ assert!(e.to_string().contains("snapshot.json"), "{e}");++ // Auto-create must not run over a failed index — that would either destroy+ // recoverable segments or fail deep in the store with an opaque message.+ let e = err_of(+ engine.get_or_create_index("broken"),+ "auto-create over a failed index",+ );+ assert!(+ matches!(e, XerjError::IndexUnavailable { .. }),+ "auto-create must refuse a failed index: {e:?}"+ );+ // …and the refusal must not have created anything.+ assert!(engine.list_indices().await.is_empty());+ assert_eq!(engine.list_failed_indices().len(), 1);++ // Explicit create is refused for the same reason, with the same message.+ let e = xerj_error(engine.create_index("broken", Schema::empty()).unwrap_err());+ assert!(+ matches!(e, XerjError::IndexUnavailable { .. }),+ "create over a failed index must name the real cause: {e:?}"+ );+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn a_failed_index_can_be_deleted_without_a_restart() {+ let (dir, _good) = data_dir_with_one_broken_index(&["broken", "healthy"], "broken").await;+ let engine = Engine::new(config_for(&dir)).expect("boot");+ let index_dir = dir.path().join("broken");+ assert!(index_dir.exists());++ engine+ .delete_index("broken")+ .await+ .expect("a failed index must be deletable");++ assert!(!index_dir.exists(), "delete must remove the bytes on disk");+ assert!(engine.list_failed_indices().is_empty());+ // Health recovers without a restart, and the name is reusable.+ assert_ne!(engine.health().await.status, "red");+ engine+ .create_index("broken", Schema::empty())+ .expect("the freed name must be reusable");++ // A name that was never an index is still a plain 404.+ let e = xerj_error(engine.delete_index("never-existed").await.unwrap_err());+ assert!(matches!(e, XerjError::IndexNotFound { .. }), "{e:?}");+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn retry_reports_the_live_reason_and_succeeds_once_the_cause_is_fixed() {+ let (dir, good) = data_dir_with_one_broken_index(&["broken"], "broken").await;+ let engine = Engine::new(config_for(&dir)).expect("boot");++ // Retry while still broken: the error is returned, not swallowed, and the+ // attempt is counted so a flapping directory is visible as one.+ let e = xerj_error(engine.retry_failed_index("broken").unwrap_err());+ assert!(+ matches!(e, XerjError::IndexUnavailable { .. }),+ "a retry that did not work must fail loudly: {e:?}"+ );+ let after = engine.list_failed_indices();+ assert_eq!(after.len(), 1);+ assert_eq!(after[0].retries, 1, "the retry must be counted");++ // Operator fixes the cause (here: restores the manifest from the backup+ // the storage error told them to look for) and retries again.+ std::fs::write(dir.path().join("broken").join("snapshot.json"), &good).unwrap();+ engine+ .retry_failed_index("broken")+ .expect("retry after the fix must reopen the index");++ assert!(engine.list_failed_indices().is_empty());+ assert_eq!(engine.list_indices().await.len(), 1);+ assert_ne!(engine.health().await.status, "red");+ // The index actually serves: the document survived the whole round trip.+ let idx = engine.get_index("broken").expect("reopened index");+ assert_eq!(idx.stats().await.doc_count, 1);++ // Retrying a name that is not a failed index is a 404, not a silent ok.+ let e = xerj_error(engine.retry_failed_index("broken").unwrap_err());+ assert!(matches!(e, XerjError::IndexNotFound { .. }), "{e:?}");+}