AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): fail closed on unsupported cluster state
bugerd8b243023f3c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9a9824dee..70cea4988 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -50,6 +50,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **A binary that does not understand `cluster_state.json` now fails closed+ before activating storage.** The previous loader accepted future and+ partially-understood shapes, dropped fields it did not know, and could later+ rewrite the reduced document over the original. Boot now accepts only the+ complete shipped format-1 envelope, including duplicate-key and nested-shape+ checks. A rejected document leaves liveness at 200 and readiness at 503, but+ opens no user index, Console system index, WAL, segment, or durable audit+ sink; Console bootstrap and storage background jobs remain disabled until a+ supported restart. After the existing request-size, authentication, and+ authorization gates, HTTP storage access returns the stable+ `cluster_state_unavailable` 503; authenticated gRPC calls return+ `UNAVAILABLE`. Existing 401/403/body-limit precedence is unchanged. Client+ responses give the category and recovery action without a local path or+ persisted object names; the server log carries the detailed classification+ and path for the operator. The original bytes and any legacy staging file+ stay in place, and a blocked boot creates no salvage copy. Format-1 rewrites+ retain the existing+ same-directory temp-file fsync and atomic rename; parent-directory sync is a+ best-effort attempt whose errors are ignored, so this change does not claim+ strict namespace durability across power loss. Data-directory creation,+ `node.lock` acquisition/PID diagnostics, and the server's earlier+ credential/TLS preparation retain their existing behavior; the no-open/no-+ replay claim begins at the cluster-state classification and covers storage+ stores, not those process-bootstrap files.+- **`PUT /{index}/_settings {"index.lifecycle.name": null}` actually detachesthe index, and the detach survives a restart** (#282, ported from #262). Thenull previously fell through a string-only settings reader, so the operator@@ -1009,7 +1034,8 @@ documented "at most one bar per 15 s" floor is really 12.5 s.over a file whose bytes were perfectly good. The load failure now latches:every management mutation is refused with a 500 naming the file, and thefile is not touched, until a boot loads it cleanly. The corrupt-parse arm- refuses too, and still keeps a `cluster_state.corrupt.json` copy.+ refuses too. The later format-compatibility fence above tightens that+ diagnostic boot further: it creates no `cluster_state.corrupt.json` copy.- `DELETE /_data_stream/<name>` recorded the removal before destroying thebacking indices, so a crash in that window stranded `.ds-<name>-00000N`directories that no data-stream API could reach — GET and DELETE answereddiff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 42844677e..793c18337 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -6442,6 +6442,7 @@ dependencies = ["prometheus","serde","serde_json",+ "sha2 0.10.9","tempfile","thiserror 2.0.18","tokio",@@ -6619,6 +6620,7 @@ dependencies = ["thiserror 2.0.18","tokio","tracing",+ "tracing-subscriber","uuid","xerj-ai","xerj-cluster",@@ -6730,6 +6732,7 @@ dependencies = ["tikv-jemalloc-ctl","tikv-jemallocator","tokio",+ "tokio-stream","tonic","tonic-build","tracing",diff --git a/engine/crates/xerj-api/Cargo.toml b/engine/crates/xerj-api/Cargo.tomlindex 3d06c3ea3..8ea6bcdf0 100644--- a/engine/crates/xerj-api/Cargo.toml+++ b/engine/crates/xerj-api/Cargo.toml@@ -44,3 +44,4 @@ libc = "0.2"tempfile = { workspace = true }tower = { workspace = true, features = ["util"] }http-body-util = "0.1"+sha2 = "0.10"diff --git a/engine/crates/xerj-api/src/error.rs b/engine/crates/xerj-api/src/error.rsindex 29fa420a0..00f8d8dd3 100644--- a/engine/crates/xerj-api/src/error.rs+++ b/engine/crates/xerj-api/src/error.rs@@ -302,6 +302,7 @@ fn xerj_error_type(e: &XerjError) -> String {XerjError::InvalidDocumentJson { .. } => "json_parse_exception",// ── Configuration ─────────────────────────────────────────────────XerjError::ConfigError { .. } => "action_request_validation_exception",+ XerjError::ClusterStateUnavailable => "cluster_state_unavailable",// ── Auth ──────────────────────────────────────────────────────────XerjError::AuthError { .. } => "security_exception",// ── TLS ───────────────────────────────────────────────────────────diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 7da2f1e96..a62918234 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -1116,7 +1116,9 @@ pub async fn create_index(if !mappings_val.is_null() {// Persists es_mapping.json into the index dir (atomic) so// the create-time mapping survives a restart.- state.engine.put_index_mapping(&index, mappings_val.clone());+ if let Err(error) = state.engine.put_index_mapping(&index, mappings_val.clone()) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}if let Some(aliases) = body.get("aliases").and_then(Value::as_object) {// Alias keys can also contain date math@@ -1127,7 +1129,9 @@ pub async fn create_index(for (alias, opts) in aliases {let resolved = resolve_date_math_index(alias);resolved_aliases.insert(resolved.clone(), opts.clone());- state.engine.add_alias(&resolved, &index);+ if let Err(error) = state.engine.add_alias(&resolved, &index) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}state.engine@@ -2314,7 +2318,9 @@ pub async fn put_mapping(for (idx_name, _, existing, _) in plans {// Persists es_mapping.json into the index dir (atomic) so the// merged mapping survives a restart.- state.engine.put_index_mapping(&idx_name, existing);+ if let Err(error) = state.engine.put_index_mapping(&idx_name, existing) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}Json(json!({ "acknowledged": true })).into_response()@@ -17292,12 +17298,16 @@ pub async fn post_aliases(match step {Resolved::Add(add, targets) => {for target in targets {- state.engine.add_alias(&add.alias, target);+ if let Err(error) = state.engine.add_alias(&add.alias, target) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}}Resolved::Remove(remove, targets) => {for target in targets {- state.engine.remove_alias(&remove.alias, target);+ if let Err(error) = state.engine.remove_alias(&remove.alias, target) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}}}@@ -17465,8 +17475,8 @@ pub async fn put_alias(.filter(|v| v.as_object().map(|o| !o.is_empty()).unwrap_or(false)).unwrap_or(json!({}));- let attach = |idx_name: &str| {- state.engine.add_alias(&alias, idx_name);+ let attach = |idx_name: &str| -> xerj_engine::Result<()> {+ state.engine.add_alias(&alias, idx_name)?;if !alias_meta.as_object().map(|o| o.is_empty()).unwrap_or(true) {let mut existing = state.engine@@ -17484,6 +17494,7 @@ pub async fn put_alias(.index_alias_metadata.insert(idx_name.to_string(), existing);}+ Ok(())};if targets.is_empty() {@@ -17499,7 +17510,9 @@ pub async fn put_alias(return ApiError::new(e).into_response();}for idx in &targets {- attach(idx);+ if let Err(error) = attach(idx) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}Json(json!({ "acknowledged": true })).into_response()}@@ -17510,10 +17523,14 @@ pub async fn delete_alias() -> impl IntoResponse {let targets = resolve_index_selector(&state, &index).await;if targets.is_empty() {- state.engine.remove_alias(&alias, &index);+ if let Err(error) = state.engine.remove_alias(&alias, &index) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }} else {for idx in &targets {- state.engine.remove_alias(&alias, idx);+ if let Err(error) = state.engine.remove_alias(&alias, idx) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}}Json(json!({ "acknowledged": true })).into_response()@@ -22400,7 +22417,9 @@ pub async fn put_settings(// tombstone for a ghost is unbounded state writable from// the public port (#262 hit exactly this in review).if state.engine.get_index(idx).is_ok() {- state.engine.detach_lifecycle(idx);+ if let Err(error) = state.engine.detach_lifecycle(idx) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}}}@@ -24408,7 +24427,9 @@ pub async fn put_ilm_policy(if let Err(e) = state.engine.put_ilm_policy(name.clone(), body.clone()) {return ApiError::new(xerj_common::XerjError::from(e)).into_response();}- state.engine.put_ism_policy(name, policy);+ if let Err(error) = state.engine.put_ism_policy(name, policy) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }Json(json!({ "acknowledged": true })).into_response()}@@ -24472,7 +24493,9 @@ pub async fn delete_ilm_policy(// lifecycle engine's storage rather than to this handler.match state.engine.delete_ilm_policy(&name) {Ok(true) => {- state.engine.remove_ism_policy(&name);+ if let Err(error) = state.engine.remove_ism_policy(&name) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }Json(json!({ "acknowledged": true })).into_response()}Ok(false) => {@@ -24612,7 +24635,9 @@ pub async fn remove_ilm_policy_from_index(Err(e) => return ApiError::new(e).into_response(),};for name in &targets {- state.engine.detach_lifecycle(name);+ if let Err(error) = state.engine.detach_lifecycle(name) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }}Json(json!({ "has_failures": false, "failed_indexes": [] })).into_response()}@@ -28110,7 +28135,7 @@ pub async fn security_create_api_key(// across restarts via `persist_api_key` (item 6: `<data_dir>/api_keys.json`).let now_ms = Utc::now().timestamp_millis().max(0) as u64;let expiration_ms = parse_api_key_expiration_ms(expiration.as_str(), now_ms);- state.engine.persist_api_key(+ if let Err(error) = state.engine.persist_api_key(key_id.clone(),xerj_engine::engine::ApiKeyRecord::new(name.clone(),@@ -28119,7 +28144,9 @@ pub async fn security_create_api_key(expiration_ms,roles,),- );+ ) {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }state.engine.audit.append("security.api_key.create",principal.label(),@@ -28511,7 +28538,12 @@ pub async fn security_invalidate_api_key(};let now_ms = Utc::now().timestamp_millis().max(0) as u64;- let (invalidated, previously) = state.engine.invalidate_api_keys(&target_ids, now_ms);+ let (invalidated, previously) = match state.engine.invalidate_api_keys(&target_ids, now_ms) {+ Ok(result) => result,+ Err(error) => {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }+ };state.engine.audit.append("security.api_key.invalidate",principal.label(),diff --git a/engine/crates/xerj-api/src/ism_api.rs b/engine/crates/xerj-api/src/ism_api.rsindex bf8aa38e5..31235b46b 100644--- a/engine/crates/xerj-api/src/ism_api.rs+++ b/engine/crates/xerj-api/src/ism_api.rs@@ -85,9 +85,12 @@ pub async fn put_ism_policy(let err = xerj_common::XerjError::invalid_query(format!("invalid ISM policy: {reason}"));return ApiError::new(err).into_response();}- state+ if let Err(error) = state.engine- .put_ism_policy(policy_id.clone(), policy.clone());+ .put_ism_policy(policy_id.clone(), policy.clone())+ {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }Json(json!({"_id": policy_id,"_version": 1,@@ -178,11 +181,14 @@ pub async fn delete_ism_policy(State(state): State<AppState>,Path(policy_id): Path<String>,) -> impl IntoResponse {- if state.engine.remove_ism_policy(&policy_id) {- Json(json!({ "_id": policy_id, "result": "deleted" })).into_response()- } else {- let e = xerj_common::XerjError::index_not_found(format!("policy [{policy_id}] not found"));- ApiError::new(e).into_response()+ match state.engine.remove_ism_policy(&policy_id) {+ Ok(true) => Json(json!({ "_id": policy_id, "result": "deleted" })).into_response(),+ Ok(false) => {+ let e =+ xerj_common::XerjError::index_not_found(format!("policy [{policy_id}] not found"));+ ApiError::new(e).into_response()+ }+ Err(error) => ApiError::new(xerj_common::XerjError::from(error)).into_response(),}}@@ -210,6 +216,9 @@ pub async fn add_ism_policy(/// `es_compat` (a single index↔policy association model, per the design —/// not two).pub async fn attach_policy(state: &AppState, index: &str, policy_id: &str) -> Response {+ if let Err(error) = state.engine.ensure_cluster_state_writable() {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }if state.engine.get_index(index).is_err() {let e = xerj_common::XerjError::index_not_found(index);return ApiError::new(e).into_response();@@ -235,7 +244,9 @@ pub async fn attach_policy(state: &AppState, index: &str, policy_id: &str) -> Re.engine.managed_indices.insert(index.to_string(), managed);- state.engine.persist_managed_indices();+ if let Err(error) = state.engine.persist_managed_indices() {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }Json(json!({"failures": false,@@ -258,12 +269,8 @@ pub async fn attach_policy(state: &AppState, index: &str, policy_id: &str) -> Re/// unambiguously "not managed", so a plain removal is the whole fix./// Returns whether the index was actually managed (false is not an error —/// detaching an already-unmanaged index is idempotent, matching real ES).-pub fn detach_ilm_policy(state: &AppState, index: &str) -> bool {- let removed = state.engine.managed_indices.remove(index).is_some();- if removed {- state.engine.persist_managed_indices();- }- removed+pub fn detach_ilm_policy(state: &AppState, index: &str) -> xerj_engine::Result<bool> {+ state.engine.detach_lifecycle(index)}// ─────────────────────────────────────────────────────────────────────────────@@ -278,8 +285,10 @@ pub async fn remove_ism_policy(// One shared detach path (engine method) for all three detach// routes, so the acknowledged detach is tombstoned and persisted// identically whichever surface the operator used (#282).- state.engine.detach_lifecycle(&index);- Json(single_index_success()).into_response()+ match state.engine.detach_lifecycle(&index) {+ Ok(_) => Json(single_index_success()).into_response(),+ Err(error) => ApiError::new(xerj_common::XerjError::from(error)).into_response(),+ }} else {Json(not_managed_failure(&index,@@ -320,6 +329,9 @@ pub async fn change_ism_policy(Path(index): Path<String>,Json(body): Json<ChangePolicyBody>,) -> impl IntoResponse {+ if let Err(error) = state.engine.ensure_cluster_state_writable() {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }let Some(mut managed) = state.engine.managed_indices@@ -383,7 +395,9 @@ pub async fn change_ism_policy(managed.info_message = format!("policy changed to '{}'", body.policy_id);managed.last_updated_ms = lifecycle::now_ms();state.engine.managed_indices.insert(index, managed);- state.engine.persist_managed_indices();+ if let Err(error) = state.engine.persist_managed_indices() {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }Json(single_index_success()).into_response()}@@ -412,6 +426,9 @@ pub async fn retry_ism_index(Path(index): Path<String>,body: OptionalJson<RetryBody>,) -> impl IntoResponse {+ if let Err(error) = state.engine.ensure_cluster_state_writable() {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }let body = body.0.unwrap_or_default();let Some(mut managed) = state.engine@@ -462,7 +479,9 @@ pub async fn retry_ism_index(managed.info_message = "retrying".to_string();managed.last_updated_ms = lifecycle::now_ms();state.engine.managed_indices.insert(index, managed);- state.engine.persist_managed_indices();+ if let Err(error) = state.engine.persist_managed_indices() {+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();+ }Json(single_index_success()).into_response()}diff --git a/engine/crates/xerj-api/src/native.rs b/engine/crates/xerj-api/src/native.rsindex 3aa0f1534..1721049cf 100644--- a/engine/crates/xerj-api/src/native.rs+++ b/engine/crates/xerj-api/src/native.rs@@ -731,6 +731,13 @@ pub async fn liveness() -> impl IntoResponse {}pub async fn readiness(State(state): State<AppState>) -> impl IntoResponse {+ if !state.engine.cluster_state_boot_status().is_writable() {+ return (+ axum::http::StatusCode::SERVICE_UNAVAILABLE,+ "not ready: persisted cluster metadata was not accepted at boot; inspect server logs",+ )+ .into_response();+ }let h = state.engine.health().await;// Read the node's own state, not a principal's filtered view: this path is// exempt from authentication so a kubelet can probe a hardened node, anddiff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rsindex 2d170650e..4ddf827c1 100644--- a/engine/crates/xerj-api/src/router.rs+++ b/engine/crates/xerj-api/src/router.rs@@ -12,7 +12,7 @@ use axum::{extract::{DefaultBodyLimit, Request, State},http::{HeaderValue, Method},middleware::{self, Next},- response::Response,+ response::{IntoResponse, Response},routing::{delete, get, post, put},Router,};@@ -173,6 +173,10 @@ pub fn build_native_router(state: AppState) -> Router {// spelling (`/v1/indices/{name}/…`), so it carries the same// authorization layer — otherwise the reserved `.xerj-memory-*`// namespace would be closed on :9200 and open on :8080.+ .layer(middleware::from_fn_with_state(+ state.clone(),+ storage_fence_middleware,+ )).layer(middleware::from_fn_with_state(state.clone(),authz::authz_middleware,@@ -886,6 +890,10 @@ pub fn build_es_compat_router(state: AppState) -> Router {// rather than a naming convention (issue #79); see `authz.rs` for the// full table of doors it closes. A superuser — `--insecure`, or the// configured admin key — short-circuits it on the first line.+ .layer(middleware::from_fn_with_state(+ state.clone(),+ storage_fence_middleware,+ )).layer(middleware::from_fn_with_state(state.clone(),authz::authz_middleware,@@ -916,6 +924,29 @@ pub fn build_es_compat_router(state: AppState) -> Router {// Middleware helpers// ─────────────────────────────────────────────────────────────────────────────+/// Admit only health probes while the boot-time persistence contract is+/// blocked. The engine deliberately opened no index in this state, so letting+/// an arbitrary handler continue would either lie from empty maps or find a+/// filesystem-shaped bypass (snapshot restore was the important example) and+/// activate storage after the preflight had refused it.+async fn storage_fence_middleware(+ State(state): State<AppState>,+ req: Request,+ next: Next,+) -> Response {+ if state.engine.cluster_state_boot_status().is_writable()+ || matches!(+ req.uri().path(),+ "/health/live" | "/health/ready" | "/v1/health/ready"+ )+ {+ return next.run(req).await;+ }++ crate::error::ApiError::from(xerj_common::XerjError::cluster_state_unavailable())+ .into_response()+}+/// Middleware that sets ES-compatible product headers on every response.////// Kibana and other Elastic clients verify the presence of `X-Elastic-Product`diff --git a/engine/crates/xerj-api/src/state.rs b/engine/crates/xerj-api/src/state.rsindex 34d7534bb..93a9ded0c 100644--- a/engine/crates/xerj-api/src/state.rs+++ b/engine/crates/xerj-api/src/state.rs@@ -494,8 +494,17 @@ impl AppState {"issuer": "xerj"})));let watcher_active = Arc::new(AtomicBool::new(true));- let ml_detectors = Arc::new(MlDetector::load_all(&config.server.data_dir));- let ml_datafeeds = Arc::new(MlDatafeed::load_all(&config.server.data_dir));+ // A blocked cluster-state boot is a diagnostic shell. Do not even+ // read auxiliary persisted registries: they are storage state whose+ // ownership may be described by the unsupported cluster metadata.+ let (ml_detectors, ml_datafeeds) = if engine.cluster_state_boot_status().is_writable() {+ (+ Arc::new(MlDetector::load_all(&config.server.data_dir)),+ Arc::new(MlDatafeed::load_all(&config.server.data_dir)),+ )+ } else {+ (Arc::new(DashMap::new()), Arc::new(DashMap::new()))+ };Self {config: Arc::new(config),engine: Arc::new(engine),diff --git a/engine/crates/xerj-api/tests/cluster_metadata_survives_restart.rs b/engine/crates/xerj-api/tests/cluster_metadata_survives_restart.rsindex d2bd3c85f..fab93f55b 100644--- a/engine/crates/xerj-api/tests/cluster_metadata_survives_restart.rs+++ b/engine/crates/xerj-api/tests/cluster_metadata_survives_restart.rs@@ -14,6 +14,7 @@use axum::body::Body;use axum::http::{Request, StatusCode};use serde_json::{json, Value};+use sha2::{Digest, Sha256};use tower::ServiceExt;fn config_for(dir: &std::path::Path) -> xerj_common::config::Config {@@ -27,11 +28,162 @@ fn config_for(dir: &std::path::Path) -> xerj_common::config::Config {/// dropped in between) is a restart: nothing is handed over in memory, the/// second boot sees only what reached the disk.fn app_over(dir: &std::path::Path) -> axum::Router {+ app_and_engine_over(dir).0+}++fn app_and_engine_over(dir: &std::path::Path) -> (axum::Router, xerj_engine::Engine) {+ let (es_app, _native_app, engine) = apps_and_engine_over(dir);+ (es_app, engine)+}++fn apps_and_engine_over(+ dir: &std::path::Path,+) -> (axum::Router, axum::Router, xerj_engine::Engine) {let config = config_for(dir);let metrics = xerj_common::metrics::Metrics::new().expect("metrics");let engine = xerj_engine::Engine::new(config.clone()).expect("engine");- let state = xerj_api::state::AppState::new(config, engine, metrics);- xerj_api::router::build_es_compat_router(state)+ let state = xerj_api::state::AppState::new(config, engine.clone(), metrics);+ (+ xerj_api::router::build_es_compat_router(state.clone()),+ xerj_api::router::build_native_router(state),+ engine,+ )+}++fn hex_digest(bytes: &[u8]) -> String {+ Sha256::digest(bytes)+ .iter()+ .map(|byte| format!("{byte:02x}"))+ .collect()+}++fn recursive_inventory(root: &std::path::Path) -> Vec<String> {+ fn walk(root: &std::path::Path, path: &std::path::Path, out: &mut Vec<String>) {+ let mut entries: Vec<_> = std::fs::read_dir(path)+ .expect("read inventory directory")+ .map(|entry| entry.expect("read inventory entry"))+ .collect();+ entries.sort_by_key(|entry| entry.file_name());+ for entry in entries {+ let entry_path = entry.path();+ let relative = entry_path+ .strip_prefix(root)+ .expect("inventory entry under root")+ .to_string_lossy()+ .into_owned();+ let kind = entry.file_type().expect("inventory file type");+ out.push(format!(+ "{}:{relative}",+ if kind.is_dir() {+ "d"+ } else if kind.is_symlink() {+ "l"+ } else {+ "f"+ }+ ));+ if kind.is_dir() {+ walk(root, &entry_path, out);+ }+ }+ }++ let mut out = Vec::new();+ walk(root, root, &mut out);+ out+}++fn recursive_storage_snapshot(root: &std::path::Path) -> Vec<String> {+ fn walk(root: &std::path::Path, path: &std::path::Path, out: &mut Vec<String>) {+ let mut entries: Vec<_> = std::fs::read_dir(path)+ .expect("read storage snapshot directory")+ .map(|entry| entry.expect("read storage snapshot entry"))+ .collect();+ entries.sort_by_key(|entry| entry.file_name());+ for entry in entries {+ let entry_path = entry.path();+ let relative = entry_path+ .strip_prefix(root)+ .expect("storage snapshot entry under root")+ .to_string_lossy();+ let kind = entry.file_type().expect("storage snapshot file type");+ if kind.is_dir() {+ out.push(format!("d:{relative}"));+ walk(root, &entry_path, out);+ } else if kind.is_symlink() {+ out.push(format!(+ "l:{relative}->{}",+ std::fs::read_link(&entry_path)+ .expect("read storage snapshot symlink")+ .to_string_lossy()+ ));+ } else {+ let bytes = std::fs::read(&entry_path).expect("read storage snapshot file");+ out.push(format!(+ "f:{relative}:{}:{}",+ bytes.len(),+ hex_digest(&bytes)+ ));+ }+ }+ }++ let mut out = Vec::new();+ walk(root, root, &mut out);+ out+}++#[derive(Debug, PartialEq, Eq)]+struct ManagedInventory {+ indices: Vec<String>,+ aliases: Vec<(String, Vec<String>)>,+ index_templates: Vec<String>,+ legacy_templates: Vec<String>,+ component_templates: Vec<String>,+ pipelines: Vec<String>,+ data_streams: Vec<String>,+ ilm_policies: Vec<String>,+ directory_entries: Vec<String>,+}++async fn managed_inventory(+ engine: &xerj_engine::Engine,+ dir: &std::path::Path,+) -> ManagedInventory {+ fn keys<V>(map: &dashmap::DashMap<String, V>) -> Vec<String> {+ let mut keys: Vec<_> = map.iter().map(|entry| entry.key().clone()).collect();+ keys.sort();+ keys+ }++ let mut indices: Vec<_> = engine+ .list_indices()+ .await+ .into_iter()+ .map(|index| index.name)+ .collect();+ indices.sort();+ let mut aliases: Vec<_> = engine+ .aliases+ .iter()+ .map(|entry| {+ let mut targets = entry.value().clone();+ targets.sort();+ (entry.key().clone(), targets)+ })+ .collect();+ aliases.sort();+ ManagedInventory {+ indices,+ aliases,+ index_templates: keys(&engine.templates),+ legacy_templates: keys(&engine.legacy_templates),+ component_templates: keys(&engine.component_templates),+ pipelines: keys(&engine.pipelines),+ data_streams: keys(&engine.data_streams),+ ilm_policies: keys(&engine.ilm_policies),+ directory_entries: recursive_inventory(dir),+ }}async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {@@ -65,6 +217,22 @@ async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {.await}+async fn get_text(app: &axum::Router, path: &str) -> (StatusCode, String) {+ let response = app+ .clone()+ .oneshot(Request::get(path).body(Body::empty()).expect("request"))+ .await+ .expect("response");+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .expect("body");+ (+ status,+ String::from_utf8(bytes.to_vec()).expect("utf8 body"),+ )+}+async fn delete(app: &axum::Router, path: &str) -> (StatusCode, Value) {send(app,@@ -342,8 +510,8 @@ async fn a_rollover_interrupted_before_its_flush_does_not_wedge_the_stream() {/// first `PUT /_index_template/...` rewrote the file — taking whatever was/// still legible in it along. Hand-recovery out of a damaged document is the/// operator's last option; it should not be silently removed. The write is-/// refused (the original stays where it is) *and* a copy is kept, because-/// un-wedging the node means moving the original aside by hand.+/// refused and the original stays where it is. A blocked diagnostic boot must+/// not create a second recovery file either.#[tokio::test(flavor = "multi_thread", worker_threads = 2)]async fn a_corrupt_cluster_state_is_preserved_before_it_is_overwritten() {let dir = tempfile::tempdir().expect("tempdir");@@ -359,14 +527,30 @@ async fn a_corrupt_cluster_state_is_preserved_before_it_is_overwritten() {let original = std::fs::read_to_string(&state_path).expect("read state");let truncated = &original[..original.len() / 2];std::fs::write(&state_path, truncated).expect("write truncated");+ let truncated_digest = hex_digest(truncated.as_bytes());- let app = app_over(dir.path());- let (st, _) = get(&app, "/_index_template/logs").await;+ let (app, native_app, engine) = apps_and_engine_over(dir.path());+ let (st, _) = get(&app, "/health/live").await;assert_eq!(st,- StatusCode::NOT_FOUND,- "a corrupt document cannot be restored — this is the honest outcome"+ StatusCode::OK,+ "malformed state blocks readiness, not liveness"+ );+ for (surface, path) in [(&app, "/health/ready"), (&native_app, "/v1/health/ready")] {+ let (st, body) = get_text(surface, path).await;+ assert_eq!(st, StatusCode::SERVICE_UNAVAILABLE, "{path}");+ assert_eq!(+ body,+ "not ready: persisted cluster metadata was not accepted at boot; inspect server logs"+ );+ }+ let (st, body) = get(&app, "/_index_template/logs").await;+ assert_eq!(+ st,+ StatusCode::SERVICE_UNAVAILABLE,+ "a corrupt document cannot be restored, but an empty-map 404 would be a lie: {body}");+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");// The write that would have destroyed the evidence.let (st, body) = put_json(@@ -377,31 +561,31 @@ async fn a_corrupt_cluster_state_is_preserved_before_it_is_overwritten() {.await;assert_eq!(st,- StatusCode::INTERNAL_SERVER_ERROR,+ StatusCode::SERVICE_UNAVAILABLE,"a management write on top of a state that did not load must be \refused, not acknowledged: {body}");- let salvage = dir.path().join("cluster_state.corrupt.json");assert!(- salvage.is_file(),- "the corrupt document must be kept for recovery"+ !dir.path().join("cluster_state.corrupt.json").exists(),+ "a blocked diagnostic boot must not create recovery files");assert_eq!(- std::fs::read_to_string(&salvage).expect("read salvage"),+ std::fs::read_to_string(&state_path).expect("read state"),truncated,- "the preserved copy must be the bytes that were found, unmodified"+ "the live document must be left exactly as it was found");assert_eq!(- std::fs::read_to_string(&state_path).expect("read state"),- truncated,- "the live document must be left exactly as it was found — the copy is \- a convenience, not a licence to overwrite the original"+ hex_digest(&std::fs::read(&state_path).expect("hash live malformed state")),+ truncated_digest,+ "the live malformed bytes must keep the exact SHA-256");// Moving the damaged file aside is the documented recovery, and it must// actually work: the next boot loads cleanly and writes are accepted again.drop(app);+ drop(native_app);+ drop(engine);std::fs::remove_file(&state_path).expect("operator moves the file aside");let app = app_over(dir.path());let (st, body) = put_json(@@ -491,15 +675,26 @@ async fn an_unreadable_cluster_state_is_never_overwritten() {"the failing target must be a symlink, not a directory");- let app = app_over(dir.path());+ let (app, native_app, engine) = apps_and_engine_over(dir.path());- let (st, _) = get(&app, "/_index_template/logs").await;+ let (st, _) = get(&app, "/health/live").await;+ assert_eq!(st, StatusCode::OK, "the blocked node remains live");+ for (surface, path) in [(&app, "/health/ready"), (&native_app, "/v1/health/ready")] {+ let (st, _) = get(surface, path).await;+ assert_eq!(+ st,+ StatusCode::SERVICE_UNAVAILABLE,+ "{path} must expose the boot-time cluster-state fence"+ );+ }++ let (st, body) = get(&app, "/_index_template/logs").await;assert_eq!(st,- StatusCode::NOT_FOUND,- "a document that could not be read cannot be restored — that part is \- honest already"+ StatusCode::SERVICE_UNAVAILABLE,+ "the diagnostic shell must not turn unloaded durable objects into a false 404: {body}");+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");// The write that used to destroy it.let (st, body) = put_json(@@ -510,17 +705,22 @@ async fn an_unreadable_cluster_state_is_never_overwritten() {.await;assert_eq!(st,- StatusCode::INTERNAL_SERVER_ERROR,+ StatusCode::SERVICE_UNAVAILABLE,"a management write must be refused while the persisted state is \unloaded, not acknowledged: {body}");- assert!(- body["error"]["reason"]- .as_str()- .unwrap_or_default()- .contains("could not be loaded at boot"),- "the 500 must say why, so the operator can act on it: {body}"- );+ assert_eq!(body["error"]["type"], "cluster_state_unavailable", "{body}");+ let reason = body["error"]["reason"].as_str().unwrap_or_default();+ for required in [+ "start a compatible XERJ build",+ "restore supported cluster_state.json bytes",+ "then restart",+ ] {+ assert!(+ reason.contains(required),+ "the refusal must give both bounded recovery actions: {body}"+ );+ }assert_eq!(std::fs::read_link(&state_path).expect("read cluster-state self-loop"),@@ -547,27 +747,36 @@ async fn an_unreadable_cluster_state_is_never_overwritten() {"a refused write must not even stage a uniquely named temp file");- // And the refusal is not a one-off: the in-memory map must not be left- // holding a change that was reported as failed.- let (st, _) = get(&app, "/_index_template/fresh").await;+ // And the refusal is not a one-off. Reads remain unavailable too: the+ // node did not open storage, so an empty-map 404 would be a lie.+ let (st, body) = get(&app, "/_index_template/fresh").await;assert_eq!(st,- StatusCode::NOT_FOUND,- "a refused PUT must roll its in-memory change back"+ StatusCode::SERVICE_UNAVAILABLE,+ "blocked reads must stay unavailable: {body}");+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");let (st, _) = delete(&app, "/_ilm/policy/hot-warm").await;assert_eq!(st,- StatusCode::INTERNAL_SERVER_ERROR,+ StatusCode::SERVICE_UNAVAILABLE,"deletes are writes too — they must be refused as well");// Recovery path: remove the bad directory entry, put the saved bytes back,// and restart. The latch belongs to the failed engine lifetime only.drop(app);+ drop(native_app);+ drop(engine);std::fs::remove_file(&state_path).expect("remove cluster-state self-loop");std::fs::rename(&saved_path, &state_path).expect("restore valid cluster state");let app = app_over(dir.path());+ let (st, _) = get(&app, "/health/ready").await;+ assert_eq!(+ st,+ StatusCode::OK,+ "valid replacement plus restart clears the fence"+ );let (st, body) = get(&app, "/_index_template/logs").await;assert_eq!(st,@@ -588,6 +797,465 @@ async fn an_unreadable_cluster_state_is_never_overwritten() {);}+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn cluster_state_format1_fence_refuses_every_current_metadata_mutation() {+ let dir = tempfile::tempdir().expect("tempdir");+ let future = include_bytes!(+ "../../xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json"+ );+ let state_path = dir.path().join("cluster_state.json");+ std::fs::write(&state_path, future).expect("write future state");++ // A complete-looking snapshot source proves the fence fires before the+ // restore handler can copy it into a new index and open its WAL. Keeping+ // the source under data_dir satisfies the normal repository allowlist.+ let snapshot_repo = dir.path().join("snapshot-repo");+ let snapshot = snapshot_repo.join("ready");+ let snapshot_index = snapshot.join("restored-index");+ std::fs::create_dir_all(snapshot_index.join("wal/s1")).expect("create snapshot fixture");+ std::fs::write(+ snapshot.join("manifest.json"),+ br#"{"indices":["restored-index"]}"#,+ )+ .expect("write snapshot manifest");+ std::fs::write(+ snapshot_index.join("wal/s1/0000000000000001.wal"),+ b"sentinel",+ )+ .expect("write snapshot WAL sentinel");++ let (app, native_app, engine) = apps_and_engine_over(dir.path());+ let before = managed_inventory(&engine, dir.path()).await;+ let before_storage = recursive_storage_snapshot(dir.path());+ assert!(before.indices.is_empty());+ assert!(before.aliases.is_empty());+ assert!(before.index_templates.is_empty());+ assert!(before.legacy_templates.is_empty());+ assert!(before.component_templates.is_empty());+ assert!(before.pipelines.is_empty());+ assert!(before.data_streams.is_empty());+ assert!(before.ilm_policies.is_empty());++ for (surface, path) in [(&app, "/health/ready"), (&native_app, "/v1/health/ready")] {+ let (status, body) = get_text(surface, path).await;+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{path}");+ assert_eq!(+ body,+ "not ready: persisted cluster metadata was not accepted at boot; inspect server logs",+ "the unauthenticated probe must not expose paths, object names, or state contents"+ );+ }+ let (status, _) = get(&app, "/health/live").await;+ assert_eq!(status, StatusCode::OK);++ // The fence is admission-wide, not a hand-maintained list of metadata+ // handlers. These routes cover read, user-index creation, direct ingest,+ // and the native surface; none may return an ordinary success/404 shape+ // from the deliberately empty diagnostic engine.+ let (status, response) = get(&app, "/").await;+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{response}");+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");+ let (status, response) = put_json(&app, "/blocked-index", json!({"mappings":{}})).await;+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{response}");+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");+ for (surface, path, body) in [+ (&app, "/_bulk", json!({"must":"not parse"})),+ (+ &native_app,+ "/v1/indices",+ json!({"name":"blocked-native","schema":{"fields":[]}}),+ ),+ ] {+ let (status, response) = post(surface, path, body).await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "{path}: {response}"+ );+ assert_eq!(+ response["error"]["type"], "cluster_state_unavailable",+ "{path}: {response}"+ );+ }++ let put_cases = [+ ("/_index_template/new", json!({"index_patterns":["new-*"]})),+ ("/_template/new", json!({"index_patterns":["legacy-*"]})),+ ("/_component_template/new", json!({"template":{}})),+ (+ "/_ilm/policy/new",+ json!({"policy":{"phases":{"hot":{"actions":{}}}}}),+ ),+ ("/_ingest/pipeline/new", json!({"processors":[]})),+ ("/_data_stream/new", json!({})),+ ];+ for (path, body) in put_cases {+ let (status, response) = put_json(&app, path, body).await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "PUT {path}: {response}"+ );+ assert_eq!(+ response["error"]["type"], "cluster_state_unavailable",+ "PUT {path}: {response}"+ );+ }++ let delete_cases = [+ "/_index_template/missing",+ "/_template/missing",+ "/_component_template/missing",+ "/_ilm/policy/missing",+ "/_ingest/pipeline/missing",+ "/_data_stream/missing",+ ];+ for path in delete_cases {+ let (status, response) = delete(&app, path).await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "DELETE {path}: {response}"+ );+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");+ }++ let (status, response) = post(&app, "/missing/_rollover", json!({})).await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "rollover must guard before stream lookup: {response}"+ );++ let (status, response) = put_json(+ &app,+ "/_snapshot/blocked-repo",+ json!({"type":"fs","settings":{"location":snapshot_repo}}),+ )+ .await;+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{response}");+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");+ let (status, response) = post(&app, "/_snapshot/blocked-repo/ready/_restore", json!({})).await;+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{response}");+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");++ let restore_err: xerj_common::XerjError = engine+ .restore_snapshot(+ snapshot_repo.to_str().expect("UTF-8 snapshot path"),+ "ready",+ None,+ )+ .await+ .expect_err("direct restore must share the storage fence")+ .into();+ assert!(matches!(+ restore_err,+ xerj_common::XerjError::ClusterStateUnavailable+ ));++ assert_eq!(+ std::fs::read(&state_path).expect("read future state"),+ future,+ "readiness and refused mutations must preserve future bytes"+ );+ assert_eq!(+ managed_inventory(&engine, dir.path()).await,+ before,+ "every refused mutation must preserve the exact alias, index, map, and directory inventories"+ );+ assert_eq!(+ recursive_storage_snapshot(dir.path()),+ before_storage,+ "refused HTTP and direct restore paths must not copy, open, replay, or rewrite storage"+ );+ assert!(+ std::fs::read_dir(dir.path())+ .unwrap()+ .filter_map(|entry| entry.ok())+ .all(|entry| !entry.file_name().to_string_lossy().starts_with(".ds-new-")),+ "refused data-stream creation must not create a backing directory"+ );+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn blocked_app_state_skips_ml_registries_and_http_preserves_auth_order() {+ const ADMIN: &str = "blocked-admin";+ let dir = tempfile::tempdir().expect("tempdir");+ std::fs::write(+ dir.path().join("cluster_state.json"),+ br#"{"format_version":2}"#,+ )+ .expect("write future cluster state");+ let ml_dir = dir.path().join("_ml");+ std::fs::create_dir_all(&ml_dir).expect("create ML registry dir");+ let detectors = br#"[{"detector_id":"seeded-detector","source_index":"logs","time_field":"@timestamp","function":"count","bucket_span":"1h","anomaly_threshold":3.0,"create_time_ms":1}]"#;+ let datafeeds = br#"[{"datafeed_id":"seeded-feed","job_id":"seeded-detector","indices":["logs"],"query":{"match_all":{}},"frequency_secs":60,"state":"started"}]"#;+ std::fs::write(ml_dir.join("detectors.json"), detectors).expect("seed detectors");+ std::fs::write(ml_dir.join("datafeeds.json"), datafeeds).expect("seed datafeeds");++ let mut config = config_for(dir.path());+ config.auth.enabled = true;+ config.auth.admin_api_key = ADMIN.to_owned();+ config.limits.max_body_bytes = 16;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("blocked engine");+ assert!(!engine.cluster_state_boot_status().is_writable());+ // Seed one non-superuser credential without using the deliberately fenced+ // persistence API. It can authenticate but has no reserved-index grant.+ engine.api_keys.insert(+ "limited".to_owned(),+ xerj_engine::engine::ApiKeyRecord::new("limited", "secret", 1, None, Vec::new()),+ );+ let before_state = recursive_storage_snapshot(dir.path());+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ assert!(+ state.ml_detectors.is_empty() && state.ml_datafeeds.is_empty(),+ "a blocked AppState must not load valid persisted ML entries"+ );+ assert_eq!(+ recursive_storage_snapshot(dir.path()),+ before_state,+ "constructing a blocked AppState must not rewrite any seeded ML byte"+ );++ let native_app = xerj_api::router::build_native_router(state.clone());+ let app = xerj_api::router::build_es_compat_router(state);+ let response = app+ .clone()+ .oneshot(+ Request::post("/_bulk")+ .header("authorization", format!("ApiKey {ADMIN}"))+ .header("content-type", "application/x-ndjson")+ // A real HTTP transport supplies Content-Length for this+ // fixed body. Set it explicitly because Router::oneshot does+ // not synthesize transport headers; without it the limit+ // layer can discover overflow only when an inner handler+ // reads the streaming body, and the storage fence correctly+ // prevents that handler from running.+ .header("content-length", "17")+ .body(Body::from("x".repeat(17)))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(+ response.status(),+ StatusCode::PAYLOAD_TOO_LARGE,+ "the existing request-size gate must keep precedence over the storage fence"+ );+ let response = app+ .clone()+ .oneshot(Request::get("/").body(Body::empty()).unwrap())+ .await+ .unwrap();+ assert_eq!(+ response.status(),+ StatusCode::UNAUTHORIZED,+ "authentication must run before the storage fence"+ );+ let response = native_app+ .clone()+ .oneshot(Request::get("/v1/indices").body(Body::empty()).unwrap())+ .await+ .unwrap();+ assert_eq!(+ response.status(),+ StatusCode::UNAUTHORIZED,+ "native authentication must run before the storage fence"+ );++ let response = app+ .clone()+ .oneshot(+ Request::get("/_memory/bob")+ .header("authorization", "ApiKey bGltaXRlZDpzZWNyZXQ=")+ .body(Body::empty())+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(+ response.status(),+ StatusCode::FORBIDDEN,+ "authorization must run before the storage fence"+ );+ let response = native_app+ .clone()+ .oneshot(+ Request::post("/v1/indices/.xerj-memory-bob-edges/search")+ .header("authorization", "ApiKey bGltaXRlZDpzZWNyZXQ=")+ .header("content-type", "application/json")+ .body(Body::from("{}"))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(+ response.status(),+ StatusCode::FORBIDDEN,+ "native authorization must run before the storage fence"+ );++ let response = app+ .clone()+ .oneshot(+ Request::get("/")+ .header("authorization", format!("ApiKey {ADMIN}"))+ .body(Body::empty())+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(+ response.status(),+ StatusCode::SERVICE_UNAVAILABLE,+ "an authenticated, authorized request must reach the storage fence"+ );+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .unwrap();+ let body: Value = serde_json::from_slice(&bytes).unwrap();+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");+ let response = native_app+ .clone()+ .oneshot(+ Request::get("/v1/indices")+ .header("authorization", format!("ApiKey {ADMIN}"))+ .body(Body::empty())+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(+ response.status(),+ StatusCode::SERVICE_UNAVAILABLE,+ "authorized native storage access must reach the storage fence"+ );+ assert_eq!(+ recursive_storage_snapshot(dir.path()),+ before_state,+ "auth and fence responses must not touch seeded ML registries"+ );+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn every_incompatible_format1_fixture_is_fenced_across_two_full_boots() {+ let fixtures = [+ "v1-unknown-top-level.json",+ "v1-unknown-index-template-field.json",+ "v1-unknown-data-stream-field.json",+ "v1-duplicate-version.json",+ "v1-duplicate-nested-value-key.json",+ "v2-format-version-minimal.json",+ ];++ for fixture in fixtures {+ let dir = tempfile::tempdir().expect("tempdir");+ let fixture_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))+ .join("../xerj-engine/tests/fixtures/cluster_state")+ .join(fixture);+ let original = std::fs::read(&fixture_path).expect("read fixture");+ let original_digest = hex_digest(&original);+ let state_path = dir.path().join("cluster_state.json");+ std::fs::write(&state_path, &original).expect("install fixture");+ let mut first_boot_inventory = None;++ for boot in 1..=2 {+ let (app, native_app, engine) = apps_and_engine_over(dir.path());+ let before = managed_inventory(&engine, dir.path()).await;+ assert!(before.indices.is_empty(), "{fixture}, boot {boot}");+ assert!(before.aliases.is_empty(), "{fixture}, boot {boot}");+ assert!(before.index_templates.is_empty(), "{fixture}, boot {boot}");+ assert!(before.legacy_templates.is_empty(), "{fixture}, boot {boot}");+ assert!(+ before.component_templates.is_empty(),+ "{fixture}, boot {boot}"+ );+ assert!(before.pipelines.is_empty(), "{fixture}, boot {boot}");+ assert!(before.data_streams.is_empty(), "{fixture}, boot {boot}");+ assert!(before.ilm_policies.is_empty(), "{fixture}, boot {boot}");+ if let Some(first) = &first_boot_inventory {+ assert_eq!(+ &before.directory_entries, first,+ "{fixture}: a refused restart must not add a directory entry"+ );+ } else {+ first_boot_inventory = Some(before.directory_entries.clone());+ }++ let (status, _) = get(&app, "/health/live").await;+ assert_eq!(status, StatusCode::OK, "{fixture}, boot {boot}");+ for (surface, path) in [(&app, "/health/ready"), (&native_app, "/v1/health/ready")] {+ let (status, body) = get_text(surface, path).await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "{fixture}, boot {boot}, {path}"+ );+ assert_eq!(+ body,+ "not ready: persisted cluster metadata was not accepted at boot; inspect server logs",+ "{fixture}, boot {boot}, {path}"+ );+ }++ let (status, body) = put_json(+ &app,+ "/_index_template/must-not-land",+ json!({"index_patterns": ["blocked-*"]}),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "{fixture}, boot {boot}, PUT: {body}"+ );+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");++ let (status, body) = delete(&app, "/_ilm/policy/must-not-land").await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "{fixture}, boot {boot}, DELETE: {body}"+ );+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");++ assert_eq!(+ managed_inventory(&engine, dir.path()).await,+ before,+ "{fixture}, boot {boot}: blocked mutations changed an alias, index, map, or directory"+ );+ let after = std::fs::read(&state_path).expect("read state after refused writes");+ assert_eq!(after, original, "{fixture}, boot {boot}");+ assert_eq!(+ hex_digest(&after),+ original_digest,+ "{fixture}, boot {boot}"+ );+ assert!(+ !dir.path().join("cluster_state.corrupt.json").exists(),+ "{fixture}: unsupported, unknown, and duplicate-key bytes are never salvage"+ );++ drop(app);+ drop(native_app);+ drop(engine);+ assert_eq!(+ recursive_inventory(dir.path()),+ first_boot_inventory.clone().unwrap(),+ "{fixture}, boot {boot}: shutdown changed the persisted inventory"+ );+ let after_shutdown = std::fs::read(&state_path).expect("read after shutdown");+ assert_eq!(after_shutdown, original, "{fixture}, boot {boot}");+ assert_eq!(+ hex_digest(&after_shutdown),+ original_digest,+ "{fixture}, boot {boot}"+ );+ }+ }+}+/// Concurrent writers must not trip over each other.////// The atomic rewrite stages through one fixed temp path, so two flushesdiff --git a/engine/crates/xerj-common/src/error.rs b/engine/crates/xerj-common/src/error.rsindex aecb43bd8..4f051bb13 100644--- a/engine/crates/xerj-common/src/error.rs+++ b/engine/crates/xerj-common/src/error.rs@@ -78,6 +78,14 @@ pub enum XerjError {#[error("config error: {reason}")]ConfigError { reason: String },+ /// Durable cluster metadata was not accepted during the one boot-time+ /// format check. The process remains live for health diagnosis, but it is+ /// not ready and must not activate or mutate storage.+ #[error(+ "cluster state is unavailable: start a compatible XERJ build or restore supported cluster_state.json bytes, then restart"+ )]+ ClusterStateUnavailable,+// ── Auth ───────────────────────────────────────────────────────────────/// Authentication or authorization failure.#[error("auth error: {reason}")]@@ -215,6 +223,10 @@ impl XerjError {}}+ pub fn cluster_state_unavailable() -> Self {+ Self::ClusterStateUnavailable+ }+pub fn auth(reason: impl Into<String>) -> Self {Self::AuthError {reason: reason.into(),@@ -289,7 +301,7 @@ impl XerjError {Self::IndexBlocked { .. } => 403,// The index exists but has no serving copy — ES answers the same// shape (no_shard_available_action_exception) with 503.- Self::IndexUnavailable { .. } => 503,+ Self::IndexUnavailable { .. } | Self::ClusterStateUnavailable => 503,Self::AuthError { .. } => 401,Self::ResourceExhausted { .. } | Self::CircuitBreaking { .. } => 429,Self::StorageError { .. }diff --git a/engine/crates/xerj-console-api/src/bootstrap.rs b/engine/crates/xerj-console-api/src/bootstrap.rsindex ab632efd5..736cf6eed 100644--- a/engine/crates/xerj-console-api/src/bootstrap.rs+++ b/engine/crates/xerj-console-api/src/bootstrap.rs@@ -48,6 +48,16 @@ pub async fn run(data_dir: &Path,bind_url: &str,) -> ConsoleResult<BootstrapOutcome> {+ // Console bootstrap is a storage migration: it creates fourteen system+ // indices, writes a master key and publishes seeded dashboard segments.+ // Keep this guard inside the crate as well as at the server call site so a+ // future embedder cannot accidentally bypass the compatibility fence.+ if !engine.cluster_state_boot_status().is_writable() {+ return Err(ConsoleApiError::Internal(+ "cluster-state storage is unavailable; Console bootstrap requires a supported restart"+ .to_owned(),+ ));+ }let master_key = load_or_init_master_key(data_dir)?;indices::ensure_all(engine)?;diff --git a/engine/crates/xerj-console-api/tests/cluster_state_fence.rs b/engine/crates/xerj-console-api/tests/cluster_state_fence.rsnew file mode 100644index 000000000..bef3a6655--- /dev/null+++ b/engine/crates/xerj-console-api/tests/cluster_state_fence.rs@@ -0,0 +1,82 @@+use xerj_common::config::{Config, WalSync};+use xerj_console_api::bootstrap;+use xerj_engine::{ClusterStateBlockKind, Engine};++fn config_for(path: &std::path::Path) -> Config {+ let mut config = Config::default();+ config.server.data_dir = path.to_string_lossy().into_owned();+ config.storage.wal_sync = WalSync::Async;+ config+}++fn snapshot(root: &std::path::Path) -> Vec<(String, Vec<u8>)> {+ fn walk(root: &std::path::Path, path: &std::path::Path, out: &mut Vec<(String, Vec<u8>)>) {+ let mut entries: Vec<_> = std::fs::read_dir(path)+ .expect("read snapshot directory")+ .map(|entry| entry.expect("read snapshot entry"))+ .collect();+ entries.sort_by_key(|entry| entry.file_name());+ for entry in entries {+ let entry_path = entry.path();+ let relative = entry_path+ .strip_prefix(root)+ .expect("snapshot path under root")+ .to_string_lossy()+ .into_owned();+ let kind = entry.file_type().expect("snapshot file type");+ if kind.is_dir() {+ out.push((format!("d:{relative}"), Vec::new()));+ walk(root, &entry_path, out);+ } else if kind.is_symlink() {+ out.push((+ format!("l:{relative}"),+ std::fs::read_link(&entry_path)+ .expect("read snapshot symlink")+ .to_string_lossy()+ .as_bytes()+ .to_vec(),+ ));+ } else {+ out.push((+ format!("f:{relative}"),+ std::fs::read(&entry_path).expect("read snapshot file"),+ ));+ }+ }+ }++ let mut out = Vec::new();+ walk(root, root, &mut out);+ out+}++#[tokio::test]+async fn console_bootstrap_is_side_effect_free_when_cluster_state_is_blocked() {+ let dir = tempfile::tempdir().expect("tempdir");+ let future = include_bytes!(+ "../../xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json"+ );+ std::fs::write(dir.path().join("cluster_state.json"), future).expect("write future state");++ let engine = Engine::new(config_for(dir.path())).expect("blocked diagnostic engine");+ assert_eq!(+ engine.cluster_state_boot_status().block_kind(),+ Some(&ClusterStateBlockKind::UnsupportedFormat)+ );+ let before = snapshot(dir.path());++ let result = bootstrap::run(&engine, dir.path(), "http://127.0.0.1:9200").await;+ assert!(+ result.is_err(),+ "Console bootstrap must refuse a blocked storage node"+ );++ assert_eq!(snapshot(dir.path()), before);+ assert!(!dir.path().join(".xerj_master_key").exists());+ for system_index in xerj_console_api::indices::ALL {+ assert!(+ !dir.path().join(system_index).exists(),+ "must not create or open {system_index}"+ );+ }+}diff --git a/engine/crates/xerj-engine/Cargo.toml b/engine/crates/xerj-engine/Cargo.tomlindex 6a6bbe93b..dbe9b3020 100644--- a/engine/crates/xerj-engine/Cargo.toml+++ b/engine/crates/xerj-engine/Cargo.toml@@ -59,6 +59,7 @@ xerj-storage = { path = "../xerj-storage", features = ["test-hooks"] }tempfile = { workspace = true }tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }criterion = { workspace = true }+tracing-subscriber = { workspace = true }[[bench]]name = "search_bench"diff --git a/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs b/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rsindex 17e56ad11..f134843b5 100644--- a/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs+++ b/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs@@ -51,10 +51,12 @@ fn engine_with_index(name: &str, semantic: bool) -> (TempDir, Engine) {}engine.create_index(name, schema).expect("create index");if semantic {- engine.put_index_mapping(- name,- json!({"properties": {"content": {"type": "semantic_text"}}}),- );+ engine+ .put_index_mapping(+ name,+ json!({"properties": {"content": {"type": "semantic_text"}}}),+ )+ .unwrap();}(dir, engine)}diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rsindex 9e9181d35..e3540e16d 100644--- a/engine/crates/xerj-engine/src/bulk.rs+++ b/engine/crates/xerj-engine/src/bulk.rs@@ -2547,7 +2547,9 @@ mod semantic_lifecycle_tests {.iter().map(|(field, _)| ((*field).to_string(), json!({"type": "semantic_text"}))).collect::<serde_json::Map<_, _>>();- engine.put_index_mapping(name, json!({"properties": properties}));+ engine+ .put_index_mapping(name, json!({"properties": properties}))+ .unwrap();(dir, engine)}@@ -3123,10 +3125,12 @@ mod semantic_lifecycle_tests {});schema.fields.push(content);engine.create_index(fails, schema).unwrap();- engine.put_index_mapping(- fails,- json!({"properties": {"content": {"type": "semantic_text"}}}),- );+ engine+ .put_index_mapping(+ fails,+ json!({"properties": {"content": {"type": "semantic_text"}}}),+ )+ .unwrap();let ledger: &'static Ledger = Box::leak(Box::new(Ledger::new()));let engine = Arc::new(engine);let hook_engine = Arc::clone(&engine);diff --git a/engine/crates/xerj-engine/src/cluster_state.rs b/engine/crates/xerj-engine/src/cluster_state.rsnew file mode 100644index 000000000..f6bf27565--- /dev/null+++ b/engine/crates/xerj-engine/src/cluster_state.rs@@ -0,0 +1,700 @@+//! Closed persistence contract for `cluster_state.json` format 1.+//!+//! This module is intentionally persistence-only. Public request DTOs stay+//! forward-compatible at the API boundary; bytes that can later be rewritten+//! as durable cluster metadata are accepted only when their complete format-1+//! schema is understood.++use std::collections::{BTreeMap, HashSet};+use std::path::Path;++use serde::de::{Error as _, MapAccess, SeqAccess, Visitor};+use serde::{Deserialize, Deserializer, Serialize};+use serde_json::{Map, Number, Value};+use tracing::{error, warn};++use crate::engine::{DataStream, IndexTemplate};++pub const CLUSTER_STATE_V1: u32 = 1;++const DUPLICATE_KEY_MARKER: &str = "duplicate JSON object key";++/// Stable reason why this process may not rewrite cluster metadata.+#[derive(Debug, Clone, PartialEq, Eq)]+pub enum ClusterStateBlockKind {+ ReadFailure,+ MalformedJson,+ DuplicateKey,+ MissingDiscriminator,+ AmbiguousDiscriminator,+ InvalidDiscriminator,+ UnsupportedFormat,+ IncompatibleSchema,+}++/// Immutable result of the one boot-time classification.+#[derive(Debug, Clone, PartialEq, Eq)]+pub enum ClusterStateBootStatus {+ Absent,+ LoadedV1,+ Blocked {+ kind: ClusterStateBlockKind,+ detail: String,+ },+}++impl ClusterStateBootStatus {+ pub fn is_writable(&self) -> bool {+ matches!(self, Self::Absent | Self::LoadedV1)+ }++ pub fn block_kind(&self) -> Option<&ClusterStateBlockKind> {+ match self {+ Self::Blocked { kind, .. } => Some(kind),+ Self::Absent | Self::LoadedV1 => None,+ }+ }++ pub(crate) fn detail(&self) -> Option<&str> {+ match self {+ Self::Blocked { detail, .. } => Some(detail),+ Self::Absent | Self::LoadedV1 => None,+ }+ }+}++#[derive(Debug)]+pub(crate) struct PreparedClusterState {+ pub(crate) status: ClusterStateBootStatus,+ pub(crate) document: Option<PersistedClusterStateV1>,+}++impl PreparedClusterState {+ fn blocked(path: &Path, kind: ClusterStateBlockKind, detail: impl Into<String>) -> Self {+ let detail = detail.into();+ error!(+ path = %path.display(),+ kind = ?kind,+ detail = %detail,+ "cluster state was rejected at boot; start a compatible XERJ build or restore supported bytes, then restart"+ );+ Self {+ status: ClusterStateBootStatus::Blocked { kind, detail },+ document: None,+ }+ }+}++/// Exact seven-field envelope emitted by the only shipped format-1 writer.+#[derive(Debug, Clone, Serialize, Deserialize)]+#[serde(deny_unknown_fields)]+pub(crate) struct PersistedClusterStateV1 {+ pub(crate) version: u32,+ pub(crate) index_templates: BTreeMap<String, PersistedIndexTemplateV1>,+ pub(crate) legacy_templates: BTreeMap<String, Value>,+ pub(crate) component_templates: BTreeMap<String, Value>,+ pub(crate) pipelines: BTreeMap<String, Value>,+ pub(crate) data_streams: BTreeMap<String, PersistedDataStreamV1>,+ pub(crate) ilm_policies: BTreeMap<String, Value>,+}++#[derive(Debug, Clone, Serialize, Deserialize)]+#[serde(deny_unknown_fields)]+pub(crate) struct PersistedIndexTemplateV1 {+ index_patterns: Vec<String>,+ settings: Value,+ mappings: Value,+ priority: i32,+}++impl From<IndexTemplate> for PersistedIndexTemplateV1 {+ fn from(value: IndexTemplate) -> Self {+ Self {+ index_patterns: value.index_patterns,+ settings: value.settings,+ mappings: value.mappings,+ priority: value.priority,+ }+ }+}++impl From<PersistedIndexTemplateV1> for IndexTemplate {+ fn from(value: PersistedIndexTemplateV1) -> Self {+ Self {+ index_patterns: value.index_patterns,+ settings: value.settings,+ mappings: value.mappings,+ priority: value.priority,+ }+ }+}++#[derive(Debug, Clone, Serialize, Deserialize)]+#[serde(deny_unknown_fields)]+pub(crate) struct PersistedDataStreamV1 {+ name: String,+ backing_indices: Vec<String>,+ timestamp_field: String,+ generation: u64,+}++impl From<DataStream> for PersistedDataStreamV1 {+ fn from(value: DataStream) -> Self {+ Self {+ name: value.name,+ backing_indices: value.backing_indices,+ timestamp_field: value.timestamp_field,+ generation: value.generation,+ }+ }+}++impl From<PersistedDataStreamV1> for DataStream {+ fn from(value: PersistedDataStreamV1) -> Self {+ Self {+ name: value.name,+ backing_indices: value.backing_indices,+ timestamp_field: value.timestamp_field,+ generation: value.generation,+ }+ }+}++/// Read and fully classify the cluster-state path once, before any index is+/// opened. The returned document is the only input later applied to live maps.+pub(crate) fn preflight(data_dir: &Path) -> PreparedClusterState {+ let path = data_dir.join("cluster_state.json");+ let bytes = match std::fs::read(&path) {+ Ok(bytes) => bytes,+ Err(err) if err.kind() == std::io::ErrorKind::NotFound => {+ return PreparedClusterState {+ status: ClusterStateBootStatus::Absent,+ document: None,+ };+ }+ Err(err) => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::ReadFailure,+ format!("read failed: {err}"),+ );+ }+ };++ let value = match parse_unique_json(&bytes) {+ Ok(value) => value,+ Err(ParseFailure::Duplicate(detail)) => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::DuplicateKey,+ detail,+ );+ }+ Err(ParseFailure::Malformed(detail)) => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::MalformedJson,+ detail,+ );+ }+ };++ let object = match value.as_object() {+ Some(object) => object,+ None => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::IncompatibleSchema,+ "format-1 cluster state must be a JSON object",+ );+ }+ };++ let legacy = object.get("version");+ let future = object.get("format_version");+ match (legacy, future) {+ (Some(_), Some(_)) => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::AmbiguousDiscriminator,+ "both version and format_version are present",+ );+ }+ (None, Some(found)) => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::UnsupportedFormat,+ format!(+ "unsupported cluster-state format: found format_version={}, supported version={CLUSTER_STATE_V1}",+ discriminator_label(found)+ ),+ );+ }+ (None, None) => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::MissingDiscriminator,+ "required version discriminator is missing",+ );+ }+ (Some(found), None) => match found.as_u64() {+ Some(v) if v == u64::from(CLUSTER_STATE_V1) => {}+ Some(v) if v > u64::from(CLUSTER_STATE_V1) => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::UnsupportedFormat,+ format!(+ "unsupported cluster-state version: found={v}, supported={CLUSTER_STATE_V1}"+ ),+ );+ }+ _ => {+ return PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::InvalidDiscriminator,+ format!(+ "version must be the integer {CLUSTER_STATE_V1}, found {}",+ discriminator_label(found)+ ),+ );+ }+ },+ }++ match serde_json::from_value::<PersistedClusterStateV1>(value) {+ Ok(document) => {+ // Only a document that was completely classified as the exact+ // writable format 1 authorizes cleanup of the writer's fixed+ // staging name. A future/malformed document (or an absent live+ // path) may belong to another binary or an interrupted install;+ // a diagnostic-shell boot must not delete or create anything.+ let legacy_tmp = path.with_extension("tmp");+ if legacy_tmp.exists() {+ warn!("discarding an incomplete format-1 cluster-state staging file");+ let _ = std::fs::remove_file(legacy_tmp);+ }+ PreparedClusterState {+ status: ClusterStateBootStatus::LoadedV1,+ document: Some(document),+ }+ }+ Err(err) => PreparedClusterState::blocked(+ &path,+ ClusterStateBlockKind::IncompatibleSchema,+ format!("format-1 schema is not understood: {err}"),+ ),+ }+}++fn discriminator_label(value: &Value) -> String {+ match value {+ Value::String(value) => format!("string {value:?}"),+ other => other.to_string(),+ }+}++enum ParseFailure {+ Duplicate(String),+ Malformed(String),+}++fn parse_unique_json(bytes: &[u8]) -> Result<Value, ParseFailure> {+ let mut deserializer = serde_json::Deserializer::from_slice(bytes);+ let value = UniqueValue::deserialize(&mut deserializer).map_err(classify_parse_error)?;+ deserializer.end().map_err(classify_parse_error)?;+ Ok(value.0)+}++fn classify_parse_error(err: serde_json::Error) -> ParseFailure {+ let detail = err.to_string();+ if detail.contains(DUPLICATE_KEY_MARKER) {+ ParseFailure::Duplicate(detail)+ } else {+ ParseFailure::Malformed(detail)+ }+}++struct UniqueValue(Value);++impl<'de> Deserialize<'de> for UniqueValue {+ fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>+ where+ D: Deserializer<'de>,+ {+ deserializer.deserialize_any(UniqueValueVisitor)+ }+}++struct UniqueValueVisitor;++impl<'de> Visitor<'de> for UniqueValueVisitor {+ type Value = UniqueValue;++ fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ formatter.write_str("one JSON value without duplicate object keys")+ }++ fn visit_bool<E>(self, value: bool) -> Result<Self::Value, E> {+ Ok(UniqueValue(Value::Bool(value)))+ }++ fn visit_i64<E>(self, value: i64) -> Result<Self::Value, E> {+ Ok(UniqueValue(Value::Number(value.into())))+ }++ fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E> {+ Ok(UniqueValue(Value::Number(value.into())))+ }++ fn visit_f64<E>(self, value: f64) -> Result<Self::Value, E>+ where+ E: serde::de::Error,+ {+ Number::from_f64(value)+ .map(Value::Number)+ .map(UniqueValue)+ .ok_or_else(|| E::custom("non-finite JSON number"))+ }++ fn visit_str<E>(self, value: &str) -> Result<Self::Value, E> {+ Ok(UniqueValue(Value::String(value.to_owned())))+ }++ fn visit_string<E>(self, value: String) -> Result<Self::Value, E> {+ Ok(UniqueValue(Value::String(value)))+ }++ fn visit_none<E>(self) -> Result<Self::Value, E> {+ Ok(UniqueValue(Value::Null))+ }++ fn visit_unit<E>(self) -> Result<Self::Value, E> {+ Ok(UniqueValue(Value::Null))+ }++ fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>+ where+ D: Deserializer<'de>,+ {+ UniqueValue::deserialize(deserializer)+ }++ fn visit_seq<A>(self, mut sequence: A) -> Result<Self::Value, A::Error>+ where+ A: SeqAccess<'de>,+ {+ let mut values = Vec::new();+ while let Some(value) = sequence.next_element::<UniqueValue>()? {+ values.push(value.0);+ }+ Ok(UniqueValue(Value::Array(values)))+ }++ fn visit_map<A>(self, mut object: A) -> Result<Self::Value, A::Error>+ where+ A: MapAccess<'de>,+ {+ let mut seen = HashSet::new();+ let mut values = Map::new();+ while let Some(key) = object.next_key::<String>()? {+ if !seen.insert(key.clone()) {+ return Err(A::Error::custom(format!("{DUPLICATE_KEY_MARKER} {key:?}")));+ }+ let value = object.next_value::<UniqueValue>()?;+ values.insert(key, value.0);+ }+ Ok(UniqueValue(Value::Object(values)))+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[derive(Clone, Default)]+ struct LogBuffer(std::sync::Arc<std::sync::Mutex<Vec<u8>>>);++ struct LogWriter(std::sync::Arc<std::sync::Mutex<Vec<u8>>>);++ impl std::io::Write for LogWriter {+ fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {+ self.0.lock().unwrap().extend_from_slice(bytes);+ Ok(bytes.len())+ }++ fn flush(&mut self) -> std::io::Result<()> {+ Ok(())+ }+ }++ impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for LogBuffer {+ type Writer = LogWriter;++ fn make_writer(&'a self) -> Self::Writer {+ LogWriter(self.0.clone())+ }+ }++ fn capture_preflight_log(dir: &Path) -> (PreparedClusterState, String) {+ let buffer = LogBuffer::default();+ let subscriber = tracing_subscriber::fmt()+ .without_time()+ .with_ansi(false)+ .with_target(false)+ .with_writer(buffer.clone())+ .finish();+ let guard = tracing::subscriber::set_default(subscriber);+ let prepared = preflight(dir);+ drop(guard);+ let bytes = buffer.0.lock().unwrap().clone();+ (prepared, String::from_utf8(bytes).unwrap())+ }++ fn exact_v1(extra: &str) -> Vec<u8> {+ format!(+ r#"{{+ "version": 1,+ "index_templates": {{}},+ "legacy_templates": {{}},+ "component_templates": {{}},+ "pipelines": {{}},+ "data_streams": {{}},+ "ilm_policies": {{{extra}}}+ }}"#+ )+ .into_bytes()+ }++ fn preflight_bytes(bytes: &[u8]) -> PreparedClusterState {+ let dir = tempfile::tempdir().unwrap();+ std::fs::write(dir.path().join("cluster_state.json"), bytes).unwrap();+ preflight(dir.path())+ }++ #[test]+ fn cluster_state_format1_parser_accepts_exact_v1_and_open_values() {+ let prepared = preflight_bytes(&exact_v1(+ r#""policy": {"arbitrary": {"future-user-key": [1, true, null]}}"#,+ ));+ assert_eq!(prepared.status, ClusterStateBootStatus::LoadedV1);+ assert!(prepared.document.is_some());+ }++ #[test]+ fn cluster_state_format1_parser_blocks_every_invalid_discriminator_shape() {+ let cases: &[(&[u8], ClusterStateBlockKind)] = &[+ (br#"{}"#, ClusterStateBlockKind::MissingDiscriminator),+ (+ br#"{"version":0}"#,+ ClusterStateBlockKind::InvalidDiscriminator,+ ),+ (+ br#"{"version":-1}"#,+ ClusterStateBlockKind::InvalidDiscriminator,+ ),+ (+ br#"{"version":1.0}"#,+ ClusterStateBlockKind::InvalidDiscriminator,+ ),+ (+ br#"{"version":"1"}"#,+ ClusterStateBlockKind::InvalidDiscriminator,+ ),+ (+ br#"{"version":null}"#,+ ClusterStateBlockKind::InvalidDiscriminator,+ ),+ (+ br#"{"version":4294967296}"#,+ ClusterStateBlockKind::UnsupportedFormat,+ ),+ (+ br#"{"version":2}"#,+ ClusterStateBlockKind::UnsupportedFormat,+ ),+ (+ br#"{"format_version":2}"#,+ ClusterStateBlockKind::UnsupportedFormat,+ ),+ (+ br#"{"version":1,"format_version":2}"#,+ ClusterStateBlockKind::AmbiguousDiscriminator,+ ),+ ];+ for (bytes, expected) in cases {+ let prepared = preflight_bytes(bytes);+ assert_eq!(prepared.status.block_kind(), Some(expected), "{bytes:?}");+ }+ }++ #[test]+ fn cluster_state_format1_parser_reports_exact_found_and_supported_versions() {+ let prepared = preflight_bytes(br#"{"format_version":2}"#);+ assert_eq!(+ prepared.status,+ ClusterStateBootStatus::Blocked {+ kind: ClusterStateBlockKind::UnsupportedFormat,+ detail:+ "unsupported cluster-state format: found format_version=2, supported version=1"+ .to_owned(),+ }+ );++ let prepared = preflight_bytes(br#"{"version":2}"#);+ assert_eq!(+ prepared.status,+ ClusterStateBootStatus::Blocked {+ kind: ClusterStateBlockKind::UnsupportedFormat,+ detail: "unsupported cluster-state version: found=2, supported=1".to_owned(),+ }+ );+ }++ #[test]+ fn cluster_state_format1_parser_rejects_trailing_non_whitespace() {+ let mut bytes = exact_v1("");+ bytes.extend_from_slice(b" true");+ let prepared = preflight_bytes(&bytes);+ assert_eq!(+ prepared.status.block_kind(),+ Some(&ClusterStateBlockKind::MalformedJson)+ );+ }++ #[test]+ fn every_representative_block_class_logs_kind_detail_and_operator_path_at_boot() {+ let cases: &[(&[u8], ClusterStateBlockKind, &str)] = &[+ (+ br#"{"format_version":2}"#,+ ClusterStateBlockKind::UnsupportedFormat,+ "found format_version=2, supported version=1",+ ),+ (+ br#"{"version":1}"#,+ ClusterStateBlockKind::IncompatibleSchema,+ "format-1 schema is not understood",+ ),+ (+ br#"{"version":1,"version":1}"#,+ ClusterStateBlockKind::DuplicateKey,+ "duplicate JSON object key",+ ),+ ];+ for (bytes, kind, detail) in cases {+ let dir = tempfile::tempdir().unwrap();+ let path = dir.path().join("cluster_state.json");+ std::fs::write(&path, bytes).unwrap();+ let (prepared, log) = capture_preflight_log(dir.path());+ assert_eq!(prepared.status.block_kind(), Some(kind));+ assert!(+ log.contains("cluster state was rejected at boot"),+ "{kind:?}: {log}"+ );+ assert!(+ log.contains(+ "start a compatible XERJ build or restore supported bytes, then restart"+ ),+ "{kind:?}: {log}"+ );+ assert!(log.contains(&format!("kind={kind:?}")), "{kind:?}: {log}");+ assert!(log.contains(detail), "{kind:?}: {log}");+ assert!(log.contains(&path.display().to_string()), "{kind:?}: {log}");+ }+ }++ #[cfg(unix)]+ #[test]+ fn read_failure_logs_the_same_boot_diagnostic_contract() {+ use std::os::unix::fs::symlink;++ let dir = tempfile::tempdir().unwrap();+ let path = dir.path().join("cluster_state.json");+ symlink("cluster_state.json", &path).unwrap();+ let (prepared, log) = capture_preflight_log(dir.path());+ assert_eq!(+ prepared.status.block_kind(),+ Some(&ClusterStateBlockKind::ReadFailure)+ );+ assert!(log.contains("kind=ReadFailure"), "{log}");+ assert!(log.contains("read failed:"), "{log}");+ assert!(log.contains(&path.display().to_string()), "{log}");+ }++ #[test]+ fn cluster_state_format1_parser_rejects_duplicates_at_every_depth() {+ let top = exact_v1(r#""a": {}, "a": {}"#);+ let nested = exact_v1(r#""a": {"policy": {"x": 1, "x": 2}}"#);+ for bytes in [&top, &nested] {+ let prepared = preflight_bytes(bytes);+ assert_eq!(+ prepared.status.block_kind(),+ Some(&ClusterStateBlockKind::DuplicateKey)+ );+ }+ }++ #[test]+ fn cluster_state_format1_parser_rejects_unknown_and_missing_typed_fields() {+ let unknown_envelope = {+ let mut bytes = exact_v1("");+ let end = bytes.iter().rposition(|byte| *byte == b'}').unwrap();+ bytes.splice(end..end, b",\"future\":true".iter().copied());+ bytes+ };+ let unknown_template = br#"{+ "version":1,+ "index_templates":{"x":{"index_patterns":[],"settings":{},"mappings":{},"priority":0,"future":true}},+ "legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{}+ }"#;+ let missing_map = br#"{+ "version":1,"index_templates":{},"legacy_templates":{},+ "component_templates":{},"pipelines":{},"data_streams":{}+ }"#;+ for bytes in [unknown_envelope.as_slice(), unknown_template, missing_map] {+ let prepared = preflight_bytes(bytes);+ assert_eq!(+ prepared.status.block_kind(),+ Some(&ClusterStateBlockKind::IncompatibleSchema)+ );+ }+ }++ #[test]+ fn blocked_or_absent_preflight_neither_cleans_staging_nor_writes_salvage() {+ let cases: &[Option<&[u8]>] = &[+ Some(br#"{"format_version":2}"#),+ Some(br#"{"version":1,"cut":true"#),+ None,+ ];+ for live in cases {+ let dir = tempfile::tempdir().unwrap();+ let state = dir.path().join("cluster_state.json");+ if let Some(bytes) = live {+ std::fs::write(&state, bytes).unwrap();+ }+ let staging = dir.path().join("cluster_state.tmp");+ let staging_bytes = b"owned-by-another-or-interrupted-writer";+ std::fs::write(&staging, staging_bytes).unwrap();++ let prepared = preflight(dir.path());+ assert!(+ !prepared.status.is_writable() || live.is_none(),+ "fixture must not classify as loaded format 1"+ );+ assert_eq!(std::fs::read(&staging).unwrap(), staging_bytes);+ assert!(!dir.path().join("cluster_state.corrupt.json").exists());+ }+ }++ #[test]+ fn loaded_format1_alone_authorizes_legacy_staging_cleanup() {+ let dir = tempfile::tempdir().unwrap();+ std::fs::write(dir.path().join("cluster_state.json"), exact_v1("")).unwrap();+ let staging = dir.path().join("cluster_state.tmp");+ std::fs::write(&staging, b"interrupted format-1 stage").unwrap();++ let prepared = preflight(dir.path());+ assert_eq!(prepared.status, ClusterStateBootStatus::LoadedV1);+ assert!(!staging.exists());+ }+}diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex 08b645e04..e81f6e742 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -12,6 +12,10 @@ use tracing::{error, info, warn};use xerj_common::config::Config;use xerj_common::types::{IndexName, Schema};+use crate::cluster_state::{+ ClusterStateBootStatus, PersistedClusterStateV1, PersistedDataStreamV1,+ PersistedIndexTemplateV1, PreparedClusterState, CLUSTER_STATE_V1,+};use crate::index::{Index, IndexStats};use crate::{EngineError, Result};@@ -138,45 +142,6 @@ pub struct IndexTemplate {pub priority: i32,}-/// Version stamped into `cluster_state.json`. Bump only when the on-disk-/// shape changes in a way an older build cannot read; new *optional* fields-/// (`#[serde(default)]`) do not need one.-const CLUSTER_STATE_VERSION: u32 = 1;--/// Everything in `<data_dir>/cluster_state.json` — the cluster-level-/// management state that used to be in-memory only (issue #203).-///-/// `BTreeMap` rather than `HashMap` so the file is byte-stable for a given-/// state: an operator diffing two nodes' `cluster_state.json`, or a backup-/// tool deduplicating it, sees a change only when the state really changed.-/// Every field is `#[serde(default)]`, so a file written by an older build-/// (or by one that gains another map later) still loads.-#[derive(Debug, Default, Serialize, Deserialize)]-struct PersistedClusterState {- #[serde(default)]- version: u32,- /// v2 index templates — `PUT /_index_template/{name}`.- #[serde(default)]- index_templates: std::collections::BTreeMap<String, IndexTemplate>,- /// v1 legacy templates — `PUT /_template/{name}`.- #[serde(default)]- legacy_templates: std::collections::BTreeMap<String, Value>,- /// Component templates — `PUT /_component_template/{name}`.- #[serde(default)]- component_templates: std::collections::BTreeMap<String, Value>,- /// Ingest pipeline definitions — `PUT /_ingest/pipeline/{id}`. Stored as- /// the document `GET` hands back, and recompiled into an executable- /// pipeline on boot.- #[serde(default)]- pipelines: std::collections::BTreeMap<String, Value>,- /// Data streams — `PUT /_data_stream/{name}` plus every rollover.- #[serde(default)]- data_streams: std::collections::BTreeMap<String, DataStream>,- /// ILM policies — `PUT /_ilm/policy/{name}`.- #[serde(default)]- ilm_policies: std::collections::BTreeMap<String, Value>,-}-/// Active scroll context holding all matching hits.////// Each context pins a fully-hydrated `Vec<Hit>` snapshot, so its lifetime@@ -654,11 +619,10 @@ pub struct Engine {/// Never taken while holding a `DashMap` guard.cluster_state_write: Arc<parking_lot::Mutex<()>>,- /// False once `<data_dir>/cluster_state.json` was found on disk and could- /// not be loaded — unreadable (`EACCES` after a uid change on a container- /// volume, `EIO`, `EMFILE` at boot) or unparseable. While false,- /// `flush_cluster_state` refuses to write and every management PUT- /// answers 500.+ /// Immutable classification of the one `cluster_state.json` document read+ /// after `node.lock` and before any index/WAL open. A blocked status keeps+ /// readiness false and refuses every metadata mutation for this process+ /// lifetime; recovery requires replacing the bytes and restarting.////// Without this the failure is silent *and* destructive. The maps boot/// empty, so the first `PUT /_index_template/...` snapshots six empty@@ -683,7 +647,7 @@ pub struct Engine {/// (`redb/src/tree_store/page_store/cached_file.rs:125-145`,/// `redb/src/error.rs:65`; Apache-2.0/MIT, shape adapted, no code/// copied).- cluster_state_loaded: Arc<std::sync::atomic::AtomicBool>,+ cluster_state_boot: Arc<ClusterStateBootStatus>,}impl Engine {@@ -715,6 +679,15 @@ impl Engine {// process serves the same directory.let node_lock = Arc::new(Self::acquire_node_lock(&data_dir)?);+ // Classify durable cluster metadata before Index::open can replay a+ // WAL or startup reconciliation can rewrite it. The parsed document+ // is retained and applied later; the path is never reread this boot.+ let PreparedClusterState {+ status: cluster_state_status,+ document: prepared_cluster_state,+ } = crate::cluster_state::preflight(&data_dir);+ let storage_available = cluster_state_status.is_writable();+// Apply operator-tunable aggregation bucket cap. Stored in a static// AtomicUsize inside aggs.rs so all per-bucket-allocator hot loops// can read it with no plumbing through every agg signature.@@ -784,10 +757,18 @@ impl Engine {// is evidence of. Falls back to in-memory (with a warning) if the// file cannot be opened — an unwritable audit log must not stop// the node booting.- audit: crate::audit::AuditLog::open(- crate::audit::DEFAULT_AUDIT_CAPACITY,- data_dir.join("audit.jsonl"),- ),+ // A blocked compatibility-fence boot is a diagnostic shell, not a+ // partially-open storage node. Opening the durable audit sink+ // rewrites audit.jsonl from its ring during construction, so keep+ // even that non-index store memory-only until a supported restart.+ audit: if storage_available {+ crate::audit::AuditLog::open(+ crate::audit::DEFAULT_AUDIT_CAPACITY,+ data_dir.join("audit.jsonl"),+ )+ } else {+ crate::audit::AuditLog::new(crate::audit::DEFAULT_AUDIT_CAPACITY)+ },roles: crate::rbac::RoleStore::new(),// Single-node default: 1 shard, "local" owner. Writes never// forward; multi-node mode overrides these via the Raft@@ -798,134 +779,145 @@ impl Engine {)),_node_lock: node_lock,cluster_state_write: Arc::new(parking_lot::Mutex::new(())),- cluster_state_loaded: Arc::new(std::sync::atomic::AtomicBool::new(true)),+ cluster_state_boot: Arc::new(cluster_state_status),};- // Scan data_dir for existing index directories.- if let Ok(read_dir) = std::fs::read_dir(&data_dir) {- for entry in read_dir.flatten() {- let path = entry.path();- if !path.is_dir() {- continue;- }- // Check if this looks like an index directory (has a WAL subdirectory).- if !path.join("wal").exists() {- continue;- }- let name_str = match path.file_name().and_then(|n| n.to_str()) {- Some(n) => n.to_string(),- None => continue,- };- let index_name = match IndexName::new(&name_str) {- Ok(n) => n,- Err(_) => {- warn!("Skipping directory '{}': not a valid index name", name_str);+ // A rejected cluster-state document means this binary cannot know the+ // storage topology that document describes. Do not open *any* index in+ // that state: Index::open replays WALs and may publish a segment. This+ // includes `.xerj_*` system indices; treating them as harmless was the+ // exact path that mutated `.xerj_dashboards` during the causal A/B.+ if storage_available {+ // Scan data_dir for existing index directories.+ if let Ok(read_dir) = std::fs::read_dir(&data_dir) {+ for entry in read_dir.flatten() {+ let path = entry.path();+ if !path.is_dir() {continue;}- };- match Index::open(index_name.clone(), &engine.config, &data_dir) {- Ok(idx) => {- info!(name = name_str.as_str(), "opened existing index");- // Restore the raw ES mapping blob (analyzers, formats,- // dims — full fidelity) BEFORE any ingest/query can run,- // so GET /_mapping and mapping-dependent code paths see- // the same mapping as pre-restart. A corrupt blob fails- // the index (#202) rather than serving it with a- // silently reduced mapping.- if let Err(e) = engine.load_persisted_es_mapping(&name_str) {- warn!(name = name_str.as_str(), error = %e, "failed to open index");- engine.record_failed_index(&name_str, e.to_string());+ // Check if this looks like an index directory (has a WAL subdirectory).+ if !path.join("wal").exists() {+ continue;+ }+ let name_str = match path.file_name().and_then(|n| n.to_str()) {+ Some(n) => n.to_string(),+ None => continue,+ };+ let index_name = match IndexName::new(&name_str) {+ Ok(n) => n,+ Err(_) => {+ warn!("Skipping directory '{}': not a valid index name", name_str);continue;}- // The index isn't registered yet, so the propagation- // inside load can't find it — set the toggles on the- // local handle instead.- if let Some(m) = engine.index_mappings.get(name_str.as_str()) {- Engine::apply_date_mapping_flags(&idx, m.value());+ };+ match Index::open(index_name.clone(), &engine.config, &data_dir) {+ Ok(idx) => {+ info!(name = name_str.as_str(), "opened existing index");+ // Restore the raw ES mapping blob (analyzers, formats,+ // dims — full fidelity) BEFORE any ingest/query can run,+ // so GET /_mapping and mapping-dependent code paths see+ // the same mapping as pre-restart. A corrupt blob fails+ // the index (#202) rather than serving it with a+ // silently reduced mapping.+ if let Err(e) = engine.load_persisted_es_mapping(&name_str) {+ warn!(name = name_str.as_str(), error = %e, "failed to open index");+ engine.record_failed_index(&name_str, e.to_string());+ continue;+ }+ // The index isn't registered yet, so the propagation+ // inside load can't find it — set the toggles on the+ // local handle instead.+ if let Some(m) = engine.index_mappings.get(name_str.as_str()) {+ Engine::apply_date_mapping_flags(&idx, m.value());+ }+ engine.load_or_backfill_index_created_at(&name_str);+ engine.indices.insert(name_str, idx);}- engine.load_or_backfill_index_created_at(&name_str);- engine.indices.insert(name_str, idx);- }- Err(e) => {- warn!(name = name_str.as_str(), error = %e, "failed to open index");- // The mapping blob lives beside the data, not inside- // the store, so it is readable even when the store- // refuses to open. Load it so the metadata surfaces- // (`GET /{index}`, `GET /{index}/_mapping`) can still- // tell the operator what was in the index they are- // trying to recover. Propagation into the (absent)- // handle no-ops. Since #202 this load can itself fail- // (unreadable/unparseable `es_mapping.json`); the index- // is already being quarantined for the store error, so- // the extra failure is reported rather than dropped —- // it tells the operator a second file needs repairing.- if let Err(map_err) = engine.load_persisted_es_mapping(&name_str) {- warn!(- name = name_str.as_str(),- error = %map_err,- "es_mapping.json is also unreadable; the mapping surfaces \- cannot show what this index held"- );+ Err(e) => {+ warn!(name = name_str.as_str(), error = %e, "failed to open index");+ // The mapping blob lives beside the data, not inside+ // the store, so it is readable even when the store+ // refuses to open. Load it so the metadata surfaces+ // (`GET /{index}`, `GET /{index}/_mapping`) can still+ // tell the operator what was in the index they are+ // trying to recover. Propagation into the (absent)+ // handle no-ops. Since #202 this load can itself fail+ // (unreadable/unparseable `es_mapping.json`); the index+ // is already being quarantined for the store error, so+ // the extra failure is reported rather than dropped —+ // it tells the operator a second file needs repairing.+ if let Err(map_err) = engine.load_persisted_es_mapping(&name_str) {+ warn!(+ name = name_str.as_str(),+ error = %map_err,+ "es_mapping.json is also unreadable; the mapping surfaces \+ cannot show what this index held"+ );+ }+ engine.record_failed_index(&name_str, e.to_string());}- engine.record_failed_index(&name_str, e.to_string());}}}- }- // Restore persisted API keys (item 6) so keys minted before a restart- // still authenticate. Must run before the server starts accepting- // requests (i.e. here in `new`), and is cheap (one small JSON file).- engine.load_persisted_api_keys();-- // Restore persisted aliases so e.g. `.kibana` (always an alias,- // never a bare index) still resolves after a restart — see- // `load_persisted_aliases`'s doc comment for the concrete failure- // this caused (OpenSearch Dashboards stuck indefinitely on every- // restart, mistaking a missing-alias 404 for a still-in-progress- // migration by another instance).- engine.load_persisted_aliases();-- // Restore the cluster-level management state — index templates,- // legacy templates, component templates, ingest pipelines, data- // streams, ILM policies (issue #203). Before this, "replace the- // binary and restart" silently reverted every one of them: the- // documents kept flowing, into indices that no longer had the shape- // the operator designed. Must run after the index scan above, so a- // restored data stream can report which of its backing indices are- // actually present, and before the server accepts requests.- engine.load_cluster_state();-- // Then say plainly which `.ds-*` indices no restored stream claims.- // Nothing is deleted — see the doc comment; this is the log line that- // stops an unreachable backing index from being invisible.- engine.warn_orphaned_backing_indices();-- // Restore ISM/ILM policies and managed-index execution state so a- // policy attached before a restart keeps running afterward instead- // of silently going idle. Separate files (`ism_policies.json`,- // `ism_managed_indices.json`) from `cluster_state.json` above, and- // deliberately so: that file holds the verbatim documents an- // operator PUT, this holds the executor's own cursor. Loaded after- // `load_cluster_state` because a restored data stream is what a- // rollover action operates on.- engine.load_persisted_ism_policies();- engine.load_persisted_managed_indices();-- // Spawn the PIT sweeper. Pre-v0.6.2 PITs accumulated forever;- // every open without close was a memory leak. The sweeper- // walks `engine.pits` every `pit.sweep_interval_secs` and- // drops any with `expires_at < now`. Cheap (DashMap iter +- // Instant compare) and bounded by the live PIT count.- engine.spawn_pit_sweeper();-- // Spawn the scroll + async-search context sweeper (RC4 blocker- // 11) — the scroll/async twin of the PIT sweeper above. Each- // scroll pins a full Vec<Hit> and each async search pins its- // response JSON; without a sweeper both maps grow until an- // explicit client DELETE, i.e. forever under normal client- // behavior.- engine.spawn_search_context_sweeper();+ // Restore persisted API keys (item 6) so keys minted before a restart+ // still authenticate. Must run before the server starts accepting+ // requests (i.e. here in `new`), and is cheap (one small JSON file).+ engine.load_persisted_api_keys();++ // Restore persisted aliases so e.g. `.kibana` (always an alias,+ // never a bare index) still resolves after a restart — see+ // `load_persisted_aliases`'s doc comment for the concrete failure+ // this caused (OpenSearch Dashboards stuck indefinitely on every+ // restart, mistaking a missing-alias 404 for a still-in-progress+ // migration by another instance).+ engine.load_persisted_aliases();++ // Restore the cluster-level management state — index templates,+ // legacy templates, component templates, ingest pipelines, data+ // streams, ILM policies (issue #203). Before this, "replace the+ // binary and restart" silently reverted every one of them: the+ // documents kept flowing, into indices that no longer had the shape+ // the operator designed. Must run after the index scan above, so a+ // restored data stream can report which of its backing indices are+ // actually present, and before the server accepts requests.+ engine.apply_cluster_state(prepared_cluster_state);++ // Then say plainly which `.ds-*` indices no restored stream claims.+ // Nothing is deleted — see the doc comment; this is the log line that+ // stops an unreachable backing index from being invisible.+ engine.warn_orphaned_backing_indices();++ // Restore ISM/ILM policies and managed-index execution state so a+ // policy attached before a restart keeps running afterward instead+ // of silently going idle. Separate files (`ism_policies.json`,+ // `ism_managed_indices.json`) from `cluster_state.json` above, and+ // deliberately so: that file holds the verbatim documents an+ // operator PUT, this holds the executor's own cursor. Loaded after+ // `apply_cluster_state` because a restored data stream is what a+ // rollover action operates on.+ engine.load_persisted_ism_policies();+ engine.load_persisted_managed_indices();++ // Spawn the PIT sweeper. Pre-v0.6.2 PITs accumulated forever;+ // every open without close was a memory leak. The sweeper+ // walks `engine.pits` every `pit.sweep_interval_secs` and+ // drops any with `expires_at < now`. Cheap (DashMap iter ++ // Instant compare) and bounded by the live PIT count.+ engine.spawn_pit_sweeper();++ // Spawn the scroll + async-search context sweeper (RC4 blocker+ // 11) — the scroll/async twin of the PIT sweeper above. Each+ // scroll pins a full Vec<Hit> and each async search pins its+ // response JSON; without a sweeper both maps grow until an+ // explicit client DELETE, i.e. forever under normal client+ // behavior.+ engine.spawn_search_context_sweeper();+ } else {+ warn!(+ "storage activation skipped because the boot-time cluster-state format fence is latched"+ );+ }Ok(engine)}@@ -1188,6 +1180,11 @@ impl Engine {schema: Schema,settings: serde_json::Value,) -> Result<()> {+ // Index creation is storage activation: it creates the directory and+ // WAL before any cluster-state serializer is involved. A blocked boot+ // must reject here so HTTP, gRPC, direct CLI, and Console callers share+ // the same fail-closed boundary.+ self.ensure_cluster_state_writable()?;let index_name = IndexName::new(name).map_err(EngineError::Common)?;// Per-index authorization backstop (issue #79): a caller that may not@@ -1241,7 +1238,8 @@ impl Engine {////// This is the single write path for `engine.index_mappings` — both/// index-create-with-mappings and PUT /_mapping go through here.- pub fn put_index_mapping(&self, name: &str, mapping: Value) {+ pub fn put_index_mapping(&self, name: &str, mapping: Value) -> Result<()> {+ self.ensure_cluster_state_writable()?;let index_dir = self.data_dir.join(name);if index_dir.is_dir() {match serde_json::to_vec_pretty(&mapping) {@@ -1259,6 +1257,7 @@ impl Engine {}self.propagate_date_detection(name, &mapping);self.index_mappings.insert(name.to_string(), mapping);+ Ok(())}/// Push the mapping's `date_detection` toggle (default true) down to the@@ -1375,12 +1374,15 @@ impl Engine {/// mutation snapshots the full map to `<data_dir>/api_keys.json` (0600,/// atomic temp+rename), reloaded on boot.///- /// The write is best-effort: a persistence failure is logged but does not- /// fail the create — the key still works until the next restart, matching- /// the old behavior rather than regressing key creation.- pub fn persist_api_key(&self, id: String, record: ApiKeyRecord) {+ /// A blocked boot returns the typed compatibility error before inserting+ /// the key. Once storage is writable, an ordinary persistence I/O failure+ /// remains best-effort: it is logged but does not fail the create — the key+ /// still works until the next restart, matching the old behavior.+ pub fn persist_api_key(&self, id: String, record: ApiKeyRecord) -> Result<()> {+ self.ensure_cluster_state_writable()?;self.api_keys.insert(id, record);self.flush_api_keys_best_effort();+ Ok(())}/// Invalidate (revoke) minted API keys by id — issue #208's missing half.@@ -1398,7 +1400,12 @@ impl Engine {/// middleware reads this same map, so revocation takes effect on the very/// next request — and the store is flushed to `api_keys.json` once at the/// end, same durability contract as [`Self::persist_api_key`].- pub fn invalidate_api_keys(&self, ids: &[String], now_ms: u64) -> (Vec<String>, Vec<String>) {+ pub fn invalidate_api_keys(+ &self,+ ids: &[String],+ now_ms: u64,+ ) -> Result<(Vec<String>, Vec<String>)> {+ self.ensure_cluster_state_writable()?;let mut invalidated = Vec::new();let mut previously = Vec::new();for id in ids {@@ -1418,7 +1425,7 @@ impl Engine {if !invalidated.is_empty() {self.flush_api_keys_best_effort();}- (invalidated, previously)+ Ok((invalidated, previously))}/// Serialize the current `api_keys` map to `<data_dir>/api_keys.json`@@ -1634,17 +1641,24 @@ impl Engine {/// store, mirroring `flush_aliases`. Both `_plugins/_ism/policies` (as/// entered) and `_ilm/policy` (after translation) write through here —/// see the `ism_policies` field doc.- pub fn put_ism_policy(&self, id: String, policy: crate::lifecycle::LifecyclePolicy) {+ pub fn put_ism_policy(+ &self,+ id: String,+ policy: crate::lifecycle::LifecyclePolicy,+ ) -> Result<()> {+ self.ensure_cluster_state_writable()?;self.ism_policies.insert(id, policy);self.flush_ism_policies();+ Ok(())}- pub fn remove_ism_policy(&self, id: &str) -> bool {+ pub fn remove_ism_policy(&self, id: &str) -> Result<bool> {+ self.ensure_cluster_state_writable()?;let removed = self.ism_policies.remove(id).is_some();if removed {self.flush_ism_policies();}- removed+ Ok(removed)}fn flush_ism_policies(&self) {@@ -1717,7 +1731,8 @@ impl Engine {/// [...]}` — so the acknowledged detaches survive a restart alongside/// the cursors. `load_persisted_managed_indices` still accepts the/// pre-#282 bare-map form.- pub fn persist_managed_indices(&self) {+ pub fn persist_managed_indices(&self) -> Result<()> {+ self.ensure_cluster_state_writable()?;let managed: std::collections::HashMap<String, crate::lifecycle::ManagedIndexState> = self.managed_indices.iter()@@ -1734,12 +1749,13 @@ impl Engine {Ok(b) => b,Err(e) => {warn!(error = %e, "failed to serialize managed_indices for persistence");- return;+ return Ok(());}};if let Err(e) = crate::index::write_file_atomic(&self.managed_indices_path(), &bytes) {warn!(error = %e, "failed to persist ism_managed_indices.json (state works until restart)");}+ Ok(())}fn load_persisted_managed_indices(&self) {@@ -1798,7 +1814,8 @@ impl Engine {/// future settings-derived re-attach would trip over.////// Returns whether the index was managed before the call.- pub fn detach_lifecycle(&self, index: &str) -> bool {+ pub fn detach_lifecycle(&self, index: &str) -> Result<bool> {+ self.ensure_cluster_state_writable()?;let was_managed = self.managed_indices.remove(index).is_some();self.lifecycle_detached.insert(index.to_string(), ());if let Some(mut stored) = self.index_settings.get_mut(index) {@@ -1813,8 +1830,8 @@ impl Engine {lifecycle.remove("name");}}- self.persist_managed_indices();- was_managed+ self.persist_managed_indices()?;+ Ok(was_managed)}/// Whether `lifecycle::tick` is allowed to act — the `POST /_ilm/stop`@@ -1921,7 +1938,7 @@ impl Engine {// engines use for their manifests (cf. `fjall/src/file.rs:17`// `fsync_directory`, which fjall calls after every atomic rewrite;// `sled/src/metadata_store.rs:696` discards leftover `*.tmp` files from- // an interrupted rewrite on the next boot, which `load_cluster_state`+ // an interrupted rewrite on the next boot, which `apply_cluster_state`// mirrors below).//// One file rather than six keeps the maps mutually consistent — a data@@ -1934,54 +1951,67 @@ impl Engine {self.data_dir.join("cluster_state.json")}- /// `Err` when boot found `cluster_state.json` and could not load it — see- /// `cluster_state_loaded` for why that has to latch.+ /// `Err` when boot did not classify `cluster_state.json` as absent or as+ /// the exact supported format 1.////// Every management mutation calls this, not only the ones that reach/// `flush_cluster_state` with something to write. A `DELETE` consults the/// in-memory map first, and while the load has failed that map is empty,/// so an unguarded delete answers `404 not found` about an object that is/// sitting in the document on disk — the same lie in a different shape.- /// A 500 that names the file is the only honest answer while the node- /// cannot see the operator's configuration.- fn ensure_cluster_state_writable(&self) -> Result<()> {- if self- .cluster_state_loaded- .load(std::sync::atomic::Ordering::Acquire)- {+ /// A typed 503 is the only honest answer while the node cannot see the+ /// operator's configuration.+ pub fn ensure_cluster_state_writable(&self) -> Result<()> {+ if self.cluster_state_boot.is_writable() {return Ok(());}let path = self.cluster_state_path();+ let detail = self+ .cluster_state_boot+ .detail()+ .unwrap_or("cluster state was blocked at boot");error!(path = %path.display(),- "refusing a cluster-metadata write: this node could not load \- cluster_state.json at boot, so its management state is not the \- operator's. Recover or move the file aside, then restart."+ detail,+ "refusing a cluster-metadata write because the boot-time format fence is latched");- Err(EngineError::Common(xerj_common::XerjError::storage(- format!(- "refusing to write {} — it could not be loaded at boot, so \- writing now would destroy cluster configuration that is still \- on disk; recover or move it aside and restart xerj",- path.display()- ),- )))+ Err(EngineError::Common(+ xerj_common::XerjError::cluster_state_unavailable(),+ ))}/// Snapshot every persisted management map into one document.- fn cluster_state_snapshot(&self) -> PersistedClusterState {+ fn cluster_state_snapshot(&self) -> PersistedClusterStateV1 {fn dump<V: Clone>(map: &DashMap<String, V>) -> std::collections::BTreeMap<String, V> {map.iter().map(|e| (e.key().clone(), e.value().clone())).collect()}- PersistedClusterState {- version: CLUSTER_STATE_VERSION,- index_templates: dump(&self.templates),+ PersistedClusterStateV1 {+ version: CLUSTER_STATE_V1,+ index_templates: self+ .templates+ .iter()+ .map(|entry| {+ (+ entry.key().clone(),+ PersistedIndexTemplateV1::from(entry.value().clone()),+ )+ })+ .collect(),legacy_templates: dump(&self.legacy_templates),component_templates: dump(&self.component_templates),pipelines: dump(&self.pipelines),- data_streams: dump(&self.data_streams),+ data_streams: self+ .data_streams+ .iter()+ .map(|entry| {+ (+ entry.key().clone(),+ PersistedDataStreamV1::from(entry.value().clone()),+ )+ })+ .collect(),ilm_policies: dump(&self.ilm_policies),}}@@ -1995,7 +2025,7 @@ impl Engine {/// the moment of the write instead of at the next restart.////// Refuses outright when boot could not load the existing document — see- /// `cluster_state_loaded`. The snapshot below is taken from the live maps,+ /// the boot-time cluster-state fence. The snapshot below is taken from the live maps,/// which in that state hold only what this process was told after boot, so/// writing it would rename an empty-ish document over configuration that is/// still intact on disk.@@ -2005,6 +2035,9 @@ impl Engine {// inside the lock also means the bytes that land are never older// than a rewrite that has already returned.let _writing = self.cluster_state_write.lock();+ // The final writer owns the fence too. A future caller that misses a+ // preguard still cannot normalize an unsupported document.+ self.ensure_cluster_state_writable()?;let snapshot = self.cluster_state_snapshot();let bytes = serde_json::to_vec_pretty(&snapshot)?;crate::index::write_file_atomic(&self.cluster_state_path(), &bytes)?;@@ -2031,107 +2064,16 @@ impl Engine {.unwrap_or(0)}- /// Load `<data_dir>/cluster_state.json` back into the management maps.- ///- /// A missing file is normal (fresh node, or a data dir written by a build- /// that predates this). A file that is present but does not load is loud- /// *and* latching: it means the operator's templates, pipelines and data- /// streams are not coming back, which is exactly the silence this whole- /// mechanism exists to remove, so `cluster_state_loaded` goes false and- /// every subsequent management write is refused rather than allowed to- /// rename a snapshot of empty maps over the document on disk.- fn load_cluster_state(&self) {- // An interrupted rewrite can leave a partial `cluster_state.tmp`- // behind. It is never read (the rename is the commit point), but- // leaving it litters the data dir and invites a hand-edit that- // "restores" a torn document — sled sweeps the same leftovers on- // boot (`sled/src/metadata_store.rs:696`).- let tmp = self.cluster_state_path().with_extension("tmp");- if tmp.exists() {- warn!(- path = %tmp.display(),- "discarding an incomplete cluster_state rewrite left by an unclean shutdown"- );- let _ = std::fs::remove_file(&tmp);- }-- let path = self.cluster_state_path();- let bytes = match std::fs::read(&path) {- Ok(b) => b,- Err(e) if e.kind() == std::io::ErrorKind::NotFound => return,- Err(e) => {- // The file is *there* and its bytes are almost certainly fine- // — `EACCES` after a uid change on a container volume, a- // backup tool's chmod, `EIO`, `EMFILE` at boot. Do not copy it- // aside (we cannot read it) and do not touch it: mark the load- // failed so no later write can rename a snapshot of empty maps- // over configuration that is still perfectly recoverable.- self.cluster_state_loaded- .store(false, std::sync::atomic::Ordering::Release);- error!(- path = %path.display(), error = %e,- "could not READ cluster_state.json — index templates, ingest \- pipelines, data streams and ILM policies are NOT restored, \- and management writes will be refused until a boot loads it \- (the file itself is left untouched)"- );- return;- }- };- let state: PersistedClusterState = match serde_json::from_slice(&bytes) {- Ok(s) => s,- Err(e) => {- // Same position as the read error above: the load failed, so- // the live maps are not the operator's configuration and- // nothing may be written over the document on disk. Refusing- // rather than salvaging-then-overwriting also closes the- // second-corruption hole in the `!salvage.exists()` guard- // below — with an overwrite still allowed, a second damaged- // document would be destroyed *and* not copied, because the- // copy from the first one is already there.- self.cluster_state_loaded- .store(false, std::sync::atomic::Ordering::Release);- error!(- path = %path.display(), error = %e,- "cluster_state.json is corrupt — index templates, ingest \- pipelines, data streams and ILM policies are NOT restored, \- and management writes will be refused until a boot loads it"- );- // Keep a copy anyway. The original is now safe from xerj, but- // un-wedging the node means moving it aside by hand, and- // hand-recovering a template out of a damaged document is the- // operator's last option — leave them something to recover- // from that survives that step. The original is deliberately- // left in place so every boot keeps logging the error above- // until someone deals with it, and an existing copy is never- // clobbered: the first corruption is the informative one.- let salvage = path.with_extension("corrupt.json");- if !salvage.exists() {- match std::fs::write(&salvage, &bytes) {- Ok(()) => warn!(- path = %salvage.display(),- "kept a copy of the corrupt cluster_state for recovery"- ),- Err(e) => warn!(- path = %salvage.display(), error = %e,- "could not preserve a copy of the corrupt cluster_state"- ),- }- }- return;- }+ /// Apply the exact format-1 document already read and validated before+ /// index discovery. `None` means either a fresh node or a blocked boot;+ /// the immutable boot status distinguishes those cases.+ fn apply_cluster_state(&self, state: Option<PersistedClusterStateV1>) {+ let Some(state) = state else {+ return;};- if state.version > CLUSTER_STATE_VERSION {- warn!(- found = state.version,- supported = CLUSTER_STATE_VERSION,- "cluster_state.json was written by a newer xerj; loading what \- this build understands (downgrade may drop metadata)"- );- }for (name, tmpl) in state.index_templates {- self.templates.insert(name, tmpl);+ self.templates.insert(name, tmpl.into());}for (name, body) in state.legacy_templates {self.legacy_templates.insert(name, body);@@ -2149,7 +2091,8 @@ impl Engine {// `"status": "GREEN"` without a word in the log is the same quiet lie// this issue is about.let mut reconciled_a_stream = false;- for (name, mut ds) in state.data_streams {+ for (name, persisted) in state.data_streams {+ let mut ds: DataStream = persisted.into();// `rollover_data_stream` creates the new backing index before it// records the new generation, so `kill -9` in that window leaves// `.ds-<name>-00000N` on disk while the document still says N-1.@@ -2262,10 +2205,7 @@ impl Engine {/// the cluster state did not load — then every stream is unknown and/// every backing index would look orphaned.fn warn_orphaned_backing_indices(&self) {- if !self- .cluster_state_loaded- .load(std::sync::atomic::Ordering::Acquire)- {+ if !self.cluster_state_boot.is_writable() {return;}// Snapshot both maps before comparing them: nothing here may hold a@@ -2478,24 +2418,30 @@ impl Engine {Ok(removed)}- /// Add an alias pointing to an index.- pub fn add_alias(&self, alias: &str, index: &str) {+ /// Add an alias pointing to an index. A blocked boot fails before the map+ /// or alias file changes.+ pub fn add_alias(&self, alias: &str, index: &str) -> Result<()> {+ self.ensure_cluster_state_writable()?;let mut entry = self.aliases.entry(alias.to_string()).or_default();if !entry.contains(&index.to_string()) {entry.push(index.to_string());}drop(entry);self.flush_aliases();+ Ok(())}- /// Remove an alias's association with an index.- pub fn remove_alias(&self, alias: &str, index: &str) {+ /// Remove an alias's association with an index. A blocked boot fails+ /// before the map or alias file changes.+ pub fn remove_alias(&self, alias: &str, index: &str) -> Result<()> {+ self.ensure_cluster_state_writable()?;if let Some(mut entry) = self.aliases.get_mut(alias) {entry.retain(|i| i != index);}// Clean up empty alias entries.self.aliases.retain(|_, v| !v.is_empty());self.flush_aliases();+ Ok(())}/// Resolve a name: if it's an alias, return the aliased index names;@@ -2518,6 +2464,7 @@ impl Engine {/// 404 `index_not_found`, which left an operator with no lever but/// stopping the server and removing the directory by hand (issue #206).pub async fn delete_index(&self, name: &str) -> Result<()> {+ self.ensure_cluster_state_writable()?;// Per-index authorization backstop (issue #79) — "destroy the brain"// is the loudest door, so it is checked before the index is removed// from the map. See `crate::index_guard`.@@ -2605,7 +2552,9 @@ impl Engine {let had_lifecycle_state = self.managed_indices.remove(name).is_some()| self.lifecycle_detached.remove(name).is_some();if had_lifecycle_state {- self.persist_managed_indices();+ // `delete_index` checked the compatibility fence before reaching+ // this private cleanup path.+ let _ = self.persist_managed_indices();}}@@ -2684,6 +2633,7 @@ impl Engine {/// new error is returned — the operator gets the *current* reason, not the/// one from boot.pub fn retry_failed_index(&self, name: &str) -> Result<()> {+ self.ensure_cluster_state_writable()?;if !crate::index_guard::visible(name) || !self.failed_indices.contains_key(name) {return Err(EngineError::Common(xerj_common::XerjError::index_not_found(name),@@ -2736,6 +2686,14 @@ impl Engine {/// the concrete name, so pointing an alias at someone else's index cannot/// launder access to it.pub fn get_index(&self, name: &str) -> Result<Arc<Index>> {+ // A blocked boot deliberately did not open any index or replay any+ // WAL. Report that unavailable state instead of turning every real+ // on-disk index into a misleading 404.+ if !self.cluster_state_boot.is_writable() {+ return Err(EngineError::Common(+ xerj_common::XerjError::cluster_state_unavailable(),+ ));+ }// Check if name is an alias — if so, resolve to the first backing index.if let Some(aliased) = self.aliases.get(name) {if let Some(real_name) = aliased.first() {@@ -3018,7 +2976,7 @@ impl Engine {let backing_name = format!(".ds-{}-000001", name);self.create_index(&backing_name, Schema::empty())?;// Alias: writing to the stream name → first backing index.- self.add_alias(name, &backing_name);+ self.add_alias(name, &backing_name)?;let ds = DataStream {name: name.to_string(),backing_indices: vec![backing_name],@@ -3039,7 +2997,13 @@ impl Engine {/// exists (#282, ported from #262's `detach_data_stream_backing_index`)/// — and persist the shrunken list, or a restart would reload the/// deleted generation from `cluster_state.json` as an orphan.- pub(crate) fn detach_data_stream_backing_index(&self, index: &str) {+ pub(crate) fn detach_data_stream_backing_index(&self, index: &str) -> Result<()> {+ // This helper is crate-visible because lifecycle deletion calls it+ // after removing the physical backing index. Keep the compatibility+ // fence here as well: a future caller must not be able to edit the+ // live stream map merely because it bypassed the lifecycle entry+ // point's guard.+ self.ensure_cluster_state_writable()?;let mut changed = false;for mut entry in self.data_streams.iter_mut() {let before = entry.backing_indices.len();@@ -3047,10 +3011,9 @@ impl Engine {changed |= entry.backing_indices.len() != before;}if changed {- if let Err(e) = self.flush_cluster_state() {- warn!(error = %e, "failed to persist data-stream state after lifecycle delete");- }+ self.flush_cluster_state()?;}+ Ok(())}/// Roll over a data stream: create the next backing index and update the alias.@@ -3078,7 +3041,7 @@ impl Engine {drop(entry);self.flush_aliases();} else {- self.add_alias(name, &new_backing);+ self.add_alias(name, &new_backing)?;}if let Some(mut ds) = self.data_streams.get_mut(name) {@@ -3117,7 +3080,7 @@ impl Engine {// `cluster_state.json` still describing the stream while some of its// `.ds-*` directories are already gone. The next boot restores the// stream and warns `restored data stream references backing indices- // that are not present in the data dir` (see `load_cluster_state`),+ // that are not present in the data dir` (see `apply_cluster_state`),// `GET /_data_stream/<name>` still answers, and re-issuing the// DELETE finishes the job.//@@ -3203,6 +3166,11 @@ impl Engine {&self.config}+ /// Immutable boot classification used by readiness and diagnostics.+ pub fn cluster_state_boot_status(&self) -> &ClusterStateBootStatus {+ &self.cluster_state_boot+ }+/// Return the effective embedding identity without exposing configuration/// secrets or local asset paths.pub fn embedding_execution_identity(&self) -> Result<EmbeddingExecutionIdentity> {@@ -3356,6 +3324,7 @@ impl Engine {name: &str,indices: Option<Vec<String>>,) -> Result<Value> {+ self.ensure_cluster_state_writable()?;let snap_dir = std::path::Path::new(repo_path).join(name);let snap_dir = validate_snapshot_path(repo_path,@@ -3454,6 +3423,7 @@ impl Engine {name: &str,indices: Option<Vec<String>>,) -> Result<Vec<String>> {+ self.ensure_cluster_state_writable()?;let snap_dir = std::path::Path::new(repo_path).join(name);let snap_dir = validate_snapshot_path(repo_path,diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rsindex f6ceca711..90c614bb7 100644--- a/engine/crates/xerj-engine/src/lib.rs+++ b/engine/crates/xerj-engine/src/lib.rs@@ -19,6 +19,7 @@pub mod aggs;pub mod audit;pub mod bulk;+mod cluster_state;mod collection_publication;pub mod engine;pub mod governor;@@ -43,6 +44,7 @@ mod bounded_ingest_resources_tests;// ── Re-exports ────────────────────────────────────────────────────────────────+pub use cluster_state::{ClusterStateBlockKind, ClusterStateBootStatus};pub use engine::{EmbeddingExecutionIdentity, Engine, HealthStatus, IndexInfo};pub use index::{detect_log_format, resolve_date_math, resolve_field_alias, EnrichTable, FieldEncodingInfo,diff --git a/engine/crates/xerj-engine/src/lifecycle.rs b/engine/crates/xerj-engine/src/lifecycle.rsindex 163aa013b..128d67be4 100644--- a/engine/crates/xerj-engine/src/lifecycle.rs+++ b/engine/crates/xerj-engine/src/lifecycle.rs@@ -539,9 +539,13 @@ async fn execute_action(format!("refusing lifecycle delete of '{index_name}': {reason}"),)));}+ // Before deleting the index: detach persists cluster metadata, so+ // a blocked boot must refuse before any backing directory or live+ // map changes.+ engine.ensure_cluster_state_writable()?;engine.delete_index(index_name).await?;engine.managed_indices.remove(index_name);- engine.detach_data_stream_backing_index(index_name);+ engine.detach_data_stream_backing_index(index_name)?;Ok(ActionOutcome::IndexDeleted)}LifecycleAction::ReadOnly(_) => {@@ -688,7 +692,9 @@ pub async fn tick(engine: &Engine) {}if any_change {- engine.persist_managed_indices();+ if let Err(error) = engine.persist_managed_indices() {+ tracing::warn!(%error, "lifecycle state was not persisted");+ }}}@@ -723,6 +729,117 @@ pub fn explain_json(index_name: &str, managed: &ManagedIndexState) -> Value {mod tests {use super::*;+ #[tokio::test]+ async fn cluster_state_format1_lifecycle_delete_and_detach_guard_before_side_effects() {+ fn storage_snapshot(root: &std::path::Path) -> Vec<(String, Vec<u8>)> {+ fn walk(+ root: &std::path::Path,+ path: &std::path::Path,+ out: &mut Vec<(String, Vec<u8>)>,+ ) {+ let mut entries: Vec<_> = std::fs::read_dir(path)+ .unwrap()+ .map(|entry| entry.unwrap())+ .collect();+ entries.sort_by_key(|entry| entry.file_name());+ for entry in entries {+ let path = entry.path();+ let relative = path.strip_prefix(root).unwrap().to_string_lossy();+ let kind = entry.file_type().unwrap();+ if kind.is_dir() {+ out.push((format!("d:{relative}"), Vec::new()));+ walk(root, &path, out);+ } else {+ out.push((format!("f:{relative}"), std::fs::read(path).unwrap()));+ }+ }+ }+ let mut out = Vec::new();+ walk(root, root, &mut out);+ out+ }++ let dir = tempfile::tempdir().unwrap();+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ {+ let engine = Engine::new(config.clone()).unwrap();+ engine+ .create_index("logs-1", xerj_common::types::Schema::empty())+ .unwrap();+ }++ let future =+ include_bytes!("../tests/fixtures/cluster_state/v2-format-version-minimal.json");+ let state_path = dir.path().join("cluster_state.json");+ std::fs::write(&state_path, future).unwrap();+ let engine = Engine::new(config).unwrap();+ assert!(matches!(+ engine.get_index("logs-1"),+ Err(EngineError::Common(+ xerj_common::XerjError::ClusterStateUnavailable+ ))+ ));+ engine.data_streams.insert(+ "events".to_owned(),+ crate::engine::DataStream {+ name: "events".to_owned(),+ backing_indices: vec!["logs-1".to_owned(), "write-current".to_owned()],+ timestamp_field: "@timestamp".to_owned(),+ generation: 2,+ },+ );+ let before_files = storage_snapshot(dir.path());+ let before_stream = engine.data_streams.get("events").unwrap().clone();++ let direct = engine.detach_data_stream_backing_index("logs-1");+ assert!(matches!(+ direct,+ Err(EngineError::Common(+ xerj_common::XerjError::ClusterStateUnavailable+ ))+ ));+ assert_eq!(+ engine.data_streams.get("events").unwrap().backing_indices,+ before_stream.backing_indices,+ "the direct detach mutator must fail before editing the live stream"+ );+ assert_eq!(+ storage_snapshot(dir.path()),+ before_files,+ "the direct detach mutator must fail before rewriting any file"+ );++ let result = execute_action(+ &engine,+ "logs-1",+ &LifecycleAction::Delete(EmptyParams {}),+ 1,+ now_ms().saturating_sub(1),+ )+ .await;+ assert!(matches!(+ result,+ Err(EngineError::Common(+ xerj_common::XerjError::ClusterStateUnavailable+ ))+ ));+ assert!(dir.path().join("logs-1").is_dir());+ assert_eq!(+ engine.data_streams.get("events").unwrap().backing_indices,+ before_stream.backing_indices,+ "the detach half must not edit the live stream"+ );+ assert_eq!(+ storage_snapshot(dir.path()),+ before_files,+ "blocked lifecycle delete/detach must not rewrite any file"+ );++ assert_eq!(std::fs::read(state_path).unwrap(), future.as_slice());+ }+#[test]fn parse_duration_handles_all_units() {assert_eq!(parse_duration_ms("500ms").unwrap(), 500);diff --git a/engine/crates/xerj-engine/tests/cluster_state_compatibility.rs b/engine/crates/xerj-engine/tests/cluster_state_compatibility.rsnew file mode 100644index 000000000..ee8c1d6bd--- /dev/null+++ b/engine/crates/xerj-engine/tests/cluster_state_compatibility.rs@@ -0,0 +1,542 @@+use sha2::{Digest, Sha256};+use xerj_common::config::{Config, WalSync};+use xerj_common::types::Schema;+use xerj_common::XerjError;+use xerj_engine::{ClusterStateBlockKind, ClusterStateBootStatus, Engine};++/// Byte-level storage snapshot. Directory and symlink topology is included,+/// and every regular file is represented by its length and SHA-256. This is+/// intentionally stronger than an index-count assertion: WAL replay can rotate+/// one file into another while leaving both the document count and directory+/// count unchanged.+fn recursive_storage_snapshot(root: &std::path::Path) -> Vec<String> {+ fn walk(root: &std::path::Path, path: &std::path::Path, out: &mut Vec<String>) {+ let mut entries: Vec<_> = std::fs::read_dir(path)+ .expect("read snapshot directory")+ .map(|entry| entry.expect("read snapshot entry"))+ .collect();+ entries.sort_by_key(|entry| entry.file_name());+ for entry in entries {+ let entry_path = entry.path();+ let relative = entry_path+ .strip_prefix(root)+ .expect("snapshot path under root")+ .to_string_lossy();+ let kind = entry.file_type().expect("snapshot file type");+ if kind.is_dir() {+ out.push(format!("d:{relative}"));+ walk(root, &entry_path, out);+ } else if kind.is_symlink() {+ out.push(format!(+ "l:{relative}->{}",+ std::fs::read_link(&entry_path)+ .expect("read snapshot symlink")+ .to_string_lossy()+ ));+ } else {+ let bytes = std::fs::read(&entry_path).expect("read snapshot file");+ out.push(format!(+ "f:{relative}:{}:{}",+ bytes.len(),+ hex_digest(&bytes)+ ));+ }+ }+ }++ let mut out = Vec::new();+ walk(root, root, &mut out);+ out+}++fn config_for(path: &std::path::Path) -> Config {+ let mut config = Config::default();+ config.server.data_dir = path.to_string_lossy().into_owned();+ config.storage.wal_sync = WalSync::Async;+ config+}++fn fixture(name: &str) -> Vec<u8> {+ std::fs::read(+ std::path::Path::new(env!("CARGO_MANIFEST_DIR"))+ .join("tests/fixtures/cluster_state")+ .join(name),+ )+ .expect("read fixture")+}++fn write_fixture(path: &std::path::Path, name: &str) -> Vec<u8> {+ let bytes = fixture(name);+ std::fs::write(path.join("cluster_state.json"), &bytes).expect("write fixture");+ bytes+}++fn hex_digest(bytes: &[u8]) -> String {+ Sha256::digest(bytes)+ .iter()+ .map(|byte| format!("{byte:02x}"))+ .collect()+}++#[test]+fn cluster_state_format1_fixture_manifest_is_exact() {+ let manifest: serde_json::Value = serde_json::from_slice(&fixture("MANIFEST.json"))+ .expect("parse cluster-state fixture manifest");+ assert_eq!(manifest["schema"], "xerj.cluster-state-format1-fixtures.v1");+ assert_eq!(+ manifest["format1_writer_commit"],+ "ee29361bccce77de527baac3fe6ed387cd1f6dff"+ );+ assert_eq!(+ manifest["format1_writer_tree"],+ "7227e51db69f9910a355bfd6b30b6a64b52dac87"+ );+ assert_eq!(+ manifest["provenance"]["method"],+ "runtime-capture-through-public-apis"+ );+ assert_eq!(+ manifest["provenance"]["writer_source"],+ "engine/crates/xerj-engine/src/engine.rs"+ );+ assert_eq!(manifest["provenance"]["reproduction"], "REPRODUCE.txt");+ let reproduction = fixture("REPRODUCE.txt");+ assert_eq!(+ hex_digest(&reproduction),+ manifest["provenance"]["reproduction_sha256"]+ .as_str()+ .unwrap()+ );+ assert!(reproduction+ .windows("/_index_template/logs".len())+ .any(|window| window == b"/_index_template/logs"));+ for (name, expected) in manifest["fixtures"]+ .as_object()+ .expect("fixture manifest object")+ {+ let bytes = fixture(name);+ assert_eq!(+ bytes.len() as u64,+ expected["bytes"].as_u64().unwrap(),+ "{name}"+ );+ assert_eq!(+ hex_digest(&bytes),+ expected["sha256"].as_str().unwrap(),+ "{name}"+ );+ }+}++#[tokio::test]+async fn cluster_state_format1_valid_v1_is_stable_and_remains_v1_after_mutation() {+ let dir = tempfile::tempdir().expect("tempdir");+ let original = write_fixture(dir.path(), "v1-ee29361-all-fields.json");++ {+ let engine = Engine::new(config_for(dir.path())).expect("open exact v1");+ assert_eq!(+ engine.cluster_state_boot_status(),+ &ClusterStateBootStatus::LoadedV1+ );+ assert_eq!(engine.templates.len(), 1);+ assert_eq!(engine.legacy_templates.len(), 1);+ assert_eq!(engine.component_templates.len(), 1);+ assert_eq!(engine.pipelines.len(), 1);+ assert_eq!(engine.data_streams.len(), 1);+ assert_eq!(engine.ilm_policies.len(), 1);+ }+ assert_eq!(+ std::fs::read(dir.path().join("cluster_state.json")).unwrap(),+ original,+ "a no-op restart must not normalize valid v1 bytes"+ );++ {+ let engine = Engine::new(config_for(dir.path())).expect("reopen exact v1");+ engine+ .put_component_template(+ "extra".to_owned(),+ serde_json::json!({"template":{"settings":{"refresh_interval":"1s"}}}),+ )+ .expect("intentional supported mutation");+ }+ let written: serde_json::Value =+ serde_json::from_slice(&std::fs::read(dir.path().join("cluster_state.json")).unwrap())+ .unwrap();+ assert_eq!(written["version"], 1);+ assert_eq!(written.as_object().unwrap().len(), 7);+ for field in [+ "index_templates",+ "legacy_templates",+ "component_templates",+ "pipelines",+ "data_streams",+ "ilm_policies",+ ] {+ assert!(written[field].is_object(), "missing v1 map {field}");+ }+ assert_eq!(written["component_templates"].as_object().unwrap().len(), 2);+ assert_eq!(+ written["index_templates"]["logs"],+ serde_json::json!({+ "index_patterns": ["logs-*"],+ "settings": {"number_of_replicas": 0},+ "mappings": {"properties": {"host": {"type": "keyword"}}},+ "priority": 200+ })+ );+ assert_eq!(+ written["legacy_templates"]["legacy-logs"],+ serde_json::json!({+ "index_patterns": ["legacy-*"],+ "settings": {"index.number_of_shards": "1"}+ })+ );+ assert_eq!(+ written["component_templates"]["base-settings"],+ serde_json::json!({"template": {"settings": {"number_of_replicas": 1}}})+ );+ assert_eq!(+ written["component_templates"]["extra"],+ serde_json::json!({"template": {"settings": {"refresh_interval": "1s"}}})+ );+ assert_eq!(+ written["pipelines"]["tagger"],+ serde_json::json!({+ "description": "tag every doc",+ "stages": [{"type": "set", "config": {"field": "tagged", "value": "yes"}}]+ })+ );+ assert_eq!(+ written["data_streams"]["metrics-app"],+ serde_json::json!({+ "name": "metrics-app",+ "backing_indices": [".ds-metrics-app-000001"],+ "timestamp_field": "@timestamp",+ "generation": 1+ })+ );+ assert_eq!(+ written["ilm_policies"]["hot-warm"],+ serde_json::json!({"policy": {"phases": {"hot": {"actions": {}}}}})+ );++ let engine = Engine::new(config_for(dir.path())).expect("reopen mutated v1");+ assert_eq!(engine.templates.len(), 1);+ assert_eq!(engine.legacy_templates.len(), 1);+ assert_eq!(engine.component_templates.len(), 2);+ assert_eq!(engine.pipelines.len(), 1);+ assert_eq!(engine.data_streams.len(), 1);+ assert_eq!(engine.ilm_policies.len(), 1);+ assert_eq!(+ engine.templates.get("logs").unwrap().index_patterns,+ vec!["logs-*".to_owned()]+ );+ assert_eq!(+ engine+ .data_streams+ .get("metrics-app")+ .unwrap()+ .backing_indices,+ vec![".ds-metrics-app-000001".to_owned()]+ );+}++#[tokio::test]+async fn cluster_state_format1_incompatible_fixtures_latch_and_preserve_bytes() {+ let cases = [+ (+ "v1-unknown-top-level.json",+ ClusterStateBlockKind::IncompatibleSchema,+ ),+ (+ "v1-unknown-index-template-field.json",+ ClusterStateBlockKind::IncompatibleSchema,+ ),+ (+ "v1-unknown-data-stream-field.json",+ ClusterStateBlockKind::IncompatibleSchema,+ ),+ (+ "v1-duplicate-version.json",+ ClusterStateBlockKind::DuplicateKey,+ ),+ (+ "v1-duplicate-nested-value-key.json",+ ClusterStateBlockKind::DuplicateKey,+ ),+ (+ "v2-format-version-minimal.json",+ ClusterStateBlockKind::UnsupportedFormat,+ ),+ ];++ for (name, expected_kind) in cases {+ let dir = tempfile::tempdir().expect("tempdir");+ let original = write_fixture(dir.path(), name);+ let original_digest = hex_digest(&original);+ for boot in 1..=2 {+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine still boots");+ assert_eq!(+ engine.cluster_state_boot_status().block_kind(),+ Some(&expected_kind),+ "{name}, boot {boot}"+ );+ let err = engine+ .put_component_template("must-not-land".to_owned(), serde_json::json!({}))+ .expect_err("blocked state must refuse mutation");+ let common: XerjError = err.into();+ assert!(matches!(&common, XerjError::ClusterStateUnavailable));+ drop(engine);++ let after = std::fs::read(dir.path().join("cluster_state.json")).unwrap();+ assert_eq!(after, original, "{name}, boot {boot}");+ assert_eq!(hex_digest(&after), original_digest, "{name}, boot {boot}");+ assert!(+ !dir.path().join("cluster_state.corrupt.json").exists(),+ "unsupported, unknown and duplicate-keyed bytes are not malformed salvage"+ );+ }+ }+}++#[tokio::test]+async fn cluster_state_format1_malformed_bytes_are_untouched_and_never_authorize_write() {+ let dir = tempfile::tempdir().expect("tempdir");+ let malformed = br#"{"version":1,"index_templates":{"cut":true"#;+ std::fs::write(dir.path().join("cluster_state.json"), malformed).unwrap();+ for boot in 1..=2 {+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine still boots");+ assert_eq!(+ engine.cluster_state_boot_status().block_kind(),+ Some(&ClusterStateBlockKind::MalformedJson),+ "boot {boot}"+ );+ assert!(engine+ .put_component_template("must-not-land".to_owned(), serde_json::json!({}))+ .is_err());+ drop(engine);+ assert_eq!(+ std::fs::read(dir.path().join("cluster_state.json")).unwrap(),+ malformed,+ "boot {boot}"+ );+ assert!(+ !dir.path().join("cluster_state.corrupt.json").exists(),+ "a blocked diagnostic boot must not create recovery files"+ );+ }+}++#[cfg(unix)]+#[tokio::test]+async fn cluster_state_format1_read_failure_is_not_mislabeled_or_salvaged() {+ use std::os::unix::fs::symlink;++ let dir = tempfile::tempdir().expect("tempdir");+ let state_path = dir.path().join("cluster_state.json");+ symlink("cluster_state.json", &state_path).expect("create self-loop");+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine still boots");+ assert_eq!(+ engine.cluster_state_boot_status().block_kind(),+ Some(&ClusterStateBlockKind::ReadFailure)+ );+ assert!(engine+ .put_component_template("must-not-land".to_owned(), serde_json::json!({}))+ .is_err());+ drop(engine);+ assert_eq!(+ std::fs::read_link(&state_path).expect("read self-loop"),+ std::path::PathBuf::from("cluster_state.json")+ );+ assert!(+ !dir.path().join("cluster_state.corrupt.json").exists(),+ "read failures are never parse-corruption salvage"+ );+}++#[tokio::test]+async fn cluster_state_format1_future_state_cannot_reconcile_and_downgrade_on_boot() {+ let dir = tempfile::tempdir().expect("tempdir");+ {+ let engine = Engine::new(config_for(dir.path())).expect("fresh engine");+ for name in ["user-events", ".xerj_dashboards", ".ds-events-000002"] {+ engine+ .create_index(name, Schema::empty())+ .unwrap_or_else(|err| panic!("create {name}: {err}"));+ engine+ .get_index(name)+ .expect("new index is open")+ .index_document(+ Some("pending-wal".to_owned()),+ serde_json::json!({"body": format!("unflushed {name}")}),+ )+ .await+ .unwrap_or_else(|err| panic!("append {name} WAL: {err}"));+ }+ }+ let original = write_fixture(dir.path(), "v2-format-version-minimal.json");+ let before = recursive_storage_snapshot(dir.path());++ for boot in 1..=2 {+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine still boots");+ assert_eq!(+ engine.cluster_state_boot_status().block_kind(),+ Some(&ClusterStateBlockKind::UnsupportedFormat)+ );+ assert!(engine.data_streams.is_empty());+ assert!(engine.index_name_list().is_empty(), "boot {boot}");+ let get_err: XerjError = match engine.get_index("user-events") {+ Ok(_) => panic!("blocked boot must not open a user index"),+ Err(err) => err.into(),+ };+ assert!(matches!(get_err, XerjError::ClusterStateUnavailable));+ let create_err: XerjError = engine+ .create_index("must-not-exist", Schema::empty())+ .expect_err("blocked boot must not activate a new WAL")+ .into();+ assert!(matches!(create_err, XerjError::ClusterStateUnavailable));+ engine.audit.append(+ "blocked-diagnostic",+ "test",+ "cluster-state",+ "refused",+ "must remain memory-only",+ );+ drop(engine);++ assert_eq!(+ recursive_storage_snapshot(dir.path()),+ before,+ "blocked boot {boot} must not open, replay, rotate or flush any user/system index, WAL, segment, or durable audit file"+ );+ assert!(!dir.path().join("must-not-exist").exists());+ }+ assert_eq!(+ std::fs::read(dir.path().join("cluster_state.json")).unwrap(),+ original,+ "startup index discovery must not downgrade future cluster state"+ );+}++#[tokio::test]+async fn cluster_state_format1_direct_durable_mutators_fail_before_any_side_effect() {+ fn unavailable<T>(result: xerj_engine::Result<T>, operation: &str) {+ let error = match result {+ Ok(_) => panic!("{operation}"),+ Err(error) => error,+ };+ let common: XerjError = error.into();+ assert!(+ matches!(common, XerjError::ClusterStateUnavailable),+ "{operation}: {common}"+ );+ }++ let dir = tempfile::tempdir().expect("tempdir");+ let future = write_fixture(dir.path(), "v2-format-version-minimal.json");+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine");+ let policy: xerj_engine::lifecycle::LifecyclePolicy =+ serde_json::from_value(serde_json::json!({+ "default_state": "hot",+ "states": [{"name":"hot","actions":[],"transitions":[]}]+ }))+ .unwrap();+ engine.index_mappings.insert(+ "seed".to_owned(),+ serde_json::json!({"properties":{"kept":{"type":"keyword"}}}),+ );+ engine.api_keys.insert(+ "seed-key".to_owned(),+ xerj_engine::engine::ApiKeyRecord::new("seed", "secret", 1, None, Vec::new()),+ );+ engine+ .ism_policies+ .insert("seed-policy".to_owned(), policy.clone());+ engine.managed_indices.insert(+ "seed".to_owned(),+ xerj_engine::lifecycle::ManagedIndexState::new(+ "seed-policy".to_owned(),+ "hot".to_owned(),+ 1,+ ),+ );+ engine.index_settings.insert(+ "seed".to_owned(),+ serde_json::json!({"index":{"lifecycle":{"name":"seed-policy"}}}),+ );+ engine+ .aliases+ .insert("seed-alias".to_owned(), vec!["seed".to_owned()]);++ let before_storage = recursive_storage_snapshot(dir.path());+ unavailable(+ engine.put_index_mapping("seed", serde_json::json!({"replaced":true})),+ "mapping mutation must be fenced",+ );+ unavailable(+ engine.persist_api_key(+ "new-key".to_owned(),+ xerj_engine::engine::ApiKeyRecord::new("new", "secret", 2, None, Vec::new()),+ ),+ "API-key insert must be fenced",+ );+ unavailable(+ engine+ .invalidate_api_keys(&["seed-key".to_owned()], 3)+ .map(|_| ()),+ "API-key invalidation must be fenced",+ );+ unavailable(+ engine.put_ism_policy("new-policy".to_owned(), policy),+ "ISM insert must be fenced",+ );+ unavailable(+ engine.remove_ism_policy("seed-policy").map(|_| ()),+ "ISM remove must be fenced",+ );+ unavailable(+ engine.persist_managed_indices(),+ "managed lifecycle persistence must be fenced",+ );+ unavailable(+ engine.detach_lifecycle("seed").map(|_| ()),+ "lifecycle detach must be fenced",+ );+ unavailable(+ engine.add_alias("new-alias", "seed"),+ "alias insert must be fenced",+ );+ unavailable(+ engine.remove_alias("seed-alias", "seed"),+ "alias remove must be fenced",+ );++ assert_eq!(engine.index_mappings.len(), 1);+ assert_eq!(+ engine.index_mappings.get("seed").unwrap().value(),+ &serde_json::json!({"properties":{"kept":{"type":"keyword"}}})+ );+ assert_eq!(engine.api_keys.len(), 1);+ assert!(!engine.api_keys.get("seed-key").unwrap().invalidated);+ assert_eq!(engine.ism_policies.len(), 1);+ assert!(engine.ism_policies.contains_key("seed-policy"));+ assert_eq!(engine.managed_indices.len(), 1);+ assert!(engine.managed_indices.contains_key("seed"));+ assert!(engine.lifecycle_detached.is_empty());+ assert_eq!(+ engine.index_settings.get("seed").unwrap().value(),+ &serde_json::json!({"index":{"lifecycle":{"name":"seed-policy"}}})+ );+ assert_eq!(engine.aliases.len(), 1);+ assert_eq!(+ engine.aliases.get("seed-alias").unwrap().value(),+ &vec!["seed".to_owned()]+ );+ assert_eq!(recursive_storage_snapshot(dir.path()), before_storage);+ assert_eq!(+ std::fs::read(dir.path().join("cluster_state.json")).unwrap(),+ future+ );+}diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/MANIFEST.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/MANIFEST.jsonnew file mode 100644index 000000000..d0a675e11--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/MANIFEST.json@@ -0,0 +1,23 @@+{+ "schema": "xerj.cluster-state-format1-fixtures.v1",+ "format1_writer_commit": "ee29361bccce77de527baac3fe6ed387cd1f6dff",+ "format1_writer_tree": "7227e51db69f9910a355bfd6b30b6a64b52dac87",+ "provenance": {+ "method": "runtime-capture-through-public-apis",+ "writer_source": "engine/crates/xerj-engine/src/engine.rs",+ "reproduction": "REPRODUCE.txt",+ "reproduction_sha256": "84a08f9fd38809fe25343b4ee503641ea1a3d25b83601066d10f4e494da10022",+ "writer_build": "git worktree add ../xerj-format1-writer ee29361bccce77de527baac3fe6ed387cd1f6dff && cd ../xerj-format1-writer/engine && cargo build --locked --profile profiling -j 32 -p xerj-server",+ "capture_contract": "six documented ES-compatible PUTs, six matching GETs, graceful shutdown, then copy cluster_state.json",+ "note": "The all-fields fixture is byte-identical to captured-cluster-state.json emitted after public PUT and GET of all six managed map types and a clean shutdown."+ },+ "fixtures": {+ "v1-duplicate-nested-value-key.json": {"bytes": 202, "sha256": "e815f8872638de2ce5fce2520dcf9ab38869cb6aceedb5baecbbdcf5837b121c"},+ "v1-duplicate-version.json": {"bytes": 145, "sha256": "6e5e671200fdda04567d4201c9abed40695f33c5f36adab4ee5de41f55093153"},+ "v1-ee29361-all-fields.json": {"bytes": 1275, "sha256": "ae9f96a1dbf70cc711eadfdf8cd2e6701004b66f39c197f1e74cde556d64bb7b"},+ "v1-unknown-data-stream-field.json": {"bytes": 259, "sha256": "dda0121ed37ff4a8cff4ac17280f07eeb932f552c78bacacaa0e4aa129391f0f"},+ "v1-unknown-index-template-field.json": {"bytes": 224, "sha256": "93a057bae38183ddda637f3c6d23a63153f8e9e77ae5387134dca189c0394f7b"},+ "v1-unknown-top-level.json": {"bytes": 155, "sha256": "d0f5b5e976899a6e4d46d648c001a9867e3cbf52f4e74834fe9bd083e09fd2c6"},+ "v2-format-version-minimal.json": {"bytes": 176, "sha256": "28894ecec5edf98baa3988d365013f6ba13b7024c9e4292bc5fe56dc92d0b7d3"}+ }+}diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/REPRODUCE.txt b/engine/crates/xerj-engine/tests/fixtures/cluster_state/REPRODUCE.txtnew file mode 100644index 000000000..1ffcaf241--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/REPRODUCE.txt@@ -0,0 +1,41 @@+Reproduce the format-1 all-fields fixture+=========================================++`v1-ee29361-all-fields.json` is output captured from the released format-1+writer. It is not a hand-authored approximation and does not depend on a+private benchmark directory.++1. Put writer commit `ee29361bccce77de527baac3fe6ed387cd1f6dff` in+ a disposable worktree (do not rewrite the tree being reviewed) and build+ only the server:++ git worktree add ../xerj-format1-writer ee29361bccce77de527baac3fe6ed387cd1f6dff+ cd ../xerj-format1-writer/engine+ CARGO_PROFILE_PROFILING_LTO=thin cargo build --locked --profile profiling -j 32 -p xerj-server++2. Start that binary with an empty data directory, authentication disabled,+ and an unused loopback ES-compatible port. In the commands below,+ `XERJ_URL` is that listener, for example `http://127.0.0.1:19200`.++3. Drive every format-1 map through its public write endpoint:++ curl -fsS -XPUT "$XERJ_URL/_index_template/logs" -H 'content-type: application/json' --data-binary '{"index_patterns":["logs-*"],"template":{"settings":{"number_of_replicas":0},"mappings":{"properties":{"host":{"type":"keyword"}}}},"priority":200}'+ curl -fsS -XPUT "$XERJ_URL/_template/legacy-logs" -H 'content-type: application/json' --data-binary '{"index_patterns":["legacy-*"],"settings":{"index.number_of_shards":"1"}}'+ curl -fsS -XPUT "$XERJ_URL/_component_template/base-settings" -H 'content-type: application/json' --data-binary '{"template":{"settings":{"number_of_replicas":1}}}'+ curl -fsS -XPUT "$XERJ_URL/_ingest/pipeline/tagger" -H 'content-type: application/json' --data-binary '{"description":"tag every doc","stages":[{"type":"set","config":{"field":"tagged","value":"yes"}}]}'+ curl -fsS -XPUT "$XERJ_URL/_data_stream/metrics-app"+ curl -fsS -XPUT "$XERJ_URL/_ilm/policy/hot-warm" -H 'content-type: application/json' --data-binary '{"policy":{"phases":{"hot":{"actions":{}}}}}'++4. Read the same six resources before accepting the capture:++ curl -fsS "$XERJ_URL/_index_template/logs"+ curl -fsS "$XERJ_URL/_template/legacy-logs"+ curl -fsS "$XERJ_URL/_component_template/base-settings"+ curl -fsS "$XERJ_URL/_ingest/pipeline/tagger"+ curl -fsS "$XERJ_URL/_data_stream/metrics-app"+ curl -fsS "$XERJ_URL/_ilm/policy/hot-warm"++5. Stop the server with `SIGTERM`, require exit status 0, and copy+ `<data_dir>/cluster_state.json`. The result is 1,275 bytes with SHA-256+ `ae9f96a1dbf70cc711eadfdf8cd2e6701004b66f39c197f1e74cde556d64bb7b`.+ `MANIFEST.json` pins that digest and the digests of every negative fixture.diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-nested-value-key.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-nested-value-key.jsonnew file mode 100644index 000000000..a1592b159--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-nested-value-key.json@@ -0,0 +1 @@+{"version":1,"index_templates":{},"legacy_templates":{"logs":{"settings":{"refresh_interval":"1s","refresh_interval":"5s"}}},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{}}diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-version.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-version.jsonnew file mode 100644index 000000000..7357e8068--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-version.json@@ -0,0 +1 @@+{"version":1,"version":2,"index_templates":{},"legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{}}diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-ee29361-all-fields.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-ee29361-all-fields.jsonnew file mode 100644index 000000000..0ee98963f--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-ee29361-all-fields.json@@ -0,0 +1,75 @@+{+ "version": 1,+ "index_templates": {+ "logs": {+ "index_patterns": [+ "logs-*"+ ],+ "settings": {+ "number_of_replicas": 0+ },+ "mappings": {+ "properties": {+ "host": {+ "type": "keyword"+ }+ }+ },+ "priority": 200+ }+ },+ "legacy_templates": {+ "legacy-logs": {+ "index_patterns": [+ "legacy-*"+ ],+ "settings": {+ "index.number_of_shards": "1"+ }+ }+ },+ "component_templates": {+ "base-settings": {+ "template": {+ "settings": {+ "number_of_replicas": 1+ }+ }+ }+ },+ "pipelines": {+ "tagger": {+ "description": "tag every doc",+ "stages": [+ {+ "type": "set",+ "config": {+ "field": "tagged",+ "value": "yes"+ }+ }+ ]+ }+ },+ "data_streams": {+ "metrics-app": {+ "name": "metrics-app",+ "backing_indices": [+ ".ds-metrics-app-000001"+ ],+ "timestamp_field": "@timestamp",+ "generation": 1+ }+ },+ "ilm_policies": {+ "hot-warm": {+ "policy": {+ "phases": {+ "hot": {+ "actions": {}+ }+ }+ }+ }+ }+}\ No newline at end of filediff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-data-stream-field.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-data-stream-field.jsonnew file mode 100644index 000000000..631f202ff--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-data-stream-field.json@@ -0,0 +1 @@+{"version":1,"index_templates":{},"legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{"events":{"name":"events","backing_indices":[".ds-events-000001"],"timestamp_field":"@timestamp","generation":1,"future":true}},"ilm_policies":{}}diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-index-template-field.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-index-template-field.jsonnew file mode 100644index 000000000..f38945d4f--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-index-template-field.json@@ -0,0 +1 @@+{"version":1,"index_templates":{"logs":{"index_patterns":["logs-*"],"settings":{},"mappings":{},"priority":0,"future":true}},"legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{}}diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-top-level.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-top-level.jsonnew file mode 100644index 000000000..928628a25--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-top-level.json@@ -0,0 +1 @@+{"version":1,"index_templates":{},"legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{},"future_authority":{}}diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.jsonnew file mode 100644index 000000000..602aa626e--- /dev/null+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json@@ -0,0 +1 @@+{"format_version":2,"data_streams":{"events":{"name":"events","backing_indices":[".ds-events-000001"],"timestamp_field":"@timestamp","generation":1}},"corpus_publications":{}}diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rsindex 208918097..962ab78e6 100644--- a/engine/crates/xerj-engine/tests/integration.rs+++ b/engine/crates/xerj-engine/tests/integration.rs@@ -1716,7 +1716,7 @@ async fn test_alias_search() {.unwrap();// Add alias "my_alias" → "real_index".- engine.add_alias("my_alias", "real_index");+ engine.add_alias("my_alias", "real_index").unwrap();// Search via alias should return the same results as searching via the real name.let idx_via_alias = engine.get_index("my_alias").unwrap();@@ -1734,7 +1734,7 @@ async fn test_alias_search() {assert!(ids.contains(&"a2"));// Remove alias and verify it no longer resolves.- engine.remove_alias("my_alias", "real_index");+ engine.remove_alias("my_alias", "real_index").unwrap();let resolved = engine.resolve_alias("my_alias");assert_eq!(resolved,diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.tomlindex d0a756590..d19165ddb 100644--- a/engine/crates/xerj-server/Cargo.toml+++ b/engine/crates/xerj-server/Cargo.toml@@ -95,3 +95,4 @@ protox = { workspace = true }reqwest = { workspace = true }tempfile = { workspace = true }flate2 = "1"+tokio-stream = "0.1"diff --git a/engine/crates/xerj-server/src/grpc.rs b/engine/crates/xerj-server/src/grpc.rsindex 56b83e582..703b15736 100644--- a/engine/crates/xerj-server/src/grpc.rs+++ b/engine/crates/xerj-server/src/grpc.rs@@ -373,6 +373,17 @@ impl tonic::service::Interceptor for GrpcAuth {));}request.extensions_mut().insert(principal);+ // Authentication deliberately runs first: an unauthenticated caller+ // must not learn whether this node rejected persisted metadata. Once+ // identity is established, reject every RPC centrally before tonic+ // dispatches to a handler. Individual handlers used to translate the+ // same blocked Engine error into not-found, invalid-argument, or a+ // partial bulk response depending on the method.+ if !self.state.engine.cluster_state_boot_status().is_writable() {+ return Err(Status::unavailable(+ xerj_common::XerjError::cluster_state_unavailable().to_string(),+ ));+ }Ok(request)}}@@ -454,6 +465,17 @@ mod tests {panic!("gRPC client could not connect to {addr}");}+ fn with_authorization<T>(+ message: T,+ authorization: &tonic::metadata::MetadataValue<tonic::metadata::Ascii>,+ ) -> tonic::Request<T> {+ let mut request = tonic::Request::new(message);+ request+ .metadata_mut()+ .insert("authorization", authorization.clone());+ request+ }+#[tokio::test]async fn grpc_health_index_get_search_roundtrip() {let dir = TempDir::new().unwrap();@@ -666,6 +688,158 @@ mod tests {let _ = server.await;}+ /// A rejected persisted-metadata format leaves the process available only+ /// for HTTP liveness/readiness diagnostics. The gRPC surface has no split+ /// liveness service, so every generated XerjSearch method is unavailable.+ /// Authentication stays outside that fence: missing credentials remain+ /// UNAUTHENTICATED and a valid credential reaches the UNAVAILABLE latch.+ #[tokio::test]+ async fn grpc_cluster_state_fence_covers_all_methods_after_authentication() {+ use tonic::Code;++ let dir = TempDir::new().unwrap();+ std::fs::write(+ dir.path().join("cluster_state.json"),+ br#"{"format_version":2}"#,+ )+ .unwrap();+ let admin = "grpc-blocked-admin";+ let state = app_state_with_auth(&dir, admin);+ assert!(!state.engine.cluster_state_boot_status().is_writable());++ let port = free_port();+ let addr: SocketAddr = format!("127.0.0.1:{port}").parse().unwrap();+ let (tx, rx) = tokio::sync::oneshot::channel::<()>();+ let server = tokio::spawn(serve_grpc(addr, state, async move {+ let _ = rx.await;+ }));+ let url = format!("http://127.0.0.1:{port}");+ let mut client = connect(&url).await;++ macro_rules! assert_code {+ ($future:expr, $code:expr, $method:literal) => {{+ let status = $future+ .await+ .expect_err(concat!($method, " must be rejected"));+ assert_eq!(status.code(), $code, "{}: {status:?}", $method);+ }};+ }++ // Authentication is evaluated before the storage-format latch for+ // every unary and streaming method.+ assert_code!(+ client.health(pb::HealthRequest {}),+ Code::Unauthenticated,+ "health"+ );+ assert_code!(+ client.search(pb::SearchRequest {+ index: "blocked".into(),+ query_json: String::new(),+ size: 0,+ from: 0,+ }),+ Code::Unauthenticated,+ "search"+ );+ assert_code!(+ client.index(pb::IndexRequest {+ index: "blocked".into(),+ id: "1".into(),+ source_json: "{}".into(),+ }),+ Code::Unauthenticated,+ "index"+ );+ assert_code!(+ client.bulk_index(tokio_stream::empty::<pb::IndexRequest>()),+ Code::Unauthenticated,+ "bulk_index"+ );+ assert_code!(+ client.get_document(pb::GetRequest {+ index: "blocked".into(),+ id: "1".into(),+ }),+ Code::Unauthenticated,+ "get_document"+ );+ assert_code!(+ client.delete_document(pb::DeleteRequest {+ index: "blocked".into(),+ id: "1".into(),+ }),+ Code::Unauthenticated,+ "delete_document"+ );++ let authorization: tonic::metadata::MetadataValue<tonic::metadata::Ascii> =+ format!("ApiKey {admin}").parse().unwrap();+ assert_code!(+ client.health(with_authorization(pb::HealthRequest {}, &authorization)),+ Code::Unavailable,+ "authenticated health"+ );+ assert_code!(+ client.search(with_authorization(+ pb::SearchRequest {+ index: "blocked".into(),+ query_json: String::new(),+ size: 0,+ from: 0,+ },+ &authorization,+ )),+ Code::Unavailable,+ "authenticated search"+ );+ assert_code!(+ client.index(with_authorization(+ pb::IndexRequest {+ index: "blocked".into(),+ id: "1".into(),+ source_json: "{}".into(),+ },+ &authorization,+ )),+ Code::Unavailable,+ "authenticated index"+ );+ let mut bulk = tonic::Request::new(tokio_stream::empty::<pb::IndexRequest>());+ bulk.metadata_mut()+ .insert("authorization", authorization.clone());+ assert_code!(+ client.bulk_index(bulk),+ Code::Unavailable,+ "authenticated bulk_index"+ );+ assert_code!(+ client.get_document(with_authorization(+ pb::GetRequest {+ index: "blocked".into(),+ id: "1".into(),+ },+ &authorization,+ )),+ Code::Unavailable,+ "authenticated get_document"+ );+ assert_code!(+ client.delete_document(with_authorization(+ pb::DeleteRequest {+ index: "blocked".into(),+ id: "1".into(),+ },+ &authorization,+ )),+ Code::Unavailable,+ "authenticated delete_document"+ );++ let _ = tx.send(());+ let _ = server.await;+ }+/// Standard-alphabet base64, so the test can build the `id:secret`/// credential a minted key presents. `xerj_api`'s encoder is crate-private.fn b64(input: &str) -> String {@@ -698,10 +872,13 @@ mod tests {/// Mint a key straight into the engine's store and return the/// `authorization` metadata value a client would present.fn mint_key(state: &AppState, id: &str, roles: Vec<xerj_engine::rbac::Role>) -> String {- state.engine.persist_api_key(- id.to_string(),- xerj_engine::engine::ApiKeyRecord::new(id, "s3cret", 0, None, roles),- );+ state+ .engine+ .persist_api_key(+ id.to_string(),+ xerj_engine::engine::ApiKeyRecord::new(id, "s3cret", 0, None, roles),+ )+ .unwrap();format!("ApiKey {}", b64(&format!("{id}:s3cret")))}diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex d9785adf8..0f13c4e02 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -1866,6 +1866,7 @@ async fn async_main() -> Result<()> {// 8. Engine (opens existing indices from disk)let engine = Engine::new(cfg.clone()).context("initialise engine")?;+ let storage_available = engine.cluster_state_boot_status().is_writable();// This node's own cluster address, composed from the parsed bind IP.// Needed twice — by the transport (8b) and by the console's node identity@@ -1874,7 +1875,7 @@ async fn async_main() -> Result<()> {let cluster_listen_addr = cfg.socket_addr(cfg.cluster.port);// 8b. Cluster runner (if cluster mode is enabled)- let _cluster_shutdown = if cfg.cluster.enabled {+ let _cluster_shutdown = if cfg.cluster.enabled && storage_available {// Fail closed (issue #75). The cluster port carries Raft control// messages; without a shared secret every frame on it is// unauthenticated, so refuse to start rather than open it. This must@@ -1945,6 +1946,11 @@ async fn async_main() -> Result<()> {}Some(shutdown_tx)+ } else if cfg.cluster.enabled {+ warn!(+ "cluster transport is disabled while cluster-state storage is unavailable; restart with supported bytes before rejoining peers"+ );+ None} else {info!("Cluster mode disabled — running in single-node mode");None@@ -1954,6 +1960,13 @@ async fn async_main() -> Result<()> {// boot, persists a 32-byte master key under data_dir/.xerj_master_key// (mode 0600), and prints the first-launch magic-link banner to// stderr if no active user exists yet. Idempotent on reboot.+ //+ // This is storage activation, not cosmetic UI setup. When the engine's+ // cluster-state compatibility fence is blocked, boot must remain live+ // only for diagnostics: opening and seeding Console indices can replay+ // their WAL and flush a new segment. The Console surface is therefore+ // absent until a supported restart instead of mounting with a made-up+ // key or partially-initialized stores.let xerj_console_bind_url = format!("http://{}:{}",if cfg.server.bind_address == "0.0.0.0" || cfg.server.bind_address == "::" {@@ -1966,14 +1979,6 @@ async fn async_main() -> Result<()> {},cfg.server.es_compat_port,);- let xerj_console_outcome = xerj_console_api::bootstrap::run(- &engine,- Path::new(&cfg.server.data_dir),- &xerj_console_bind_url,- )- .await- .context("xerj-console bootstrap")?;-let xerj_console_node_id: String = match (cfg.cluster.enabled, cluster_listen_addr) {// The same string the cluster transport registered as this node's id.(true, Some(addr)) => addr.to_string(),@@ -2005,13 +2010,29 @@ async fn async_main() -> Result<()> {});}- let xerj_console_state = ConsoleState::new(- engine.clone(),- xerj_console_node_id,- xerj_console_outcome.master_key,- xerj_console_cluster_mode,- )- .with_trusted_proxies(trusted_proxies);+ let xerj_console_state = if storage_available {+ let outcome = xerj_console_api::bootstrap::run(+ &engine,+ Path::new(&cfg.server.data_dir),+ &xerj_console_bind_url,+ )+ .await+ .context("xerj-console bootstrap")?;+ Some(+ ConsoleState::new(+ engine.clone(),+ xerj_console_node_id,+ outcome.master_key,+ xerj_console_cluster_mode,+ )+ .with_trusted_proxies(trusted_proxies),+ )+ } else {+ warn!(+ "Xerj Console bootstrap and routes are disabled while cluster-state storage is unavailable"+ );+ None+ };// 9b. Application statelet state = AppState::new(cfg.clone(), engine, metrics);@@ -2025,7 +2046,9 @@ async fn async_main() -> Result<()> {// / wal_size_bytes / memory_usage live (otherwise a flat zero even at// millions of docs, hiding the RSS-runaway and WAL-growth signals).xerj_engine::set_engine_metrics(state.metrics.clone());- tokio::spawn(xerj_api::es_compat::run_metrics_gauge_loop(state.clone()));+ if storage_available {+ tokio::spawn(xerj_api::es_compat::run_metrics_gauge_loop(state.clone()));+ }// 9c. Routers — engine and Xerj Console are *peer* surfaces. Each crate// builds a complete Router (routes + its own auth + its own@@ -2035,9 +2058,13 @@ async fn async_main() -> Result<()> {// /_xerj-console/api/v1/* routes. Yanking xerj-console-api out of this// merge would leave xerj-api compiling and serving on its own// — a property worth preserving.- let xerj_console_router = xerj_console_api::xerj_console_router(xerj_console_state);- let native_router = build_native_router(state.clone()).merge(xerj_console_router.clone());- let es_router = build_es_compat_router(state.clone()).merge(xerj_console_router);+ let mut native_router = build_native_router(state.clone());+ let mut es_router = build_es_compat_router(state.clone());+ if let Some(console_state) = xerj_console_state {+ let xerj_console_router = xerj_console_api::xerj_console_router(console_state);+ native_router = native_router.merge(xerj_console_router.clone());+ es_router = es_router.merge(xerj_console_router);+ }// 10. Banner (includes total startup time)let startup_ms = startup_start.elapsed().as_millis();@@ -2065,24 +2092,30 @@ async fn async_main() -> Result<()> {.with_context(|| bad_bind_address(&cfg))?;// 12. Background flush timer- let flusher = spawn_periodic_flusher(- state.engine.clone(),- std::time::Duration::from_secs(cfg.storage.flush_interval_secs),- );+ let flusher = storage_available.then(|| {+ spawn_periodic_flusher(+ state.engine.clone(),+ std::time::Duration::from_secs(cfg.storage.flush_interval_secs),+ )+ });// 12b. Resource governor sampler (RC4 W3 items 1 & 3): refreshes the// process-wide memtable / RSS / disk-usage atomics that drive the// parent circuit breaker and the disk flood-stage write block. This// is the structural guard against the 112 GiB OOM class — writes get// a 429 circuit_breaking_exception before the kernel OOM-kills us.- state.engine.spawn_resource_sampler();+ if storage_available {+ state.engine.spawn_resource_sampler();+ }// 12c. ISM/ILM lifecycle-management background job: drives every// managed index's state machine (actions, then transitions) on// `lifecycle.tick_interval_secs` (default 5 minutes).- state.engine.spawn_lifecycle_manager();+ if storage_available {+ state.engine.spawn_lifecycle_manager();+ }- let _ingest_memory_trace = ingest_memory_trace::spawn(&state.engine);+ let _ingest_memory_trace = storage_available.then(|| ingest_memory_trace::spawn(&state.engine));// 13. Start servers concurrentlylet rest_tls = tls_config.clone();let rest = tokio::spawn(async move {@@ -2122,7 +2155,9 @@ async fn async_main() -> Result<()> {// boot, flush-permit exhaustion on the last write) sat in memory —// pinning its WAL generations on disk — until shutdown or the next// explicit `_flush`.- flusher.abort();+ if let Some(flusher) = flusher {+ flusher.abort();+ }// 15. Final synchronous flush across every index.//@@ -2135,10 +2170,12 @@ async fn async_main() -> Result<()> {// sessions) loses 100 % of its data on restart because startup index// discovery looks for segment-bearing directories first. See POV// report 2026-04-24T23-58-00 (B-2) for the full failure mode.- info!("flushing in-memory state before exit…");- let flush_started = std::time::Instant::now();- state.engine.flush_all_force().await;- info!("final flush complete in {:.0?}", flush_started.elapsed());+ if storage_available {+ info!("flushing in-memory state before exit…");+ let flush_started = std::time::Instant::now();+ state.engine.flush_all_force().await;+ info!("final flush complete in {:.0?}", flush_started.elapsed());+ }info!("xerj v{} stopped. Goodbye.", env!("CARGO_PKG_VERSION"));Ok(())