Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): fail closed on unsupported cluster state

bugerd8b243023f3c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9a9824dee..70cea4988 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -50,6 +50,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **A binary that does not understand `cluster_state.json` now fails closed
+ before activating storage.** The previous loader accepted future and
+ partially-understood shapes, dropped fields it did not know, and could later
+ rewrite the reduced document over the original. Boot now accepts only the
+ complete shipped format-1 envelope, including duplicate-key and nested-shape
+ checks. A rejected document leaves liveness at 200 and readiness at 503, but
+ opens no user index, Console system index, WAL, segment, or durable audit
+ sink; Console bootstrap and storage background jobs remain disabled until a
+ supported restart. After the existing request-size, authentication, and
+ authorization gates, HTTP storage access returns the stable
+ `cluster_state_unavailable` 503; authenticated gRPC calls return
+ `UNAVAILABLE`. Existing 401/403/body-limit precedence is unchanged. Client
+ responses give the category and recovery action without a local path or
+ persisted object names; the server log carries the detailed classification
+ and path for the operator. The original bytes and any legacy staging file
+ stay in place, and a blocked boot creates no salvage copy. Format-1 rewrites
+ retain the existing
+ same-directory temp-file fsync and atomic rename; parent-directory sync is a
+ best-effort attempt whose errors are ignored, so this change does not claim
+ strict namespace durability across power loss. Data-directory creation,
+ `node.lock` acquisition/PID diagnostics, and the server's earlier
+ credential/TLS preparation retain their existing behavior; the no-open/no-
+ replay claim begins at the cluster-state classification and covers storage
+ stores, not those process-bootstrap files.
+
- **`PUT /{index}/_settings {"index.lifecycle.name": null}` actually detaches
the index, and the detach survives a restart** (#282, ported from #262). The
null previously fell through a string-only settings reader, so the operator
@@ -1009,7 +1034,8 @@ documented "at most one bar per 15 s" floor is really 12.5 s.
over a file whose bytes were perfectly good. The load failure now latches:
every management mutation is refused with a 500 naming the file, and the
file is not touched, until a boot loads it cleanly. The corrupt-parse arm
- refuses too, and still keeps a `cluster_state.corrupt.json` copy.
+ refuses too. The later format-compatibility fence above tightens that
+ diagnostic boot further: it creates no `cluster_state.corrupt.json` copy.
- `DELETE /_data_stream/<name>` recorded the removal before destroying the
backing indices, so a crash in that window stranded `.ds-<name>-00000N`
directories that no data-stream API could reach — GET and DELETE answered
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index 42844677e..793c18337 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -6442,6 +6442,7 @@ dependencies = [
"prometheus",
"serde",
"serde_json",
+ "sha2 0.10.9",
"tempfile",
"thiserror 2.0.18",
"tokio",
@@ -6619,6 +6620,7 @@ dependencies = [
"thiserror 2.0.18",
"tokio",
"tracing",
+ "tracing-subscriber",
"uuid",
"xerj-ai",
"xerj-cluster",
@@ -6730,6 +6732,7 @@ dependencies = [
"tikv-jemalloc-ctl",
"tikv-jemallocator",
"tokio",
+ "tokio-stream",
"tonic",
"tonic-build",
"tracing",
diff --git a/engine/crates/xerj-api/Cargo.toml b/engine/crates/xerj-api/Cargo.toml
index 3d06c3ea3..8ea6bcdf0 100644
--- a/engine/crates/xerj-api/Cargo.toml
+++ b/engine/crates/xerj-api/Cargo.toml
@@ -44,3 +44,4 @@ libc = "0.2"
tempfile = { workspace = true }
tower = { workspace = true, features = ["util"] }
http-body-util = "0.1"
+sha2 = "0.10"
diff --git a/engine/crates/xerj-api/src/error.rs b/engine/crates/xerj-api/src/error.rs
index 29fa420a0..00f8d8dd3 100644
--- a/engine/crates/xerj-api/src/error.rs
+++ b/engine/crates/xerj-api/src/error.rs
@@ -302,6 +302,7 @@ fn xerj_error_type(e: &XerjError) -> String {
XerjError::InvalidDocumentJson { .. } => "json_parse_exception",
// ── Configuration ─────────────────────────────────────────────────
XerjError::ConfigError { .. } => "action_request_validation_exception",
+ XerjError::ClusterStateUnavailable => "cluster_state_unavailable",
// ── Auth ──────────────────────────────────────────────────────────
XerjError::AuthError { .. } => "security_exception",
// ── TLS ───────────────────────────────────────────────────────────
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 7da2f1e96..a62918234 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -1116,7 +1116,9 @@ pub async fn create_index(
if !mappings_val.is_null() {
// Persists es_mapping.json into the index dir (atomic) so
// the create-time mapping survives a restart.
- state.engine.put_index_mapping(&index, mappings_val.clone());
+ if let Err(error) = state.engine.put_index_mapping(&index, mappings_val.clone()) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
if let Some(aliases) = body.get("aliases").and_then(Value::as_object) {
// Alias keys can also contain date math
@@ -1127,7 +1129,9 @@ pub async fn create_index(
for (alias, opts) in aliases {
let resolved = resolve_date_math_index(alias);
resolved_aliases.insert(resolved.clone(), opts.clone());
- state.engine.add_alias(&resolved, &index);
+ if let Err(error) = state.engine.add_alias(&resolved, &index) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
state
.engine
@@ -2314,7 +2318,9 @@ pub async fn put_mapping(
for (idx_name, _, existing, _) in plans {
// Persists es_mapping.json into the index dir (atomic) so the
// merged mapping survives a restart.
- state.engine.put_index_mapping(&idx_name, existing);
+ if let Err(error) = state.engine.put_index_mapping(&idx_name, existing) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
Json(json!({ "acknowledged": true })).into_response()
@@ -17292,12 +17298,16 @@ pub async fn post_aliases(
match step {
Resolved::Add(add, targets) => {
for target in targets {
- state.engine.add_alias(&add.alias, target);
+ if let Err(error) = state.engine.add_alias(&add.alias, target) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
}
Resolved::Remove(remove, targets) => {
for target in targets {
- state.engine.remove_alias(&remove.alias, target);
+ if let Err(error) = state.engine.remove_alias(&remove.alias, target) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
}
}
@@ -17465,8 +17475,8 @@ pub async fn put_alias(
.filter(|v| v.as_object().map(|o| !o.is_empty()).unwrap_or(false))
.unwrap_or(json!({}));
- let attach = |idx_name: &str| {
- state.engine.add_alias(&alias, idx_name);
+ let attach = |idx_name: &str| -> xerj_engine::Result<()> {
+ state.engine.add_alias(&alias, idx_name)?;
if !alias_meta.as_object().map(|o| o.is_empty()).unwrap_or(true) {
let mut existing = state
.engine
@@ -17484,6 +17494,7 @@ pub async fn put_alias(
.index_alias_metadata
.insert(idx_name.to_string(), existing);
}
+ Ok(())
};
if targets.is_empty() {
@@ -17499,7 +17510,9 @@ pub async fn put_alias(
return ApiError::new(e).into_response();
}
for idx in &targets {
- attach(idx);
+ if let Err(error) = attach(idx) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
Json(json!({ "acknowledged": true })).into_response()
}
@@ -17510,10 +17523,14 @@ pub async fn delete_alias(
) -> impl IntoResponse {
let targets = resolve_index_selector(&state, &index).await;
if targets.is_empty() {
- state.engine.remove_alias(&alias, &index);
+ if let Err(error) = state.engine.remove_alias(&alias, &index) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
} else {
for idx in &targets {
- state.engine.remove_alias(&alias, idx);
+ if let Err(error) = state.engine.remove_alias(&alias, idx) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
}
Json(json!({ "acknowledged": true })).into_response()
@@ -22400,7 +22417,9 @@ pub async fn put_settings(
// tombstone for a ghost is unbounded state writable from
// the public port (#262 hit exactly this in review).
if state.engine.get_index(idx).is_ok() {
- state.engine.detach_lifecycle(idx);
+ if let Err(error) = state.engine.detach_lifecycle(idx) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
}
}
@@ -24408,7 +24427,9 @@ pub async fn put_ilm_policy(
if let Err(e) = state.engine.put_ilm_policy(name.clone(), body.clone()) {
return ApiError::new(xerj_common::XerjError::from(e)).into_response();
}
- state.engine.put_ism_policy(name, policy);
+ if let Err(error) = state.engine.put_ism_policy(name, policy) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
Json(json!({ "acknowledged": true })).into_response()
}
@@ -24472,7 +24493,9 @@ pub async fn delete_ilm_policy(
// lifecycle engine's storage rather than to this handler.
match state.engine.delete_ilm_policy(&name) {
Ok(true) => {
- state.engine.remove_ism_policy(&name);
+ if let Err(error) = state.engine.remove_ism_policy(&name) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
Json(json!({ "acknowledged": true })).into_response()
}
Ok(false) => {
@@ -24612,7 +24635,9 @@ pub async fn remove_ilm_policy_from_index(
Err(e) => return ApiError::new(e).into_response(),
};
for name in &targets {
- state.engine.detach_lifecycle(name);
+ if let Err(error) = state.engine.detach_lifecycle(name) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
}
Json(json!({ "has_failures": false, "failed_indexes": [] })).into_response()
}
@@ -28110,7 +28135,7 @@ pub async fn security_create_api_key(
// across restarts via `persist_api_key` (item 6: `<data_dir>/api_keys.json`).
let now_ms = Utc::now().timestamp_millis().max(0) as u64;
let expiration_ms = parse_api_key_expiration_ms(expiration.as_str(), now_ms);
- state.engine.persist_api_key(
+ if let Err(error) = state.engine.persist_api_key(
key_id.clone(),
xerj_engine::engine::ApiKeyRecord::new(
name.clone(),
@@ -28119,7 +28144,9 @@ pub async fn security_create_api_key(
expiration_ms,
roles,
),
- );
+ ) {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
state.engine.audit.append(
"security.api_key.create",
principal.label(),
@@ -28511,7 +28538,12 @@ pub async fn security_invalidate_api_key(
};
let now_ms = Utc::now().timestamp_millis().max(0) as u64;
- let (invalidated, previously) = state.engine.invalidate_api_keys(&target_ids, now_ms);
+ let (invalidated, previously) = match state.engine.invalidate_api_keys(&target_ids, now_ms) {
+ Ok(result) => result,
+ Err(error) => {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
+ };
state.engine.audit.append(
"security.api_key.invalidate",
principal.label(),
diff --git a/engine/crates/xerj-api/src/ism_api.rs b/engine/crates/xerj-api/src/ism_api.rs
index bf8aa38e5..31235b46b 100644
--- a/engine/crates/xerj-api/src/ism_api.rs
+++ b/engine/crates/xerj-api/src/ism_api.rs
@@ -85,9 +85,12 @@ pub async fn put_ism_policy(
let err = xerj_common::XerjError::invalid_query(format!("invalid ISM policy: {reason}"));
return ApiError::new(err).into_response();
}
- state
+ if let Err(error) = state
.engine
- .put_ism_policy(policy_id.clone(), policy.clone());
+ .put_ism_policy(policy_id.clone(), policy.clone())
+ {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
Json(json!({
"_id": policy_id,
"_version": 1,
@@ -178,11 +181,14 @@ pub async fn delete_ism_policy(
State(state): State<AppState>,
Path(policy_id): Path<String>,
) -> impl IntoResponse {
- if state.engine.remove_ism_policy(&policy_id) {
- Json(json!({ "_id": policy_id, "result": "deleted" })).into_response()
- } else {
- let e = xerj_common::XerjError::index_not_found(format!("policy [{policy_id}] not found"));
- ApiError::new(e).into_response()
+ match state.engine.remove_ism_policy(&policy_id) {
+ Ok(true) => Json(json!({ "_id": policy_id, "result": "deleted" })).into_response(),
+ Ok(false) => {
+ let e =
+ xerj_common::XerjError::index_not_found(format!("policy [{policy_id}] not found"));
+ ApiError::new(e).into_response()
+ }
+ Err(error) => ApiError::new(xerj_common::XerjError::from(error)).into_response(),
}
}
@@ -210,6 +216,9 @@ pub async fn add_ism_policy(
/// `es_compat` (a single index↔policy association model, per the design —
/// not two).
pub async fn attach_policy(state: &AppState, index: &str, policy_id: &str) -> Response {
+ if let Err(error) = state.engine.ensure_cluster_state_writable() {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
if state.engine.get_index(index).is_err() {
let e = xerj_common::XerjError::index_not_found(index);
return ApiError::new(e).into_response();
@@ -235,7 +244,9 @@ pub async fn attach_policy(state: &AppState, index: &str, policy_id: &str) -> Re
.engine
.managed_indices
.insert(index.to_string(), managed);
- state.engine.persist_managed_indices();
+ if let Err(error) = state.engine.persist_managed_indices() {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
Json(json!({
"failures": false,
@@ -258,12 +269,8 @@ pub async fn attach_policy(state: &AppState, index: &str, policy_id: &str) -> Re
/// unambiguously "not managed", so a plain removal is the whole fix.
/// Returns whether the index was actually managed (false is not an error —
/// detaching an already-unmanaged index is idempotent, matching real ES).
-pub fn detach_ilm_policy(state: &AppState, index: &str) -> bool {
- let removed = state.engine.managed_indices.remove(index).is_some();
- if removed {
- state.engine.persist_managed_indices();
- }
- removed
+pub fn detach_ilm_policy(state: &AppState, index: &str) -> xerj_engine::Result<bool> {
+ state.engine.detach_lifecycle(index)
}
// ─────────────────────────────────────────────────────────────────────────────
@@ -278,8 +285,10 @@ pub async fn remove_ism_policy(
// One shared detach path (engine method) for all three detach
// routes, so the acknowledged detach is tombstoned and persisted
// identically whichever surface the operator used (#282).
- state.engine.detach_lifecycle(&index);
- Json(single_index_success()).into_response()
+ match state.engine.detach_lifecycle(&index) {
+ Ok(_) => Json(single_index_success()).into_response(),
+ Err(error) => ApiError::new(xerj_common::XerjError::from(error)).into_response(),
+ }
} else {
Json(not_managed_failure(
&index,
@@ -320,6 +329,9 @@ pub async fn change_ism_policy(
Path(index): Path<String>,
Json(body): Json<ChangePolicyBody>,
) -> impl IntoResponse {
+ if let Err(error) = state.engine.ensure_cluster_state_writable() {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
let Some(mut managed) = state
.engine
.managed_indices
@@ -383,7 +395,9 @@ pub async fn change_ism_policy(
managed.info_message = format!("policy changed to '{}'", body.policy_id);
managed.last_updated_ms = lifecycle::now_ms();
state.engine.managed_indices.insert(index, managed);
- state.engine.persist_managed_indices();
+ if let Err(error) = state.engine.persist_managed_indices() {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
Json(single_index_success()).into_response()
}
@@ -412,6 +426,9 @@ pub async fn retry_ism_index(
Path(index): Path<String>,
body: OptionalJson<RetryBody>,
) -> impl IntoResponse {
+ if let Err(error) = state.engine.ensure_cluster_state_writable() {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
let body = body.0.unwrap_or_default();
let Some(mut managed) = state
.engine
@@ -462,7 +479,9 @@ pub async fn retry_ism_index(
managed.info_message = "retrying".to_string();
managed.last_updated_ms = lifecycle::now_ms();
state.engine.managed_indices.insert(index, managed);
- state.engine.persist_managed_indices();
+ if let Err(error) = state.engine.persist_managed_indices() {
+ return ApiError::new(xerj_common::XerjError::from(error)).into_response();
+ }
Json(single_index_success()).into_response()
}
diff --git a/engine/crates/xerj-api/src/native.rs b/engine/crates/xerj-api/src/native.rs
index 3aa0f1534..1721049cf 100644
--- a/engine/crates/xerj-api/src/native.rs
+++ b/engine/crates/xerj-api/src/native.rs
@@ -731,6 +731,13 @@ pub async fn liveness() -> impl IntoResponse {
}
pub async fn readiness(State(state): State<AppState>) -> impl IntoResponse {
+ if !state.engine.cluster_state_boot_status().is_writable() {
+ return (
+ axum::http::StatusCode::SERVICE_UNAVAILABLE,
+ "not ready: persisted cluster metadata was not accepted at boot; inspect server logs",
+ )
+ .into_response();
+ }
let h = state.engine.health().await;
// Read the node's own state, not a principal's filtered view: this path is
// exempt from authentication so a kubelet can probe a hardened node, and
diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rs
index 2d170650e..4ddf827c1 100644
--- a/engine/crates/xerj-api/src/router.rs
+++ b/engine/crates/xerj-api/src/router.rs
@@ -12,7 +12,7 @@ use axum::{
extract::{DefaultBodyLimit, Request, State},
http::{HeaderValue, Method},
middleware::{self, Next},
- response::Response,
+ response::{IntoResponse, Response},
routing::{delete, get, post, put},
Router,
};
@@ -173,6 +173,10 @@ pub fn build_native_router(state: AppState) -> Router {
// spelling (`/v1/indices/{name}/…`), so it carries the same
// authorization layer — otherwise the reserved `.xerj-memory-*`
// namespace would be closed on :9200 and open on :8080.
+ .layer(middleware::from_fn_with_state(
+ state.clone(),
+ storage_fence_middleware,
+ ))
.layer(middleware::from_fn_with_state(
state.clone(),
authz::authz_middleware,
@@ -886,6 +890,10 @@ pub fn build_es_compat_router(state: AppState) -> Router {
// rather than a naming convention (issue #79); see `authz.rs` for the
// full table of doors it closes. A superuser — `--insecure`, or the
// configured admin key — short-circuits it on the first line.
+ .layer(middleware::from_fn_with_state(
+ state.clone(),
+ storage_fence_middleware,
+ ))
.layer(middleware::from_fn_with_state(
state.clone(),
authz::authz_middleware,
@@ -916,6 +924,29 @@ pub fn build_es_compat_router(state: AppState) -> Router {
// Middleware helpers
// ─────────────────────────────────────────────────────────────────────────────
+/// Admit only health probes while the boot-time persistence contract is
+/// blocked. The engine deliberately opened no index in this state, so letting
+/// an arbitrary handler continue would either lie from empty maps or find a
+/// filesystem-shaped bypass (snapshot restore was the important example) and
+/// activate storage after the preflight had refused it.
+async fn storage_fence_middleware(
+ State(state): State<AppState>,
+ req: Request,
+ next: Next,
+) -> Response {
+ if state.engine.cluster_state_boot_status().is_writable()
+ || matches!(
+ req.uri().path(),
+ "/health/live" | "/health/ready" | "/v1/health/ready"
+ )
+ {
+ return next.run(req).await;
+ }
+
+ crate::error::ApiError::from(xerj_common::XerjError::cluster_state_unavailable())
+ .into_response()
+}
+
/// Middleware that sets ES-compatible product headers on every response.
///
/// Kibana and other Elastic clients verify the presence of `X-Elastic-Product`
diff --git a/engine/crates/xerj-api/src/state.rs b/engine/crates/xerj-api/src/state.rs
index 34d7534bb..93a9ded0c 100644
--- a/engine/crates/xerj-api/src/state.rs
+++ b/engine/crates/xerj-api/src/state.rs
@@ -494,8 +494,17 @@ impl AppState {
"issuer": "xerj"
})));
let watcher_active = Arc::new(AtomicBool::new(true));
- let ml_detectors = Arc::new(MlDetector::load_all(&config.server.data_dir));
- let ml_datafeeds = Arc::new(MlDatafeed::load_all(&config.server.data_dir));
+ // A blocked cluster-state boot is a diagnostic shell. Do not even
+ // read auxiliary persisted registries: they are storage state whose
+ // ownership may be described by the unsupported cluster metadata.
+ let (ml_detectors, ml_datafeeds) = if engine.cluster_state_boot_status().is_writable() {
+ (
+ Arc::new(MlDetector::load_all(&config.server.data_dir)),
+ Arc::new(MlDatafeed::load_all(&config.server.data_dir)),
+ )
+ } else {
+ (Arc::new(DashMap::new()), Arc::new(DashMap::new()))
+ };
Self {
config: Arc::new(config),
engine: Arc::new(engine),
diff --git a/engine/crates/xerj-api/tests/cluster_metadata_survives_restart.rs b/engine/crates/xerj-api/tests/cluster_metadata_survives_restart.rs
index d2bd3c85f..fab93f55b 100644
--- a/engine/crates/xerj-api/tests/cluster_metadata_survives_restart.rs
+++ b/engine/crates/xerj-api/tests/cluster_metadata_survives_restart.rs
@@ -14,6 +14,7 @@
use axum::body::Body;
use axum::http::{Request, StatusCode};
use serde_json::{json, Value};
+use sha2::{Digest, Sha256};
use tower::ServiceExt;
fn config_for(dir: &std::path::Path) -> xerj_common::config::Config {
@@ -27,11 +28,162 @@ fn config_for(dir: &std::path::Path) -> xerj_common::config::Config {
/// dropped in between) is a restart: nothing is handed over in memory, the
/// second boot sees only what reached the disk.
fn app_over(dir: &std::path::Path) -> axum::Router {
+ app_and_engine_over(dir).0
+}
+
+fn app_and_engine_over(dir: &std::path::Path) -> (axum::Router, xerj_engine::Engine) {
+ let (es_app, _native_app, engine) = apps_and_engine_over(dir);
+ (es_app, engine)
+}
+
+fn apps_and_engine_over(
+ dir: &std::path::Path,
+) -> (axum::Router, axum::Router, xerj_engine::Engine) {
let config = config_for(dir);
let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
- let state = xerj_api::state::AppState::new(config, engine, metrics);
- xerj_api::router::build_es_compat_router(state)
+ let state = xerj_api::state::AppState::new(config, engine.clone(), metrics);
+ (
+ xerj_api::router::build_es_compat_router(state.clone()),
+ xerj_api::router::build_native_router(state),
+ engine,
+ )
+}
+
+fn hex_digest(bytes: &[u8]) -> String {
+ Sha256::digest(bytes)
+ .iter()
+ .map(|byte| format!("{byte:02x}"))
+ .collect()
+}
+
+fn recursive_inventory(root: &std::path::Path) -> Vec<String> {
+ fn walk(root: &std::path::Path, path: &std::path::Path, out: &mut Vec<String>) {
+ let mut entries: Vec<_> = std::fs::read_dir(path)
+ .expect("read inventory directory")
+ .map(|entry| entry.expect("read inventory entry"))
+ .collect();
+ entries.sort_by_key(|entry| entry.file_name());
+ for entry in entries {
+ let entry_path = entry.path();
+ let relative = entry_path
+ .strip_prefix(root)
+ .expect("inventory entry under root")
+ .to_string_lossy()
+ .into_owned();
+ let kind = entry.file_type().expect("inventory file type");
+ out.push(format!(
+ "{}:{relative}",
+ if kind.is_dir() {
+ "d"
+ } else if kind.is_symlink() {
+ "l"
+ } else {
+ "f"
+ }
+ ));
+ if kind.is_dir() {
+ walk(root, &entry_path, out);
+ }
+ }
+ }
+
+ let mut out = Vec::new();
+ walk(root, root, &mut out);
+ out
+}
+
+fn recursive_storage_snapshot(root: &std::path::Path) -> Vec<String> {
+ fn walk(root: &std::path::Path, path: &std::path::Path, out: &mut Vec<String>) {
+ let mut entries: Vec<_> = std::fs::read_dir(path)
+ .expect("read storage snapshot directory")
+ .map(|entry| entry.expect("read storage snapshot entry"))
+ .collect();
+ entries.sort_by_key(|entry| entry.file_name());
+ for entry in entries {
+ let entry_path = entry.path();
+ let relative = entry_path
+ .strip_prefix(root)
+ .expect("storage snapshot entry under root")
+ .to_string_lossy();
+ let kind = entry.file_type().expect("storage snapshot file type");
+ if kind.is_dir() {
+ out.push(format!("d:{relative}"));
+ walk(root, &entry_path, out);
+ } else if kind.is_symlink() {
+ out.push(format!(
+ "l:{relative}->{}",
+ std::fs::read_link(&entry_path)
+ .expect("read storage snapshot symlink")
+ .to_string_lossy()
+ ));
+ } else {
+ let bytes = std::fs::read(&entry_path).expect("read storage snapshot file");
+ out.push(format!(
+ "f:{relative}:{}:{}",
+ bytes.len(),
+ hex_digest(&bytes)
+ ));
+ }
+ }
+ }
+
+ let mut out = Vec::new();
+ walk(root, root, &mut out);
+ out
+}
+
+#[derive(Debug, PartialEq, Eq)]
+struct ManagedInventory {
+ indices: Vec<String>,
+ aliases: Vec<(String, Vec<String>)>,
+ index_templates: Vec<String>,
+ legacy_templates: Vec<String>,
+ component_templates: Vec<String>,
+ pipelines: Vec<String>,
+ data_streams: Vec<String>,
+ ilm_policies: Vec<String>,
+ directory_entries: Vec<String>,
+}
+
+async fn managed_inventory(
+ engine: &xerj_engine::Engine,
+ dir: &std::path::Path,
+) -> ManagedInventory {
+ fn keys<V>(map: &dashmap::DashMap<String, V>) -> Vec<String> {
+ let mut keys: Vec<_> = map.iter().map(|entry| entry.key().clone()).collect();
+ keys.sort();
+ keys
+ }
+
+ let mut indices: Vec<_> = engine
+ .list_indices()
+ .await
+ .into_iter()
+ .map(|index| index.name)
+ .collect();
+ indices.sort();
+ let mut aliases: Vec<_> = engine
+ .aliases
+ .iter()
+ .map(|entry| {
+ let mut targets = entry.value().clone();
+ targets.sort();
+ (entry.key().clone(), targets)
+ })
+ .collect();
+ aliases.sort();
+ ManagedInventory {
+ indices,
+ aliases,
+ index_templates: keys(&engine.templates),
+ legacy_templates: keys(&engine.legacy_templates),
+ component_templates: keys(&engine.component_templates),
+ pipelines: keys(&engine.pipelines),
+ data_streams: keys(&engine.data_streams),
+ ilm_policies: keys(&engine.ilm_policies),
+ directory_entries: recursive_inventory(dir),
+ }
}
async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {
@@ -65,6 +217,22 @@ async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {
.await
}
+async fn get_text(app: &axum::Router, path: &str) -> (StatusCode, String) {
+ let response = app
+ .clone()
+ .oneshot(Request::get(path).body(Body::empty()).expect("request"))
+ .await
+ .expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("body");
+ (
+ status,
+ String::from_utf8(bytes.to_vec()).expect("utf8 body"),
+ )
+}
+
async fn delete(app: &axum::Router, path: &str) -> (StatusCode, Value) {
send(
app,
@@ -342,8 +510,8 @@ async fn a_rollover_interrupted_before_its_flush_does_not_wedge_the_stream() {
/// first `PUT /_index_template/...` rewrote the file — taking whatever was
/// still legible in it along. Hand-recovery out of a damaged document is the
/// operator's last option; it should not be silently removed. The write is
-/// refused (the original stays where it is) *and* a copy is kept, because
-/// un-wedging the node means moving the original aside by hand.
+/// refused and the original stays where it is. A blocked diagnostic boot must
+/// not create a second recovery file either.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn a_corrupt_cluster_state_is_preserved_before_it_is_overwritten() {
let dir = tempfile::tempdir().expect("tempdir");
@@ -359,14 +527,30 @@ async fn a_corrupt_cluster_state_is_preserved_before_it_is_overwritten() {
let original = std::fs::read_to_string(&state_path).expect("read state");
let truncated = &original[..original.len() / 2];
std::fs::write(&state_path, truncated).expect("write truncated");
+ let truncated_digest = hex_digest(truncated.as_bytes());
- let app = app_over(dir.path());
- let (st, _) = get(&app, "/_index_template/logs").await;
+ let (app, native_app, engine) = apps_and_engine_over(dir.path());
+ let (st, _) = get(&app, "/health/live").await;
assert_eq!(
st,
- StatusCode::NOT_FOUND,
- "a corrupt document cannot be restored — this is the honest outcome"
+ StatusCode::OK,
+ "malformed state blocks readiness, not liveness"
+ );
+ for (surface, path) in [(&app, "/health/ready"), (&native_app, "/v1/health/ready")] {
+ let (st, body) = get_text(surface, path).await;
+ assert_eq!(st, StatusCode::SERVICE_UNAVAILABLE, "{path}");
+ assert_eq!(
+ body,
+ "not ready: persisted cluster metadata was not accepted at boot; inspect server logs"
+ );
+ }
+ let (st, body) = get(&app, "/_index_template/logs").await;
+ assert_eq!(
+ st,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "a corrupt document cannot be restored, but an empty-map 404 would be a lie: {body}"
);
+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");
// The write that would have destroyed the evidence.
let (st, body) = put_json(
@@ -377,31 +561,31 @@ async fn a_corrupt_cluster_state_is_preserved_before_it_is_overwritten() {
.await;
assert_eq!(
st,
- StatusCode::INTERNAL_SERVER_ERROR,
+ StatusCode::SERVICE_UNAVAILABLE,
"a management write on top of a state that did not load must be \
refused, not acknowledged: {body}"
);
- let salvage = dir.path().join("cluster_state.corrupt.json");
assert!(
- salvage.is_file(),
- "the corrupt document must be kept for recovery"
+ !dir.path().join("cluster_state.corrupt.json").exists(),
+ "a blocked diagnostic boot must not create recovery files"
);
assert_eq!(
- std::fs::read_to_string(&salvage).expect("read salvage"),
+ std::fs::read_to_string(&state_path).expect("read state"),
truncated,
- "the preserved copy must be the bytes that were found, unmodified"
+ "the live document must be left exactly as it was found"
);
assert_eq!(
- std::fs::read_to_string(&state_path).expect("read state"),
- truncated,
- "the live document must be left exactly as it was found — the copy is \
- a convenience, not a licence to overwrite the original"
+ hex_digest(&std::fs::read(&state_path).expect("hash live malformed state")),
+ truncated_digest,
+ "the live malformed bytes must keep the exact SHA-256"
);
// Moving the damaged file aside is the documented recovery, and it must
// actually work: the next boot loads cleanly and writes are accepted again.
drop(app);
+ drop(native_app);
+ drop(engine);
std::fs::remove_file(&state_path).expect("operator moves the file aside");
let app = app_over(dir.path());
let (st, body) = put_json(
@@ -491,15 +675,26 @@ async fn an_unreadable_cluster_state_is_never_overwritten() {
"the failing target must be a symlink, not a directory"
);
- let app = app_over(dir.path());
+ let (app, native_app, engine) = apps_and_engine_over(dir.path());
- let (st, _) = get(&app, "/_index_template/logs").await;
+ let (st, _) = get(&app, "/health/live").await;
+ assert_eq!(st, StatusCode::OK, "the blocked node remains live");
+ for (surface, path) in [(&app, "/health/ready"), (&native_app, "/v1/health/ready")] {
+ let (st, _) = get(surface, path).await;
+ assert_eq!(
+ st,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "{path} must expose the boot-time cluster-state fence"
+ );
+ }
+
+ let (st, body) = get(&app, "/_index_template/logs").await;
assert_eq!(
st,
- StatusCode::NOT_FOUND,
- "a document that could not be read cannot be restored — that part is \
- honest already"
+ StatusCode::SERVICE_UNAVAILABLE,
+ "the diagnostic shell must not turn unloaded durable objects into a false 404: {body}"
);
+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");
// The write that used to destroy it.
let (st, body) = put_json(
@@ -510,17 +705,22 @@ async fn an_unreadable_cluster_state_is_never_overwritten() {
.await;
assert_eq!(
st,
- StatusCode::INTERNAL_SERVER_ERROR,
+ StatusCode::SERVICE_UNAVAILABLE,
"a management write must be refused while the persisted state is \
unloaded, not acknowledged: {body}"
);
- assert!(
- body["error"]["reason"]
- .as_str()
- .unwrap_or_default()
- .contains("could not be loaded at boot"),
- "the 500 must say why, so the operator can act on it: {body}"
- );
+ assert_eq!(body["error"]["type"], "cluster_state_unavailable", "{body}");
+ let reason = body["error"]["reason"].as_str().unwrap_or_default();
+ for required in [
+ "start a compatible XERJ build",
+ "restore supported cluster_state.json bytes",
+ "then restart",
+ ] {
+ assert!(
+ reason.contains(required),
+ "the refusal must give both bounded recovery actions: {body}"
+ );
+ }
assert_eq!(
std::fs::read_link(&state_path).expect("read cluster-state self-loop"),
@@ -547,27 +747,36 @@ async fn an_unreadable_cluster_state_is_never_overwritten() {
"a refused write must not even stage a uniquely named temp file"
);
- // And the refusal is not a one-off: the in-memory map must not be left
- // holding a change that was reported as failed.
- let (st, _) = get(&app, "/_index_template/fresh").await;
+ // And the refusal is not a one-off. Reads remain unavailable too: the
+ // node did not open storage, so an empty-map 404 would be a lie.
+ let (st, body) = get(&app, "/_index_template/fresh").await;
assert_eq!(
st,
- StatusCode::NOT_FOUND,
- "a refused PUT must roll its in-memory change back"
+ StatusCode::SERVICE_UNAVAILABLE,
+ "blocked reads must stay unavailable: {body}"
);
+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");
let (st, _) = delete(&app, "/_ilm/policy/hot-warm").await;
assert_eq!(
st,
- StatusCode::INTERNAL_SERVER_ERROR,
+ StatusCode::SERVICE_UNAVAILABLE,
"deletes are writes too — they must be refused as well"
);
// Recovery path: remove the bad directory entry, put the saved bytes back,
// and restart. The latch belongs to the failed engine lifetime only.
drop(app);
+ drop(native_app);
+ drop(engine);
std::fs::remove_file(&state_path).expect("remove cluster-state self-loop");
std::fs::rename(&saved_path, &state_path).expect("restore valid cluster state");
let app = app_over(dir.path());
+ let (st, _) = get(&app, "/health/ready").await;
+ assert_eq!(
+ st,
+ StatusCode::OK,
+ "valid replacement plus restart clears the fence"
+ );
let (st, body) = get(&app, "/_index_template/logs").await;
assert_eq!(
st,
@@ -588,6 +797,465 @@ async fn an_unreadable_cluster_state_is_never_overwritten() {
);
}
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn cluster_state_format1_fence_refuses_every_current_metadata_mutation() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let future = include_bytes!(
+ "../../xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json"
+ );
+ let state_path = dir.path().join("cluster_state.json");
+ std::fs::write(&state_path, future).expect("write future state");
+
+ // A complete-looking snapshot source proves the fence fires before the
+ // restore handler can copy it into a new index and open its WAL. Keeping
+ // the source under data_dir satisfies the normal repository allowlist.
+ let snapshot_repo = dir.path().join("snapshot-repo");
+ let snapshot = snapshot_repo.join("ready");
+ let snapshot_index = snapshot.join("restored-index");
+ std::fs::create_dir_all(snapshot_index.join("wal/s1")).expect("create snapshot fixture");
+ std::fs::write(
+ snapshot.join("manifest.json"),
+ br#"{"indices":["restored-index"]}"#,
+ )
+ .expect("write snapshot manifest");
+ std::fs::write(
+ snapshot_index.join("wal/s1/0000000000000001.wal"),
+ b"sentinel",
+ )
+ .expect("write snapshot WAL sentinel");
+
+ let (app, native_app, engine) = apps_and_engine_over(dir.path());
+ let before = managed_inventory(&engine, dir.path()).await;
+ let before_storage = recursive_storage_snapshot(dir.path());
+ assert!(before.indices.is_empty());
+ assert!(before.aliases.is_empty());
+ assert!(before.index_templates.is_empty());
+ assert!(before.legacy_templates.is_empty());
+ assert!(before.component_templates.is_empty());
+ assert!(before.pipelines.is_empty());
+ assert!(before.data_streams.is_empty());
+ assert!(before.ilm_policies.is_empty());
+
+ for (surface, path) in [(&app, "/health/ready"), (&native_app, "/v1/health/ready")] {
+ let (status, body) = get_text(surface, path).await;
+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{path}");
+ assert_eq!(
+ body,
+ "not ready: persisted cluster metadata was not accepted at boot; inspect server logs",
+ "the unauthenticated probe must not expose paths, object names, or state contents"
+ );
+ }
+ let (status, _) = get(&app, "/health/live").await;
+ assert_eq!(status, StatusCode::OK);
+
+ // The fence is admission-wide, not a hand-maintained list of metadata
+ // handlers. These routes cover read, user-index creation, direct ingest,
+ // and the native surface; none may return an ordinary success/404 shape
+ // from the deliberately empty diagnostic engine.
+ let (status, response) = get(&app, "/").await;
+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{response}");
+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");
+ let (status, response) = put_json(&app, "/blocked-index", json!({"mappings":{}})).await;
+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{response}");
+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");
+ for (surface, path, body) in [
+ (&app, "/_bulk", json!({"must":"not parse"})),
+ (
+ &native_app,
+ "/v1/indices",
+ json!({"name":"blocked-native","schema":{"fields":[]}}),
+ ),
+ ] {
+ let (status, response) = post(surface, path, body).await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "{path}: {response}"
+ );
+ assert_eq!(
+ response["error"]["type"], "cluster_state_unavailable",
+ "{path}: {response}"
+ );
+ }
+
+ let put_cases = [
+ ("/_index_template/new", json!({"index_patterns":["new-*"]})),
+ ("/_template/new", json!({"index_patterns":["legacy-*"]})),
+ ("/_component_template/new", json!({"template":{}})),
+ (
+ "/_ilm/policy/new",
+ json!({"policy":{"phases":{"hot":{"actions":{}}}}}),
+ ),
+ ("/_ingest/pipeline/new", json!({"processors":[]})),
+ ("/_data_stream/new", json!({})),
+ ];
+ for (path, body) in put_cases {
+ let (status, response) = put_json(&app, path, body).await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "PUT {path}: {response}"
+ );
+ assert_eq!(
+ response["error"]["type"], "cluster_state_unavailable",
+ "PUT {path}: {response}"
+ );
+ }
+
+ let delete_cases = [
+ "/_index_template/missing",
+ "/_template/missing",
+ "/_component_template/missing",
+ "/_ilm/policy/missing",
+ "/_ingest/pipeline/missing",
+ "/_data_stream/missing",
+ ];
+ for path in delete_cases {
+ let (status, response) = delete(&app, path).await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "DELETE {path}: {response}"
+ );
+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");
+ }
+
+ let (status, response) = post(&app, "/missing/_rollover", json!({})).await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "rollover must guard before stream lookup: {response}"
+ );
+
+ let (status, response) = put_json(
+ &app,
+ "/_snapshot/blocked-repo",
+ json!({"type":"fs","settings":{"location":snapshot_repo}}),
+ )
+ .await;
+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{response}");
+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");
+ let (status, response) = post(&app, "/_snapshot/blocked-repo/ready/_restore", json!({})).await;
+ assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{response}");
+ assert_eq!(response["error"]["type"], "cluster_state_unavailable");
+
+ let restore_err: xerj_common::XerjError = engine
+ .restore_snapshot(
+ snapshot_repo.to_str().expect("UTF-8 snapshot path"),
+ "ready",
+ None,
+ )
+ .await
+ .expect_err("direct restore must share the storage fence")
+ .into();
+ assert!(matches!(
+ restore_err,
+ xerj_common::XerjError::ClusterStateUnavailable
+ ));
+
+ assert_eq!(
+ std::fs::read(&state_path).expect("read future state"),
+ future,
+ "readiness and refused mutations must preserve future bytes"
+ );
+ assert_eq!(
+ managed_inventory(&engine, dir.path()).await,
+ before,
+ "every refused mutation must preserve the exact alias, index, map, and directory inventories"
+ );
+ assert_eq!(
+ recursive_storage_snapshot(dir.path()),
+ before_storage,
+ "refused HTTP and direct restore paths must not copy, open, replay, or rewrite storage"
+ );
+ assert!(
+ std::fs::read_dir(dir.path())
+ .unwrap()
+ .filter_map(|entry| entry.ok())
+ .all(|entry| !entry.file_name().to_string_lossy().starts_with(".ds-new-")),
+ "refused data-stream creation must not create a backing directory"
+ );
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn blocked_app_state_skips_ml_registries_and_http_preserves_auth_order() {
+ const ADMIN: &str = "blocked-admin";
+ let dir = tempfile::tempdir().expect("tempdir");
+ std::fs::write(
+ dir.path().join("cluster_state.json"),
+ br#"{"format_version":2}"#,
+ )
+ .expect("write future cluster state");
+ let ml_dir = dir.path().join("_ml");
+ std::fs::create_dir_all(&ml_dir).expect("create ML registry dir");
+ let detectors = br#"[{"detector_id":"seeded-detector","source_index":"logs","time_field":"@timestamp","function":"count","bucket_span":"1h","anomaly_threshold":3.0,"create_time_ms":1}]"#;
+ let datafeeds = br#"[{"datafeed_id":"seeded-feed","job_id":"seeded-detector","indices":["logs"],"query":{"match_all":{}},"frequency_secs":60,"state":"started"}]"#;
+ std::fs::write(ml_dir.join("detectors.json"), detectors).expect("seed detectors");
+ std::fs::write(ml_dir.join("datafeeds.json"), datafeeds).expect("seed datafeeds");
+
+ let mut config = config_for(dir.path());
+ config.auth.enabled = true;
+ config.auth.admin_api_key = ADMIN.to_owned();
+ config.limits.max_body_bytes = 16;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("blocked engine");
+ assert!(!engine.cluster_state_boot_status().is_writable());
+ // Seed one non-superuser credential without using the deliberately fenced
+ // persistence API. It can authenticate but has no reserved-index grant.
+ engine.api_keys.insert(
+ "limited".to_owned(),
+ xerj_engine::engine::ApiKeyRecord::new("limited", "secret", 1, None, Vec::new()),
+ );
+ let before_state = recursive_storage_snapshot(dir.path());
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+ assert!(
+ state.ml_detectors.is_empty() && state.ml_datafeeds.is_empty(),
+ "a blocked AppState must not load valid persisted ML entries"
+ );
+ assert_eq!(
+ recursive_storage_snapshot(dir.path()),
+ before_state,
+ "constructing a blocked AppState must not rewrite any seeded ML byte"
+ );
+
+ let native_app = xerj_api::router::build_native_router(state.clone());
+ let app = xerj_api::router::build_es_compat_router(state);
+ let response = app
+ .clone()
+ .oneshot(
+ Request::post("/_bulk")
+ .header("authorization", format!("ApiKey {ADMIN}"))
+ .header("content-type", "application/x-ndjson")
+ // A real HTTP transport supplies Content-Length for this
+ // fixed body. Set it explicitly because Router::oneshot does
+ // not synthesize transport headers; without it the limit
+ // layer can discover overflow only when an inner handler
+ // reads the streaming body, and the storage fence correctly
+ // prevents that handler from running.
+ .header("content-length", "17")
+ .body(Body::from("x".repeat(17)))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(
+ response.status(),
+ StatusCode::PAYLOAD_TOO_LARGE,
+ "the existing request-size gate must keep precedence over the storage fence"
+ );
+ let response = app
+ .clone()
+ .oneshot(Request::get("/").body(Body::empty()).unwrap())
+ .await
+ .unwrap();
+ assert_eq!(
+ response.status(),
+ StatusCode::UNAUTHORIZED,
+ "authentication must run before the storage fence"
+ );
+ let response = native_app
+ .clone()
+ .oneshot(Request::get("/v1/indices").body(Body::empty()).unwrap())
+ .await
+ .unwrap();
+ assert_eq!(
+ response.status(),
+ StatusCode::UNAUTHORIZED,
+ "native authentication must run before the storage fence"
+ );
+
+ let response = app
+ .clone()
+ .oneshot(
+ Request::get("/_memory/bob")
+ .header("authorization", "ApiKey bGltaXRlZDpzZWNyZXQ=")
+ .body(Body::empty())
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(
+ response.status(),
+ StatusCode::FORBIDDEN,
+ "authorization must run before the storage fence"
+ );
+ let response = native_app
+ .clone()
+ .oneshot(
+ Request::post("/v1/indices/.xerj-memory-bob-edges/search")
+ .header("authorization", "ApiKey bGltaXRlZDpzZWNyZXQ=")
+ .header("content-type", "application/json")
+ .body(Body::from("{}"))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(
+ response.status(),
+ StatusCode::FORBIDDEN,
+ "native authorization must run before the storage fence"
+ );
+
+ let response = app
+ .clone()
+ .oneshot(
+ Request::get("/")
+ .header("authorization", format!("ApiKey {ADMIN}"))
+ .body(Body::empty())
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(
+ response.status(),
+ StatusCode::SERVICE_UNAVAILABLE,
+ "an authenticated, authorized request must reach the storage fence"
+ );
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .unwrap();
+ let body: Value = serde_json::from_slice(&bytes).unwrap();
+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");
+ let response = native_app
+ .clone()
+ .oneshot(
+ Request::get("/v1/indices")
+ .header("authorization", format!("ApiKey {ADMIN}"))
+ .body(Body::empty())
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(
+ response.status(),
+ StatusCode::SERVICE_UNAVAILABLE,
+ "authorized native storage access must reach the storage fence"
+ );
+ assert_eq!(
+ recursive_storage_snapshot(dir.path()),
+ before_state,
+ "auth and fence responses must not touch seeded ML registries"
+ );
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn every_incompatible_format1_fixture_is_fenced_across_two_full_boots() {
+ let fixtures = [
+ "v1-unknown-top-level.json",
+ "v1-unknown-index-template-field.json",
+ "v1-unknown-data-stream-field.json",
+ "v1-duplicate-version.json",
+ "v1-duplicate-nested-value-key.json",
+ "v2-format-version-minimal.json",
+ ];
+
+ for fixture in fixtures {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let fixture_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
+ .join("../xerj-engine/tests/fixtures/cluster_state")
+ .join(fixture);
+ let original = std::fs::read(&fixture_path).expect("read fixture");
+ let original_digest = hex_digest(&original);
+ let state_path = dir.path().join("cluster_state.json");
+ std::fs::write(&state_path, &original).expect("install fixture");
+ let mut first_boot_inventory = None;
+
+ for boot in 1..=2 {
+ let (app, native_app, engine) = apps_and_engine_over(dir.path());
+ let before = managed_inventory(&engine, dir.path()).await;
+ assert!(before.indices.is_empty(), "{fixture}, boot {boot}");
+ assert!(before.aliases.is_empty(), "{fixture}, boot {boot}");
+ assert!(before.index_templates.is_empty(), "{fixture}, boot {boot}");
+ assert!(before.legacy_templates.is_empty(), "{fixture}, boot {boot}");
+ assert!(
+ before.component_templates.is_empty(),
+ "{fixture}, boot {boot}"
+ );
+ assert!(before.pipelines.is_empty(), "{fixture}, boot {boot}");
+ assert!(before.data_streams.is_empty(), "{fixture}, boot {boot}");
+ assert!(before.ilm_policies.is_empty(), "{fixture}, boot {boot}");
+ if let Some(first) = &first_boot_inventory {
+ assert_eq!(
+ &before.directory_entries, first,
+ "{fixture}: a refused restart must not add a directory entry"
+ );
+ } else {
+ first_boot_inventory = Some(before.directory_entries.clone());
+ }
+
+ let (status, _) = get(&app, "/health/live").await;
+ assert_eq!(status, StatusCode::OK, "{fixture}, boot {boot}");
+ for (surface, path) in [(&app, "/health/ready"), (&native_app, "/v1/health/ready")] {
+ let (status, body) = get_text(surface, path).await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "{fixture}, boot {boot}, {path}"
+ );
+ assert_eq!(
+ body,
+ "not ready: persisted cluster metadata was not accepted at boot; inspect server logs",
+ "{fixture}, boot {boot}, {path}"
+ );
+ }
+
+ let (status, body) = put_json(
+ &app,
+ "/_index_template/must-not-land",
+ json!({"index_patterns": ["blocked-*"]}),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "{fixture}, boot {boot}, PUT: {body}"
+ );
+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");
+
+ let (status, body) = delete(&app, "/_ilm/policy/must-not-land").await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "{fixture}, boot {boot}, DELETE: {body}"
+ );
+ assert_eq!(body["error"]["type"], "cluster_state_unavailable");
+
+ assert_eq!(
+ managed_inventory(&engine, dir.path()).await,
+ before,
+ "{fixture}, boot {boot}: blocked mutations changed an alias, index, map, or directory"
+ );
+ let after = std::fs::read(&state_path).expect("read state after refused writes");
+ assert_eq!(after, original, "{fixture}, boot {boot}");
+ assert_eq!(
+ hex_digest(&after),
+ original_digest,
+ "{fixture}, boot {boot}"
+ );
+ assert!(
+ !dir.path().join("cluster_state.corrupt.json").exists(),
+ "{fixture}: unsupported, unknown, and duplicate-key bytes are never salvage"
+ );
+
+ drop(app);
+ drop(native_app);
+ drop(engine);
+ assert_eq!(
+ recursive_inventory(dir.path()),
+ first_boot_inventory.clone().unwrap(),
+ "{fixture}, boot {boot}: shutdown changed the persisted inventory"
+ );
+ let after_shutdown = std::fs::read(&state_path).expect("read after shutdown");
+ assert_eq!(after_shutdown, original, "{fixture}, boot {boot}");
+ assert_eq!(
+ hex_digest(&after_shutdown),
+ original_digest,
+ "{fixture}, boot {boot}"
+ );
+ }
+ }
+}
+
/// Concurrent writers must not trip over each other.
///
/// The atomic rewrite stages through one fixed temp path, so two flushes
diff --git a/engine/crates/xerj-common/src/error.rs b/engine/crates/xerj-common/src/error.rs
index aecb43bd8..4f051bb13 100644
--- a/engine/crates/xerj-common/src/error.rs
+++ b/engine/crates/xerj-common/src/error.rs
@@ -78,6 +78,14 @@ pub enum XerjError {
#[error("config error: {reason}")]
ConfigError { reason: String },
+ /// Durable cluster metadata was not accepted during the one boot-time
+ /// format check. The process remains live for health diagnosis, but it is
+ /// not ready and must not activate or mutate storage.
+ #[error(
+ "cluster state is unavailable: start a compatible XERJ build or restore supported cluster_state.json bytes, then restart"
+ )]
+ ClusterStateUnavailable,
+
// ── Auth ───────────────────────────────────────────────────────────────
/// Authentication or authorization failure.
#[error("auth error: {reason}")]
@@ -215,6 +223,10 @@ impl XerjError {
}
}
+ pub fn cluster_state_unavailable() -> Self {
+ Self::ClusterStateUnavailable
+ }
+
pub fn auth(reason: impl Into<String>) -> Self {
Self::AuthError {
reason: reason.into(),
@@ -289,7 +301,7 @@ impl XerjError {
Self::IndexBlocked { .. } => 403,
// The index exists but has no serving copy — ES answers the same
// shape (no_shard_available_action_exception) with 503.
- Self::IndexUnavailable { .. } => 503,
+ Self::IndexUnavailable { .. } | Self::ClusterStateUnavailable => 503,
Self::AuthError { .. } => 401,
Self::ResourceExhausted { .. } | Self::CircuitBreaking { .. } => 429,
Self::StorageError { .. }
diff --git a/engine/crates/xerj-console-api/src/bootstrap.rs b/engine/crates/xerj-console-api/src/bootstrap.rs
index ab632efd5..736cf6eed 100644
--- a/engine/crates/xerj-console-api/src/bootstrap.rs
+++ b/engine/crates/xerj-console-api/src/bootstrap.rs
@@ -48,6 +48,16 @@ pub async fn run(
data_dir: &Path,
bind_url: &str,
) -> ConsoleResult<BootstrapOutcome> {
+ // Console bootstrap is a storage migration: it creates fourteen system
+ // indices, writes a master key and publishes seeded dashboard segments.
+ // Keep this guard inside the crate as well as at the server call site so a
+ // future embedder cannot accidentally bypass the compatibility fence.
+ if !engine.cluster_state_boot_status().is_writable() {
+ return Err(ConsoleApiError::Internal(
+ "cluster-state storage is unavailable; Console bootstrap requires a supported restart"
+ .to_owned(),
+ ));
+ }
let master_key = load_or_init_master_key(data_dir)?;
indices::ensure_all(engine)?;
diff --git a/engine/crates/xerj-console-api/tests/cluster_state_fence.rs b/engine/crates/xerj-console-api/tests/cluster_state_fence.rs
new file mode 100644
index 000000000..bef3a6655
--- /dev/null
+++ b/engine/crates/xerj-console-api/tests/cluster_state_fence.rs
@@ -0,0 +1,82 @@
+use xerj_common::config::{Config, WalSync};
+use xerj_console_api::bootstrap;
+use xerj_engine::{ClusterStateBlockKind, Engine};
+
+fn config_for(path: &std::path::Path) -> Config {
+ let mut config = Config::default();
+ config.server.data_dir = path.to_string_lossy().into_owned();
+ config.storage.wal_sync = WalSync::Async;
+ config
+}
+
+fn snapshot(root: &std::path::Path) -> Vec<(String, Vec<u8>)> {
+ fn walk(root: &std::path::Path, path: &std::path::Path, out: &mut Vec<(String, Vec<u8>)>) {
+ let mut entries: Vec<_> = std::fs::read_dir(path)
+ .expect("read snapshot directory")
+ .map(|entry| entry.expect("read snapshot entry"))
+ .collect();
+ entries.sort_by_key(|entry| entry.file_name());
+ for entry in entries {
+ let entry_path = entry.path();
+ let relative = entry_path
+ .strip_prefix(root)
+ .expect("snapshot path under root")
+ .to_string_lossy()
+ .into_owned();
+ let kind = entry.file_type().expect("snapshot file type");
+ if kind.is_dir() {
+ out.push((format!("d:{relative}"), Vec::new()));
+ walk(root, &entry_path, out);
+ } else if kind.is_symlink() {
+ out.push((
+ format!("l:{relative}"),
+ std::fs::read_link(&entry_path)
+ .expect("read snapshot symlink")
+ .to_string_lossy()
+ .as_bytes()
+ .to_vec(),
+ ));
+ } else {
+ out.push((
+ format!("f:{relative}"),
+ std::fs::read(&entry_path).expect("read snapshot file"),
+ ));
+ }
+ }
+ }
+
+ let mut out = Vec::new();
+ walk(root, root, &mut out);
+ out
+}
+
+#[tokio::test]
+async fn console_bootstrap_is_side_effect_free_when_cluster_state_is_blocked() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let future = include_bytes!(
+ "../../xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json"
+ );
+ std::fs::write(dir.path().join("cluster_state.json"), future).expect("write future state");
+
+ let engine = Engine::new(config_for(dir.path())).expect("blocked diagnostic engine");
+ assert_eq!(
+ engine.cluster_state_boot_status().block_kind(),
+ Some(&ClusterStateBlockKind::UnsupportedFormat)
+ );
+ let before = snapshot(dir.path());
+
+ let result = bootstrap::run(&engine, dir.path(), "http://127.0.0.1:9200").await;
+ assert!(
+ result.is_err(),
+ "Console bootstrap must refuse a blocked storage node"
+ );
+
+ assert_eq!(snapshot(dir.path()), before);
+ assert!(!dir.path().join(".xerj_master_key").exists());
+ for system_index in xerj_console_api::indices::ALL {
+ assert!(
+ !dir.path().join(system_index).exists(),
+ "must not create or open {system_index}"
+ );
+ }
+}
diff --git a/engine/crates/xerj-engine/Cargo.toml b/engine/crates/xerj-engine/Cargo.toml
index 6a6bbe93b..dbe9b3020 100644
--- a/engine/crates/xerj-engine/Cargo.toml
+++ b/engine/crates/xerj-engine/Cargo.toml
@@ -59,6 +59,7 @@ xerj-storage = { path = "../xerj-storage", features = ["test-hooks"] }
tempfile = { workspace = true }
tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
criterion = { workspace = true }
+tracing-subscriber = { workspace = true }
[[bench]]
name = "search_bench"
diff --git a/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs b/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs
index 17e56ad11..f134843b5 100644
--- a/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs
+++ b/engine/crates/xerj-engine/src/bounded_ingest_resources_tests.rs
@@ -51,10 +51,12 @@ fn engine_with_index(name: &str, semantic: bool) -> (TempDir, Engine) {
}
engine.create_index(name, schema).expect("create index");
if semantic {
- engine.put_index_mapping(
- name,
- json!({"properties": {"content": {"type": "semantic_text"}}}),
- );
+ engine
+ .put_index_mapping(
+ name,
+ json!({"properties": {"content": {"type": "semantic_text"}}}),
+ )
+ .unwrap();
}
(dir, engine)
}
diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rs
index 9e9181d35..e3540e16d 100644
--- a/engine/crates/xerj-engine/src/bulk.rs
+++ b/engine/crates/xerj-engine/src/bulk.rs
@@ -2547,7 +2547,9 @@ mod semantic_lifecycle_tests {
.iter()
.map(|(field, _)| ((*field).to_string(), json!({"type": "semantic_text"})))
.collect::<serde_json::Map<_, _>>();
- engine.put_index_mapping(name, json!({"properties": properties}));
+ engine
+ .put_index_mapping(name, json!({"properties": properties}))
+ .unwrap();
(dir, engine)
}
@@ -3123,10 +3125,12 @@ mod semantic_lifecycle_tests {
});
schema.fields.push(content);
engine.create_index(fails, schema).unwrap();
- engine.put_index_mapping(
- fails,
- json!({"properties": {"content": {"type": "semantic_text"}}}),
- );
+ engine
+ .put_index_mapping(
+ fails,
+ json!({"properties": {"content": {"type": "semantic_text"}}}),
+ )
+ .unwrap();
let ledger: &'static Ledger = Box::leak(Box::new(Ledger::new()));
let engine = Arc::new(engine);
let hook_engine = Arc::clone(&engine);
diff --git a/engine/crates/xerj-engine/src/cluster_state.rs b/engine/crates/xerj-engine/src/cluster_state.rs
new file mode 100644
index 000000000..f6bf27565
--- /dev/null
+++ b/engine/crates/xerj-engine/src/cluster_state.rs
@@ -0,0 +1,700 @@
+//! Closed persistence contract for `cluster_state.json` format 1.
+//!
+//! This module is intentionally persistence-only. Public request DTOs stay
+//! forward-compatible at the API boundary; bytes that can later be rewritten
+//! as durable cluster metadata are accepted only when their complete format-1
+//! schema is understood.
+
+use std::collections::{BTreeMap, HashSet};
+use std::path::Path;
+
+use serde::de::{Error as _, MapAccess, SeqAccess, Visitor};
+use serde::{Deserialize, Deserializer, Serialize};
+use serde_json::{Map, Number, Value};
+use tracing::{error, warn};
+
+use crate::engine::{DataStream, IndexTemplate};
+
+pub const CLUSTER_STATE_V1: u32 = 1;
+
+const DUPLICATE_KEY_MARKER: &str = "duplicate JSON object key";
+
+/// Stable reason why this process may not rewrite cluster metadata.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ClusterStateBlockKind {
+ ReadFailure,
+ MalformedJson,
+ DuplicateKey,
+ MissingDiscriminator,
+ AmbiguousDiscriminator,
+ InvalidDiscriminator,
+ UnsupportedFormat,
+ IncompatibleSchema,
+}
+
+/// Immutable result of the one boot-time classification.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ClusterStateBootStatus {
+ Absent,
+ LoadedV1,
+ Blocked {
+ kind: ClusterStateBlockKind,
+ detail: String,
+ },
+}
+
+impl ClusterStateBootStatus {
+ pub fn is_writable(&self) -> bool {
+ matches!(self, Self::Absent | Self::LoadedV1)
+ }
+
+ pub fn block_kind(&self) -> Option<&ClusterStateBlockKind> {
+ match self {
+ Self::Blocked { kind, .. } => Some(kind),
+ Self::Absent | Self::LoadedV1 => None,
+ }
+ }
+
+ pub(crate) fn detail(&self) -> Option<&str> {
+ match self {
+ Self::Blocked { detail, .. } => Some(detail),
+ Self::Absent | Self::LoadedV1 => None,
+ }
+ }
+}
+
+#[derive(Debug)]
+pub(crate) struct PreparedClusterState {
+ pub(crate) status: ClusterStateBootStatus,
+ pub(crate) document: Option<PersistedClusterStateV1>,
+}
+
+impl PreparedClusterState {
+ fn blocked(path: &Path, kind: ClusterStateBlockKind, detail: impl Into<String>) -> Self {
+ let detail = detail.into();
+ error!(
+ path = %path.display(),
+ kind = ?kind,
+ detail = %detail,
+ "cluster state was rejected at boot; start a compatible XERJ build or restore supported bytes, then restart"
+ );
+ Self {
+ status: ClusterStateBootStatus::Blocked { kind, detail },
+ document: None,
+ }
+ }
+}
+
+/// Exact seven-field envelope emitted by the only shipped format-1 writer.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub(crate) struct PersistedClusterStateV1 {
+ pub(crate) version: u32,
+ pub(crate) index_templates: BTreeMap<String, PersistedIndexTemplateV1>,
+ pub(crate) legacy_templates: BTreeMap<String, Value>,
+ pub(crate) component_templates: BTreeMap<String, Value>,
+ pub(crate) pipelines: BTreeMap<String, Value>,
+ pub(crate) data_streams: BTreeMap<String, PersistedDataStreamV1>,
+ pub(crate) ilm_policies: BTreeMap<String, Value>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub(crate) struct PersistedIndexTemplateV1 {
+ index_patterns: Vec<String>,
+ settings: Value,
+ mappings: Value,
+ priority: i32,
+}
+
+impl From<IndexTemplate> for PersistedIndexTemplateV1 {
+ fn from(value: IndexTemplate) -> Self {
+ Self {
+ index_patterns: value.index_patterns,
+ settings: value.settings,
+ mappings: value.mappings,
+ priority: value.priority,
+ }
+ }
+}
+
+impl From<PersistedIndexTemplateV1> for IndexTemplate {
+ fn from(value: PersistedIndexTemplateV1) -> Self {
+ Self {
+ index_patterns: value.index_patterns,
+ settings: value.settings,
+ mappings: value.mappings,
+ priority: value.priority,
+ }
+ }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub(crate) struct PersistedDataStreamV1 {
+ name: String,
+ backing_indices: Vec<String>,
+ timestamp_field: String,
+ generation: u64,
+}
+
+impl From<DataStream> for PersistedDataStreamV1 {
+ fn from(value: DataStream) -> Self {
+ Self {
+ name: value.name,
+ backing_indices: value.backing_indices,
+ timestamp_field: value.timestamp_field,
+ generation: value.generation,
+ }
+ }
+}
+
+impl From<PersistedDataStreamV1> for DataStream {
+ fn from(value: PersistedDataStreamV1) -> Self {
+ Self {
+ name: value.name,
+ backing_indices: value.backing_indices,
+ timestamp_field: value.timestamp_field,
+ generation: value.generation,
+ }
+ }
+}
+
+/// Read and fully classify the cluster-state path once, before any index is
+/// opened. The returned document is the only input later applied to live maps.
+pub(crate) fn preflight(data_dir: &Path) -> PreparedClusterState {
+ let path = data_dir.join("cluster_state.json");
+ let bytes = match std::fs::read(&path) {
+ Ok(bytes) => bytes,
+ Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
+ return PreparedClusterState {
+ status: ClusterStateBootStatus::Absent,
+ document: None,
+ };
+ }
+ Err(err) => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::ReadFailure,
+ format!("read failed: {err}"),
+ );
+ }
+ };
+
+ let value = match parse_unique_json(&bytes) {
+ Ok(value) => value,
+ Err(ParseFailure::Duplicate(detail)) => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::DuplicateKey,
+ detail,
+ );
+ }
+ Err(ParseFailure::Malformed(detail)) => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::MalformedJson,
+ detail,
+ );
+ }
+ };
+
+ let object = match value.as_object() {
+ Some(object) => object,
+ None => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::IncompatibleSchema,
+ "format-1 cluster state must be a JSON object",
+ );
+ }
+ };
+
+ let legacy = object.get("version");
+ let future = object.get("format_version");
+ match (legacy, future) {
+ (Some(_), Some(_)) => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::AmbiguousDiscriminator,
+ "both version and format_version are present",
+ );
+ }
+ (None, Some(found)) => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::UnsupportedFormat,
+ format!(
+ "unsupported cluster-state format: found format_version={}, supported version={CLUSTER_STATE_V1}",
+ discriminator_label(found)
+ ),
+ );
+ }
+ (None, None) => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::MissingDiscriminator,
+ "required version discriminator is missing",
+ );
+ }
+ (Some(found), None) => match found.as_u64() {
+ Some(v) if v == u64::from(CLUSTER_STATE_V1) => {}
+ Some(v) if v > u64::from(CLUSTER_STATE_V1) => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::UnsupportedFormat,
+ format!(
+ "unsupported cluster-state version: found={v}, supported={CLUSTER_STATE_V1}"
+ ),
+ );
+ }
+ _ => {
+ return PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::InvalidDiscriminator,
+ format!(
+ "version must be the integer {CLUSTER_STATE_V1}, found {}",
+ discriminator_label(found)
+ ),
+ );
+ }
+ },
+ }
+
+ match serde_json::from_value::<PersistedClusterStateV1>(value) {
+ Ok(document) => {
+ // Only a document that was completely classified as the exact
+ // writable format 1 authorizes cleanup of the writer's fixed
+ // staging name. A future/malformed document (or an absent live
+ // path) may belong to another binary or an interrupted install;
+ // a diagnostic-shell boot must not delete or create anything.
+ let legacy_tmp = path.with_extension("tmp");
+ if legacy_tmp.exists() {
+ warn!("discarding an incomplete format-1 cluster-state staging file");
+ let _ = std::fs::remove_file(legacy_tmp);
+ }
+ PreparedClusterState {
+ status: ClusterStateBootStatus::LoadedV1,
+ document: Some(document),
+ }
+ }
+ Err(err) => PreparedClusterState::blocked(
+ &path,
+ ClusterStateBlockKind::IncompatibleSchema,
+ format!("format-1 schema is not understood: {err}"),
+ ),
+ }
+}
+
+fn discriminator_label(value: &Value) -> String {
+ match value {
+ Value::String(value) => format!("string {value:?}"),
+ other => other.to_string(),
+ }
+}
+
+enum ParseFailure {
+ Duplicate(String),
+ Malformed(String),
+}
+
+fn parse_unique_json(bytes: &[u8]) -> Result<Value, ParseFailure> {
+ let mut deserializer = serde_json::Deserializer::from_slice(bytes);
+ let value = UniqueValue::deserialize(&mut deserializer).map_err(classify_parse_error)?;
+ deserializer.end().map_err(classify_parse_error)?;
+ Ok(value.0)
+}
+
+fn classify_parse_error(err: serde_json::Error) -> ParseFailure {
+ let detail = err.to_string();
+ if detail.contains(DUPLICATE_KEY_MARKER) {
+ ParseFailure::Duplicate(detail)
+ } else {
+ ParseFailure::Malformed(detail)
+ }
+}
+
+struct UniqueValue(Value);
+
+impl<'de> Deserialize<'de> for UniqueValue {
+ fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ deserializer.deserialize_any(UniqueValueVisitor)
+ }
+}
+
+struct UniqueValueVisitor;
+
+impl<'de> Visitor<'de> for UniqueValueVisitor {
+ type Value = UniqueValue;
+
+ fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ formatter.write_str("one JSON value without duplicate object keys")
+ }
+
+ fn visit_bool<E>(self, value: bool) -> Result<Self::Value, E> {
+ Ok(UniqueValue(Value::Bool(value)))
+ }
+
+ fn visit_i64<E>(self, value: i64) -> Result<Self::Value, E> {
+ Ok(UniqueValue(Value::Number(value.into())))
+ }
+
+ fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E> {
+ Ok(UniqueValue(Value::Number(value.into())))
+ }
+
+ fn visit_f64<E>(self, value: f64) -> Result<Self::Value, E>
+ where
+ E: serde::de::Error,
+ {
+ Number::from_f64(value)
+ .map(Value::Number)
+ .map(UniqueValue)
+ .ok_or_else(|| E::custom("non-finite JSON number"))
+ }
+
+ fn visit_str<E>(self, value: &str) -> Result<Self::Value, E> {
+ Ok(UniqueValue(Value::String(value.to_owned())))
+ }
+
+ fn visit_string<E>(self, value: String) -> Result<Self::Value, E> {
+ Ok(UniqueValue(Value::String(value)))
+ }
+
+ fn visit_none<E>(self) -> Result<Self::Value, E> {
+ Ok(UniqueValue(Value::Null))
+ }
+
+ fn visit_unit<E>(self) -> Result<Self::Value, E> {
+ Ok(UniqueValue(Value::Null))
+ }
+
+ fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
+ where
+ D: Deserializer<'de>,
+ {
+ UniqueValue::deserialize(deserializer)
+ }
+
+ fn visit_seq<A>(self, mut sequence: A) -> Result<Self::Value, A::Error>
+ where
+ A: SeqAccess<'de>,
+ {
+ let mut values = Vec::new();
+ while let Some(value) = sequence.next_element::<UniqueValue>()? {
+ values.push(value.0);
+ }
+ Ok(UniqueValue(Value::Array(values)))
+ }
+
+ fn visit_map<A>(self, mut object: A) -> Result<Self::Value, A::Error>
+ where
+ A: MapAccess<'de>,
+ {
+ let mut seen = HashSet::new();
+ let mut values = Map::new();
+ while let Some(key) = object.next_key::<String>()? {
+ if !seen.insert(key.clone()) {
+ return Err(A::Error::custom(format!("{DUPLICATE_KEY_MARKER} {key:?}")));
+ }
+ let value = object.next_value::<UniqueValue>()?;
+ values.insert(key, value.0);
+ }
+ Ok(UniqueValue(Value::Object(values)))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[derive(Clone, Default)]
+ struct LogBuffer(std::sync::Arc<std::sync::Mutex<Vec<u8>>>);
+
+ struct LogWriter(std::sync::Arc<std::sync::Mutex<Vec<u8>>>);
+
+ impl std::io::Write for LogWriter {
+ fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
+ self.0.lock().unwrap().extend_from_slice(bytes);
+ Ok(bytes.len())
+ }
+
+ fn flush(&mut self) -> std::io::Result<()> {
+ Ok(())
+ }
+ }
+
+ impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for LogBuffer {
+ type Writer = LogWriter;
+
+ fn make_writer(&'a self) -> Self::Writer {
+ LogWriter(self.0.clone())
+ }
+ }
+
+ fn capture_preflight_log(dir: &Path) -> (PreparedClusterState, String) {
+ let buffer = LogBuffer::default();
+ let subscriber = tracing_subscriber::fmt()
+ .without_time()
+ .with_ansi(false)
+ .with_target(false)
+ .with_writer(buffer.clone())
+ .finish();
+ let guard = tracing::subscriber::set_default(subscriber);
+ let prepared = preflight(dir);
+ drop(guard);
+ let bytes = buffer.0.lock().unwrap().clone();
+ (prepared, String::from_utf8(bytes).unwrap())
+ }
+
+ fn exact_v1(extra: &str) -> Vec<u8> {
+ format!(
+ r#"{{
+ "version": 1,
+ "index_templates": {{}},
+ "legacy_templates": {{}},
+ "component_templates": {{}},
+ "pipelines": {{}},
+ "data_streams": {{}},
+ "ilm_policies": {{{extra}}}
+ }}"#
+ )
+ .into_bytes()
+ }
+
+ fn preflight_bytes(bytes: &[u8]) -> PreparedClusterState {
+ let dir = tempfile::tempdir().unwrap();
+ std::fs::write(dir.path().join("cluster_state.json"), bytes).unwrap();
+ preflight(dir.path())
+ }
+
+ #[test]
+ fn cluster_state_format1_parser_accepts_exact_v1_and_open_values() {
+ let prepared = preflight_bytes(&exact_v1(
+ r#""policy": {"arbitrary": {"future-user-key": [1, true, null]}}"#,
+ ));
+ assert_eq!(prepared.status, ClusterStateBootStatus::LoadedV1);
+ assert!(prepared.document.is_some());
+ }
+
+ #[test]
+ fn cluster_state_format1_parser_blocks_every_invalid_discriminator_shape() {
+ let cases: &[(&[u8], ClusterStateBlockKind)] = &[
+ (br#"{}"#, ClusterStateBlockKind::MissingDiscriminator),
+ (
+ br#"{"version":0}"#,
+ ClusterStateBlockKind::InvalidDiscriminator,
+ ),
+ (
+ br#"{"version":-1}"#,
+ ClusterStateBlockKind::InvalidDiscriminator,
+ ),
+ (
+ br#"{"version":1.0}"#,
+ ClusterStateBlockKind::InvalidDiscriminator,
+ ),
+ (
+ br#"{"version":"1"}"#,
+ ClusterStateBlockKind::InvalidDiscriminator,
+ ),
+ (
+ br#"{"version":null}"#,
+ ClusterStateBlockKind::InvalidDiscriminator,
+ ),
+ (
+ br#"{"version":4294967296}"#,
+ ClusterStateBlockKind::UnsupportedFormat,
+ ),
+ (
+ br#"{"version":2}"#,
+ ClusterStateBlockKind::UnsupportedFormat,
+ ),
+ (
+ br#"{"format_version":2}"#,
+ ClusterStateBlockKind::UnsupportedFormat,
+ ),
+ (
+ br#"{"version":1,"format_version":2}"#,
+ ClusterStateBlockKind::AmbiguousDiscriminator,
+ ),
+ ];
+ for (bytes, expected) in cases {
+ let prepared = preflight_bytes(bytes);
+ assert_eq!(prepared.status.block_kind(), Some(expected), "{bytes:?}");
+ }
+ }
+
+ #[test]
+ fn cluster_state_format1_parser_reports_exact_found_and_supported_versions() {
+ let prepared = preflight_bytes(br#"{"format_version":2}"#);
+ assert_eq!(
+ prepared.status,
+ ClusterStateBootStatus::Blocked {
+ kind: ClusterStateBlockKind::UnsupportedFormat,
+ detail:
+ "unsupported cluster-state format: found format_version=2, supported version=1"
+ .to_owned(),
+ }
+ );
+
+ let prepared = preflight_bytes(br#"{"version":2}"#);
+ assert_eq!(
+ prepared.status,
+ ClusterStateBootStatus::Blocked {
+ kind: ClusterStateBlockKind::UnsupportedFormat,
+ detail: "unsupported cluster-state version: found=2, supported=1".to_owned(),
+ }
+ );
+ }
+
+ #[test]
+ fn cluster_state_format1_parser_rejects_trailing_non_whitespace() {
+ let mut bytes = exact_v1("");
+ bytes.extend_from_slice(b" true");
+ let prepared = preflight_bytes(&bytes);
+ assert_eq!(
+ prepared.status.block_kind(),
+ Some(&ClusterStateBlockKind::MalformedJson)
+ );
+ }
+
+ #[test]
+ fn every_representative_block_class_logs_kind_detail_and_operator_path_at_boot() {
+ let cases: &[(&[u8], ClusterStateBlockKind, &str)] = &[
+ (
+ br#"{"format_version":2}"#,
+ ClusterStateBlockKind::UnsupportedFormat,
+ "found format_version=2, supported version=1",
+ ),
+ (
+ br#"{"version":1}"#,
+ ClusterStateBlockKind::IncompatibleSchema,
+ "format-1 schema is not understood",
+ ),
+ (
+ br#"{"version":1,"version":1}"#,
+ ClusterStateBlockKind::DuplicateKey,
+ "duplicate JSON object key",
+ ),
+ ];
+ for (bytes, kind, detail) in cases {
+ let dir = tempfile::tempdir().unwrap();
+ let path = dir.path().join("cluster_state.json");
+ std::fs::write(&path, bytes).unwrap();
+ let (prepared, log) = capture_preflight_log(dir.path());
+ assert_eq!(prepared.status.block_kind(), Some(kind));
+ assert!(
+ log.contains("cluster state was rejected at boot"),
+ "{kind:?}: {log}"
+ );
+ assert!(
+ log.contains(
+ "start a compatible XERJ build or restore supported bytes, then restart"
+ ),
+ "{kind:?}: {log}"
+ );
+ assert!(log.contains(&format!("kind={kind:?}")), "{kind:?}: {log}");
+ assert!(log.contains(detail), "{kind:?}: {log}");
+ assert!(log.contains(&path.display().to_string()), "{kind:?}: {log}");
+ }
+ }
+
+ #[cfg(unix)]
+ #[test]
+ fn read_failure_logs_the_same_boot_diagnostic_contract() {
+ use std::os::unix::fs::symlink;
+
+ let dir = tempfile::tempdir().unwrap();
+ let path = dir.path().join("cluster_state.json");
+ symlink("cluster_state.json", &path).unwrap();
+ let (prepared, log) = capture_preflight_log(dir.path());
+ assert_eq!(
+ prepared.status.block_kind(),
+ Some(&ClusterStateBlockKind::ReadFailure)
+ );
+ assert!(log.contains("kind=ReadFailure"), "{log}");
+ assert!(log.contains("read failed:"), "{log}");
+ assert!(log.contains(&path.display().to_string()), "{log}");
+ }
+
+ #[test]
+ fn cluster_state_format1_parser_rejects_duplicates_at_every_depth() {
+ let top = exact_v1(r#""a": {}, "a": {}"#);
+ let nested = exact_v1(r#""a": {"policy": {"x": 1, "x": 2}}"#);
+ for bytes in [&top, &nested] {
+ let prepared = preflight_bytes(bytes);
+ assert_eq!(
+ prepared.status.block_kind(),
+ Some(&ClusterStateBlockKind::DuplicateKey)
+ );
+ }
+ }
+
+ #[test]
+ fn cluster_state_format1_parser_rejects_unknown_and_missing_typed_fields() {
+ let unknown_envelope = {
+ let mut bytes = exact_v1("");
+ let end = bytes.iter().rposition(|byte| *byte == b'}').unwrap();
+ bytes.splice(end..end, b",\"future\":true".iter().copied());
+ bytes
+ };
+ let unknown_template = br#"{
+ "version":1,
+ "index_templates":{"x":{"index_patterns":[],"settings":{},"mappings":{},"priority":0,"future":true}},
+ "legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{}
+ }"#;
+ let missing_map = br#"{
+ "version":1,"index_templates":{},"legacy_templates":{},
+ "component_templates":{},"pipelines":{},"data_streams":{}
+ }"#;
+ for bytes in [unknown_envelope.as_slice(), unknown_template, missing_map] {
+ let prepared = preflight_bytes(bytes);
+ assert_eq!(
+ prepared.status.block_kind(),
+ Some(&ClusterStateBlockKind::IncompatibleSchema)
+ );
+ }
+ }
+
+ #[test]
+ fn blocked_or_absent_preflight_neither_cleans_staging_nor_writes_salvage() {
+ let cases: &[Option<&[u8]>] = &[
+ Some(br#"{"format_version":2}"#),
+ Some(br#"{"version":1,"cut":true"#),
+ None,
+ ];
+ for live in cases {
+ let dir = tempfile::tempdir().unwrap();
+ let state = dir.path().join("cluster_state.json");
+ if let Some(bytes) = live {
+ std::fs::write(&state, bytes).unwrap();
+ }
+ let staging = dir.path().join("cluster_state.tmp");
+ let staging_bytes = b"owned-by-another-or-interrupted-writer";
+ std::fs::write(&staging, staging_bytes).unwrap();
+
+ let prepared = preflight(dir.path());
+ assert!(
+ !prepared.status.is_writable() || live.is_none(),
+ "fixture must not classify as loaded format 1"
+ );
+ assert_eq!(std::fs::read(&staging).unwrap(), staging_bytes);
+ assert!(!dir.path().join("cluster_state.corrupt.json").exists());
+ }
+ }
+
+ #[test]
+ fn loaded_format1_alone_authorizes_legacy_staging_cleanup() {
+ let dir = tempfile::tempdir().unwrap();
+ std::fs::write(dir.path().join("cluster_state.json"), exact_v1("")).unwrap();
+ let staging = dir.path().join("cluster_state.tmp");
+ std::fs::write(&staging, b"interrupted format-1 stage").unwrap();
+
+ let prepared = preflight(dir.path());
+ assert_eq!(prepared.status, ClusterStateBootStatus::LoadedV1);
+ assert!(!staging.exists());
+ }
+}
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index 08b645e04..e81f6e742 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -12,6 +12,10 @@ use tracing::{error, info, warn};
use xerj_common::config::Config;
use xerj_common::types::{IndexName, Schema};
+use crate::cluster_state::{
+ ClusterStateBootStatus, PersistedClusterStateV1, PersistedDataStreamV1,
+ PersistedIndexTemplateV1, PreparedClusterState, CLUSTER_STATE_V1,
+};
use crate::index::{Index, IndexStats};
use crate::{EngineError, Result};
@@ -138,45 +142,6 @@ pub struct IndexTemplate {
pub priority: i32,
}
-/// Version stamped into `cluster_state.json`. Bump only when the on-disk
-/// shape changes in a way an older build cannot read; new *optional* fields
-/// (`#[serde(default)]`) do not need one.
-const CLUSTER_STATE_VERSION: u32 = 1;
-
-/// Everything in `<data_dir>/cluster_state.json` — the cluster-level
-/// management state that used to be in-memory only (issue #203).
-///
-/// `BTreeMap` rather than `HashMap` so the file is byte-stable for a given
-/// state: an operator diffing two nodes' `cluster_state.json`, or a backup
-/// tool deduplicating it, sees a change only when the state really changed.
-/// Every field is `#[serde(default)]`, so a file written by an older build
-/// (or by one that gains another map later) still loads.
-#[derive(Debug, Default, Serialize, Deserialize)]
-struct PersistedClusterState {
- #[serde(default)]
- version: u32,
- /// v2 index templates — `PUT /_index_template/{name}`.
- #[serde(default)]
- index_templates: std::collections::BTreeMap<String, IndexTemplate>,
- /// v1 legacy templates — `PUT /_template/{name}`.
- #[serde(default)]
- legacy_templates: std::collections::BTreeMap<String, Value>,
- /// Component templates — `PUT /_component_template/{name}`.
- #[serde(default)]
- component_templates: std::collections::BTreeMap<String, Value>,
- /// Ingest pipeline definitions — `PUT /_ingest/pipeline/{id}`. Stored as
- /// the document `GET` hands back, and recompiled into an executable
- /// pipeline on boot.
- #[serde(default)]
- pipelines: std::collections::BTreeMap<String, Value>,
- /// Data streams — `PUT /_data_stream/{name}` plus every rollover.
- #[serde(default)]
- data_streams: std::collections::BTreeMap<String, DataStream>,
- /// ILM policies — `PUT /_ilm/policy/{name}`.
- #[serde(default)]
- ilm_policies: std::collections::BTreeMap<String, Value>,
-}
-
/// Active scroll context holding all matching hits.
///
/// Each context pins a fully-hydrated `Vec<Hit>` snapshot, so its lifetime
@@ -654,11 +619,10 @@ pub struct Engine {
/// Never taken while holding a `DashMap` guard.
cluster_state_write: Arc<parking_lot::Mutex<()>>,
- /// False once `<data_dir>/cluster_state.json` was found on disk and could
- /// not be loaded — unreadable (`EACCES` after a uid change on a container
- /// volume, `EIO`, `EMFILE` at boot) or unparseable. While false,
- /// `flush_cluster_state` refuses to write and every management PUT
- /// answers 500.
+ /// Immutable classification of the one `cluster_state.json` document read
+ /// after `node.lock` and before any index/WAL open. A blocked status keeps
+ /// readiness false and refuses every metadata mutation for this process
+ /// lifetime; recovery requires replacing the bytes and restarting.
///
/// Without this the failure is silent *and* destructive. The maps boot
/// empty, so the first `PUT /_index_template/...` snapshots six empty
@@ -683,7 +647,7 @@ pub struct Engine {
/// (`redb/src/tree_store/page_store/cached_file.rs:125-145`,
/// `redb/src/error.rs:65`; Apache-2.0/MIT, shape adapted, no code
/// copied).
- cluster_state_loaded: Arc<std::sync::atomic::AtomicBool>,
+ cluster_state_boot: Arc<ClusterStateBootStatus>,
}
impl Engine {
@@ -715,6 +679,15 @@ impl Engine {
// process serves the same directory.
let node_lock = Arc::new(Self::acquire_node_lock(&data_dir)?);
+ // Classify durable cluster metadata before Index::open can replay a
+ // WAL or startup reconciliation can rewrite it. The parsed document
+ // is retained and applied later; the path is never reread this boot.
+ let PreparedClusterState {
+ status: cluster_state_status,
+ document: prepared_cluster_state,
+ } = crate::cluster_state::preflight(&data_dir);
+ let storage_available = cluster_state_status.is_writable();
+
// Apply operator-tunable aggregation bucket cap. Stored in a static
// AtomicUsize inside aggs.rs so all per-bucket-allocator hot loops
// can read it with no plumbing through every agg signature.
@@ -784,10 +757,18 @@ impl Engine {
// is evidence of. Falls back to in-memory (with a warning) if the
// file cannot be opened — an unwritable audit log must not stop
// the node booting.
- audit: crate::audit::AuditLog::open(
- crate::audit::DEFAULT_AUDIT_CAPACITY,
- data_dir.join("audit.jsonl"),
- ),
+ // A blocked compatibility-fence boot is a diagnostic shell, not a
+ // partially-open storage node. Opening the durable audit sink
+ // rewrites audit.jsonl from its ring during construction, so keep
+ // even that non-index store memory-only until a supported restart.
+ audit: if storage_available {
+ crate::audit::AuditLog::open(
+ crate::audit::DEFAULT_AUDIT_CAPACITY,
+ data_dir.join("audit.jsonl"),
+ )
+ } else {
+ crate::audit::AuditLog::new(crate::audit::DEFAULT_AUDIT_CAPACITY)
+ },
roles: crate::rbac::RoleStore::new(),
// Single-node default: 1 shard, "local" owner. Writes never
// forward; multi-node mode overrides these via the Raft
@@ -798,134 +779,145 @@ impl Engine {
)),
_node_lock: node_lock,
cluster_state_write: Arc::new(parking_lot::Mutex::new(())),
- cluster_state_loaded: Arc::new(std::sync::atomic::AtomicBool::new(true)),
+ cluster_state_boot: Arc::new(cluster_state_status),
};
- // Scan data_dir for existing index directories.
- if let Ok(read_dir) = std::fs::read_dir(&data_dir) {
- for entry in read_dir.flatten() {
- let path = entry.path();
- if !path.is_dir() {
- continue;
- }
- // Check if this looks like an index directory (has a WAL subdirectory).
- if !path.join("wal").exists() {
- continue;
- }
- let name_str = match path.file_name().and_then(|n| n.to_str()) {
- Some(n) => n.to_string(),
- None => continue,
- };
- let index_name = match IndexName::new(&name_str) {
- Ok(n) => n,
- Err(_) => {
- warn!("Skipping directory '{}': not a valid index name", name_str);
+ // A rejected cluster-state document means this binary cannot know the
+ // storage topology that document describes. Do not open *any* index in
+ // that state: Index::open replays WALs and may publish a segment. This
+ // includes `.xerj_*` system indices; treating them as harmless was the
+ // exact path that mutated `.xerj_dashboards` during the causal A/B.
+ if storage_available {
+ // Scan data_dir for existing index directories.
+ if let Ok(read_dir) = std::fs::read_dir(&data_dir) {
+ for entry in read_dir.flatten() {
+ let path = entry.path();
+ if !path.is_dir() {
continue;
}
- };
- match Index::open(index_name.clone(), &engine.config, &data_dir) {
- Ok(idx) => {
- info!(name = name_str.as_str(), "opened existing index");
- // Restore the raw ES mapping blob (analyzers, formats,
- // dims — full fidelity) BEFORE any ingest/query can run,
- // so GET /_mapping and mapping-dependent code paths see
- // the same mapping as pre-restart. A corrupt blob fails
- // the index (#202) rather than serving it with a
- // silently reduced mapping.
- if let Err(e) = engine.load_persisted_es_mapping(&name_str) {
- warn!(name = name_str.as_str(), error = %e, "failed to open index");
- engine.record_failed_index(&name_str, e.to_string());
+ // Check if this looks like an index directory (has a WAL subdirectory).
+ if !path.join("wal").exists() {
+ continue;
+ }
+ let name_str = match path.file_name().and_then(|n| n.to_str()) {
+ Some(n) => n.to_string(),
+ None => continue,
+ };
+ let index_name = match IndexName::new(&name_str) {
+ Ok(n) => n,
+ Err(_) => {
+ warn!("Skipping directory '{}': not a valid index name", name_str);
continue;
}
- // The index isn't registered yet, so the propagation
- // inside load can't find it — set the toggles on the
- // local handle instead.
- if let Some(m) = engine.index_mappings.get(name_str.as_str()) {
- Engine::apply_date_mapping_flags(&idx, m.value());
+ };
+ match Index::open(index_name.clone(), &engine.config, &data_dir) {
+ Ok(idx) => {
+ info!(name = name_str.as_str(), "opened existing index");
+ // Restore the raw ES mapping blob (analyzers, formats,
+ // dims — full fidelity) BEFORE any ingest/query can run,
+ // so GET /_mapping and mapping-dependent code paths see
+ // the same mapping as pre-restart. A corrupt blob fails
+ // the index (#202) rather than serving it with a
+ // silently reduced mapping.
+ if let Err(e) = engine.load_persisted_es_mapping(&name_str) {
+ warn!(name = name_str.as_str(), error = %e, "failed to open index");
+ engine.record_failed_index(&name_str, e.to_string());
+ continue;
+ }
+ // The index isn't registered yet, so the propagation
+ // inside load can't find it — set the toggles on the
+ // local handle instead.
+ if let Some(m) = engine.index_mappings.get(name_str.as_str()) {
+ Engine::apply_date_mapping_flags(&idx, m.value());
+ }
+ engine.load_or_backfill_index_created_at(&name_str);
+ engine.indices.insert(name_str, idx);
}
- engine.load_or_backfill_index_created_at(&name_str);
- engine.indices.insert(name_str, idx);
- }
- Err(e) => {
- warn!(name = name_str.as_str(), error = %e, "failed to open index");
- // The mapping blob lives beside the data, not inside
- // the store, so it is readable even when the store
- // refuses to open. Load it so the metadata surfaces
- // (`GET /{index}`, `GET /{index}/_mapping`) can still
- // tell the operator what was in the index they are
- // trying to recover. Propagation into the (absent)
- // handle no-ops. Since #202 this load can itself fail
- // (unreadable/unparseable `es_mapping.json`); the index
- // is already being quarantined for the store error, so
- // the extra failure is reported rather than dropped —
- // it tells the operator a second file needs repairing.
- if let Err(map_err) = engine.load_persisted_es_mapping(&name_str) {
- warn!(
- name = name_str.as_str(),
- error = %map_err,
- "es_mapping.json is also unreadable; the mapping surfaces \
- cannot show what this index held"
- );
+ Err(e) => {
+ warn!(name = name_str.as_str(), error = %e, "failed to open index");
+ // The mapping blob lives beside the data, not inside
+ // the store, so it is readable even when the store
+ // refuses to open. Load it so the metadata surfaces
+ // (`GET /{index}`, `GET /{index}/_mapping`) can still
+ // tell the operator what was in the index they are
+ // trying to recover. Propagation into the (absent)
+ // handle no-ops. Since #202 this load can itself fail
+ // (unreadable/unparseable `es_mapping.json`); the index
+ // is already being quarantined for the store error, so
+ // the extra failure is reported rather than dropped —
+ // it tells the operator a second file needs repairing.
+ if let Err(map_err) = engine.load_persisted_es_mapping(&name_str) {
+ warn!(
+ name = name_str.as_str(),
+ error = %map_err,
+ "es_mapping.json is also unreadable; the mapping surfaces \
+ cannot show what this index held"
+ );
+ }
+ engine.record_failed_index(&name_str, e.to_string());
}
- engine.record_failed_index(&name_str, e.to_string());
}
}
}
- }
- // Restore persisted API keys (item 6) so keys minted before a restart
- // still authenticate. Must run before the server starts accepting
- // requests (i.e. here in `new`), and is cheap (one small JSON file).
- engine.load_persisted_api_keys();
-
- // Restore persisted aliases so e.g. `.kibana` (always an alias,
- // never a bare index) still resolves after a restart — see
- // `load_persisted_aliases`'s doc comment for the concrete failure
- // this caused (OpenSearch Dashboards stuck indefinitely on every
- // restart, mistaking a missing-alias 404 for a still-in-progress
- // migration by another instance).
- engine.load_persisted_aliases();
-
- // Restore the cluster-level management state — index templates,
- // legacy templates, component templates, ingest pipelines, data
- // streams, ILM policies (issue #203). Before this, "replace the
- // binary and restart" silently reverted every one of them: the
- // documents kept flowing, into indices that no longer had the shape
- // the operator designed. Must run after the index scan above, so a
- // restored data stream can report which of its backing indices are
- // actually present, and before the server accepts requests.
- engine.load_cluster_state();
-
- // Then say plainly which `.ds-*` indices no restored stream claims.
- // Nothing is deleted — see the doc comment; this is the log line that
- // stops an unreachable backing index from being invisible.
- engine.warn_orphaned_backing_indices();
-
- // Restore ISM/ILM policies and managed-index execution state so a
- // policy attached before a restart keeps running afterward instead
- // of silently going idle. Separate files (`ism_policies.json`,
- // `ism_managed_indices.json`) from `cluster_state.json` above, and
- // deliberately so: that file holds the verbatim documents an
- // operator PUT, this holds the executor's own cursor. Loaded after
- // `load_cluster_state` because a restored data stream is what a
- // rollover action operates on.
- engine.load_persisted_ism_policies();
- engine.load_persisted_managed_indices();
-
- // Spawn the PIT sweeper. Pre-v0.6.2 PITs accumulated forever;
- // every open without close was a memory leak. The sweeper
- // walks `engine.pits` every `pit.sweep_interval_secs` and
- // drops any with `expires_at < now`. Cheap (DashMap iter +
- // Instant compare) and bounded by the live PIT count.
- engine.spawn_pit_sweeper();
-
- // Spawn the scroll + async-search context sweeper (RC4 blocker
- // 11) — the scroll/async twin of the PIT sweeper above. Each
- // scroll pins a full Vec<Hit> and each async search pins its
- // response JSON; without a sweeper both maps grow until an
- // explicit client DELETE, i.e. forever under normal client
- // behavior.
- engine.spawn_search_context_sweeper();
+ // Restore persisted API keys (item 6) so keys minted before a restart
+ // still authenticate. Must run before the server starts accepting
+ // requests (i.e. here in `new`), and is cheap (one small JSON file).
+ engine.load_persisted_api_keys();
+
+ // Restore persisted aliases so e.g. `.kibana` (always an alias,
+ // never a bare index) still resolves after a restart — see
+ // `load_persisted_aliases`'s doc comment for the concrete failure
+ // this caused (OpenSearch Dashboards stuck indefinitely on every
+ // restart, mistaking a missing-alias 404 for a still-in-progress
+ // migration by another instance).
+ engine.load_persisted_aliases();
+
+ // Restore the cluster-level management state — index templates,
+ // legacy templates, component templates, ingest pipelines, data
+ // streams, ILM policies (issue #203). Before this, "replace the
+ // binary and restart" silently reverted every one of them: the
+ // documents kept flowing, into indices that no longer had the shape
+ // the operator designed. Must run after the index scan above, so a
+ // restored data stream can report which of its backing indices are
+ // actually present, and before the server accepts requests.
+ engine.apply_cluster_state(prepared_cluster_state);
+
+ // Then say plainly which `.ds-*` indices no restored stream claims.
+ // Nothing is deleted — see the doc comment; this is the log line that
+ // stops an unreachable backing index from being invisible.
+ engine.warn_orphaned_backing_indices();
+
+ // Restore ISM/ILM policies and managed-index execution state so a
+ // policy attached before a restart keeps running afterward instead
+ // of silently going idle. Separate files (`ism_policies.json`,
+ // `ism_managed_indices.json`) from `cluster_state.json` above, and
+ // deliberately so: that file holds the verbatim documents an
+ // operator PUT, this holds the executor's own cursor. Loaded after
+ // `apply_cluster_state` because a restored data stream is what a
+ // rollover action operates on.
+ engine.load_persisted_ism_policies();
+ engine.load_persisted_managed_indices();
+
+ // Spawn the PIT sweeper. Pre-v0.6.2 PITs accumulated forever;
+ // every open without close was a memory leak. The sweeper
+ // walks `engine.pits` every `pit.sweep_interval_secs` and
+ // drops any with `expires_at < now`. Cheap (DashMap iter +
+ // Instant compare) and bounded by the live PIT count.
+ engine.spawn_pit_sweeper();
+
+ // Spawn the scroll + async-search context sweeper (RC4 blocker
+ // 11) — the scroll/async twin of the PIT sweeper above. Each
+ // scroll pins a full Vec<Hit> and each async search pins its
+ // response JSON; without a sweeper both maps grow until an
+ // explicit client DELETE, i.e. forever under normal client
+ // behavior.
+ engine.spawn_search_context_sweeper();
+ } else {
+ warn!(
+ "storage activation skipped because the boot-time cluster-state format fence is latched"
+ );
+ }
Ok(engine)
}
@@ -1188,6 +1180,11 @@ impl Engine {
schema: Schema,
settings: serde_json::Value,
) -> Result<()> {
+ // Index creation is storage activation: it creates the directory and
+ // WAL before any cluster-state serializer is involved. A blocked boot
+ // must reject here so HTTP, gRPC, direct CLI, and Console callers share
+ // the same fail-closed boundary.
+ self.ensure_cluster_state_writable()?;
let index_name = IndexName::new(name).map_err(EngineError::Common)?;
// Per-index authorization backstop (issue #79): a caller that may not
@@ -1241,7 +1238,8 @@ impl Engine {
///
/// This is the single write path for `engine.index_mappings` — both
/// index-create-with-mappings and PUT /_mapping go through here.
- pub fn put_index_mapping(&self, name: &str, mapping: Value) {
+ pub fn put_index_mapping(&self, name: &str, mapping: Value) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
let index_dir = self.data_dir.join(name);
if index_dir.is_dir() {
match serde_json::to_vec_pretty(&mapping) {
@@ -1259,6 +1257,7 @@ impl Engine {
}
self.propagate_date_detection(name, &mapping);
self.index_mappings.insert(name.to_string(), mapping);
+ Ok(())
}
/// Push the mapping's `date_detection` toggle (default true) down to the
@@ -1375,12 +1374,15 @@ impl Engine {
/// mutation snapshots the full map to `<data_dir>/api_keys.json` (0600,
/// atomic temp+rename), reloaded on boot.
///
- /// The write is best-effort: a persistence failure is logged but does not
- /// fail the create — the key still works until the next restart, matching
- /// the old behavior rather than regressing key creation.
- pub fn persist_api_key(&self, id: String, record: ApiKeyRecord) {
+ /// A blocked boot returns the typed compatibility error before inserting
+ /// the key. Once storage is writable, an ordinary persistence I/O failure
+ /// remains best-effort: it is logged but does not fail the create — the key
+ /// still works until the next restart, matching the old behavior.
+ pub fn persist_api_key(&self, id: String, record: ApiKeyRecord) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
self.api_keys.insert(id, record);
self.flush_api_keys_best_effort();
+ Ok(())
}
/// Invalidate (revoke) minted API keys by id — issue #208's missing half.
@@ -1398,7 +1400,12 @@ impl Engine {
/// middleware reads this same map, so revocation takes effect on the very
/// next request — and the store is flushed to `api_keys.json` once at the
/// end, same durability contract as [`Self::persist_api_key`].
- pub fn invalidate_api_keys(&self, ids: &[String], now_ms: u64) -> (Vec<String>, Vec<String>) {
+ pub fn invalidate_api_keys(
+ &self,
+ ids: &[String],
+ now_ms: u64,
+ ) -> Result<(Vec<String>, Vec<String>)> {
+ self.ensure_cluster_state_writable()?;
let mut invalidated = Vec::new();
let mut previously = Vec::new();
for id in ids {
@@ -1418,7 +1425,7 @@ impl Engine {
if !invalidated.is_empty() {
self.flush_api_keys_best_effort();
}
- (invalidated, previously)
+ Ok((invalidated, previously))
}
/// Serialize the current `api_keys` map to `<data_dir>/api_keys.json`
@@ -1634,17 +1641,24 @@ impl Engine {
/// store, mirroring `flush_aliases`. Both `_plugins/_ism/policies` (as
/// entered) and `_ilm/policy` (after translation) write through here —
/// see the `ism_policies` field doc.
- pub fn put_ism_policy(&self, id: String, policy: crate::lifecycle::LifecyclePolicy) {
+ pub fn put_ism_policy(
+ &self,
+ id: String,
+ policy: crate::lifecycle::LifecyclePolicy,
+ ) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
self.ism_policies.insert(id, policy);
self.flush_ism_policies();
+ Ok(())
}
- pub fn remove_ism_policy(&self, id: &str) -> bool {
+ pub fn remove_ism_policy(&self, id: &str) -> Result<bool> {
+ self.ensure_cluster_state_writable()?;
let removed = self.ism_policies.remove(id).is_some();
if removed {
self.flush_ism_policies();
}
- removed
+ Ok(removed)
}
fn flush_ism_policies(&self) {
@@ -1717,7 +1731,8 @@ impl Engine {
/// [...]}` — so the acknowledged detaches survive a restart alongside
/// the cursors. `load_persisted_managed_indices` still accepts the
/// pre-#282 bare-map form.
- pub fn persist_managed_indices(&self) {
+ pub fn persist_managed_indices(&self) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
let managed: std::collections::HashMap<String, crate::lifecycle::ManagedIndexState> = self
.managed_indices
.iter()
@@ -1734,12 +1749,13 @@ impl Engine {
Ok(b) => b,
Err(e) => {
warn!(error = %e, "failed to serialize managed_indices for persistence");
- return;
+ return Ok(());
}
};
if let Err(e) = crate::index::write_file_atomic(&self.managed_indices_path(), &bytes) {
warn!(error = %e, "failed to persist ism_managed_indices.json (state works until restart)");
}
+ Ok(())
}
fn load_persisted_managed_indices(&self) {
@@ -1798,7 +1814,8 @@ impl Engine {
/// future settings-derived re-attach would trip over.
///
/// Returns whether the index was managed before the call.
- pub fn detach_lifecycle(&self, index: &str) -> bool {
+ pub fn detach_lifecycle(&self, index: &str) -> Result<bool> {
+ self.ensure_cluster_state_writable()?;
let was_managed = self.managed_indices.remove(index).is_some();
self.lifecycle_detached.insert(index.to_string(), ());
if let Some(mut stored) = self.index_settings.get_mut(index) {
@@ -1813,8 +1830,8 @@ impl Engine {
lifecycle.remove("name");
}
}
- self.persist_managed_indices();
- was_managed
+ self.persist_managed_indices()?;
+ Ok(was_managed)
}
/// Whether `lifecycle::tick` is allowed to act — the `POST /_ilm/stop`
@@ -1921,7 +1938,7 @@ impl Engine {
// engines use for their manifests (cf. `fjall/src/file.rs:17`
// `fsync_directory`, which fjall calls after every atomic rewrite;
// `sled/src/metadata_store.rs:696` discards leftover `*.tmp` files from
- // an interrupted rewrite on the next boot, which `load_cluster_state`
+ // an interrupted rewrite on the next boot, which `apply_cluster_state`
// mirrors below).
//
// One file rather than six keeps the maps mutually consistent — a data
@@ -1934,54 +1951,67 @@ impl Engine {
self.data_dir.join("cluster_state.json")
}
- /// `Err` when boot found `cluster_state.json` and could not load it — see
- /// `cluster_state_loaded` for why that has to latch.
+ /// `Err` when boot did not classify `cluster_state.json` as absent or as
+ /// the exact supported format 1.
///
/// Every management mutation calls this, not only the ones that reach
/// `flush_cluster_state` with something to write. A `DELETE` consults the
/// in-memory map first, and while the load has failed that map is empty,
/// so an unguarded delete answers `404 not found` about an object that is
/// sitting in the document on disk — the same lie in a different shape.
- /// A 500 that names the file is the only honest answer while the node
- /// cannot see the operator's configuration.
- fn ensure_cluster_state_writable(&self) -> Result<()> {
- if self
- .cluster_state_loaded
- .load(std::sync::atomic::Ordering::Acquire)
- {
+ /// A typed 503 is the only honest answer while the node cannot see the
+ /// operator's configuration.
+ pub fn ensure_cluster_state_writable(&self) -> Result<()> {
+ if self.cluster_state_boot.is_writable() {
return Ok(());
}
let path = self.cluster_state_path();
+ let detail = self
+ .cluster_state_boot
+ .detail()
+ .unwrap_or("cluster state was blocked at boot");
error!(
path = %path.display(),
- "refusing a cluster-metadata write: this node could not load \
- cluster_state.json at boot, so its management state is not the \
- operator's. Recover or move the file aside, then restart."
+ detail,
+ "refusing a cluster-metadata write because the boot-time format fence is latched"
);
- Err(EngineError::Common(xerj_common::XerjError::storage(
- format!(
- "refusing to write {} — it could not be loaded at boot, so \
- writing now would destroy cluster configuration that is still \
- on disk; recover or move it aside and restart xerj",
- path.display()
- ),
- )))
+ Err(EngineError::Common(
+ xerj_common::XerjError::cluster_state_unavailable(),
+ ))
}
/// Snapshot every persisted management map into one document.
- fn cluster_state_snapshot(&self) -> PersistedClusterState {
+ fn cluster_state_snapshot(&self) -> PersistedClusterStateV1 {
fn dump<V: Clone>(map: &DashMap<String, V>) -> std::collections::BTreeMap<String, V> {
map.iter()
.map(|e| (e.key().clone(), e.value().clone()))
.collect()
}
- PersistedClusterState {
- version: CLUSTER_STATE_VERSION,
- index_templates: dump(&self.templates),
+ PersistedClusterStateV1 {
+ version: CLUSTER_STATE_V1,
+ index_templates: self
+ .templates
+ .iter()
+ .map(|entry| {
+ (
+ entry.key().clone(),
+ PersistedIndexTemplateV1::from(entry.value().clone()),
+ )
+ })
+ .collect(),
legacy_templates: dump(&self.legacy_templates),
component_templates: dump(&self.component_templates),
pipelines: dump(&self.pipelines),
- data_streams: dump(&self.data_streams),
+ data_streams: self
+ .data_streams
+ .iter()
+ .map(|entry| {
+ (
+ entry.key().clone(),
+ PersistedDataStreamV1::from(entry.value().clone()),
+ )
+ })
+ .collect(),
ilm_policies: dump(&self.ilm_policies),
}
}
@@ -1995,7 +2025,7 @@ impl Engine {
/// the moment of the write instead of at the next restart.
///
/// Refuses outright when boot could not load the existing document — see
- /// `cluster_state_loaded`. The snapshot below is taken from the live maps,
+ /// the boot-time cluster-state fence. The snapshot below is taken from the live maps,
/// which in that state hold only what this process was told after boot, so
/// writing it would rename an empty-ish document over configuration that is
/// still intact on disk.
@@ -2005,6 +2035,9 @@ impl Engine {
// inside the lock also means the bytes that land are never older
// than a rewrite that has already returned.
let _writing = self.cluster_state_write.lock();
+ // The final writer owns the fence too. A future caller that misses a
+ // preguard still cannot normalize an unsupported document.
+ self.ensure_cluster_state_writable()?;
let snapshot = self.cluster_state_snapshot();
let bytes = serde_json::to_vec_pretty(&snapshot)?;
crate::index::write_file_atomic(&self.cluster_state_path(), &bytes)?;
@@ -2031,107 +2064,16 @@ impl Engine {
.unwrap_or(0)
}
- /// Load `<data_dir>/cluster_state.json` back into the management maps.
- ///
- /// A missing file is normal (fresh node, or a data dir written by a build
- /// that predates this). A file that is present but does not load is loud
- /// *and* latching: it means the operator's templates, pipelines and data
- /// streams are not coming back, which is exactly the silence this whole
- /// mechanism exists to remove, so `cluster_state_loaded` goes false and
- /// every subsequent management write is refused rather than allowed to
- /// rename a snapshot of empty maps over the document on disk.
- fn load_cluster_state(&self) {
- // An interrupted rewrite can leave a partial `cluster_state.tmp`
- // behind. It is never read (the rename is the commit point), but
- // leaving it litters the data dir and invites a hand-edit that
- // "restores" a torn document — sled sweeps the same leftovers on
- // boot (`sled/src/metadata_store.rs:696`).
- let tmp = self.cluster_state_path().with_extension("tmp");
- if tmp.exists() {
- warn!(
- path = %tmp.display(),
- "discarding an incomplete cluster_state rewrite left by an unclean shutdown"
- );
- let _ = std::fs::remove_file(&tmp);
- }
-
- let path = self.cluster_state_path();
- let bytes = match std::fs::read(&path) {
- Ok(b) => b,
- Err(e) if e.kind() == std::io::ErrorKind::NotFound => return,
- Err(e) => {
- // The file is *there* and its bytes are almost certainly fine
- // — `EACCES` after a uid change on a container volume, a
- // backup tool's chmod, `EIO`, `EMFILE` at boot. Do not copy it
- // aside (we cannot read it) and do not touch it: mark the load
- // failed so no later write can rename a snapshot of empty maps
- // over configuration that is still perfectly recoverable.
- self.cluster_state_loaded
- .store(false, std::sync::atomic::Ordering::Release);
- error!(
- path = %path.display(), error = %e,
- "could not READ cluster_state.json — index templates, ingest \
- pipelines, data streams and ILM policies are NOT restored, \
- and management writes will be refused until a boot loads it \
- (the file itself is left untouched)"
- );
- return;
- }
- };
- let state: PersistedClusterState = match serde_json::from_slice(&bytes) {
- Ok(s) => s,
- Err(e) => {
- // Same position as the read error above: the load failed, so
- // the live maps are not the operator's configuration and
- // nothing may be written over the document on disk. Refusing
- // rather than salvaging-then-overwriting also closes the
- // second-corruption hole in the `!salvage.exists()` guard
- // below — with an overwrite still allowed, a second damaged
- // document would be destroyed *and* not copied, because the
- // copy from the first one is already there.
- self.cluster_state_loaded
- .store(false, std::sync::atomic::Ordering::Release);
- error!(
- path = %path.display(), error = %e,
- "cluster_state.json is corrupt — index templates, ingest \
- pipelines, data streams and ILM policies are NOT restored, \
- and management writes will be refused until a boot loads it"
- );
- // Keep a copy anyway. The original is now safe from xerj, but
- // un-wedging the node means moving it aside by hand, and
- // hand-recovering a template out of a damaged document is the
- // operator's last option — leave them something to recover
- // from that survives that step. The original is deliberately
- // left in place so every boot keeps logging the error above
- // until someone deals with it, and an existing copy is never
- // clobbered: the first corruption is the informative one.
- let salvage = path.with_extension("corrupt.json");
- if !salvage.exists() {
- match std::fs::write(&salvage, &bytes) {
- Ok(()) => warn!(
- path = %salvage.display(),
- "kept a copy of the corrupt cluster_state for recovery"
- ),
- Err(e) => warn!(
- path = %salvage.display(), error = %e,
- "could not preserve a copy of the corrupt cluster_state"
- ),
- }
- }
- return;
- }
+ /// Apply the exact format-1 document already read and validated before
+ /// index discovery. `None` means either a fresh node or a blocked boot;
+ /// the immutable boot status distinguishes those cases.
+ fn apply_cluster_state(&self, state: Option<PersistedClusterStateV1>) {
+ let Some(state) = state else {
+ return;
};
- if state.version > CLUSTER_STATE_VERSION {
- warn!(
- found = state.version,
- supported = CLUSTER_STATE_VERSION,
- "cluster_state.json was written by a newer xerj; loading what \
- this build understands (downgrade may drop metadata)"
- );
- }
for (name, tmpl) in state.index_templates {
- self.templates.insert(name, tmpl);
+ self.templates.insert(name, tmpl.into());
}
for (name, body) in state.legacy_templates {
self.legacy_templates.insert(name, body);
@@ -2149,7 +2091,8 @@ impl Engine {
// `"status": "GREEN"` without a word in the log is the same quiet lie
// this issue is about.
let mut reconciled_a_stream = false;
- for (name, mut ds) in state.data_streams {
+ for (name, persisted) in state.data_streams {
+ let mut ds: DataStream = persisted.into();
// `rollover_data_stream` creates the new backing index before it
// records the new generation, so `kill -9` in that window leaves
// `.ds-<name>-00000N` on disk while the document still says N-1.
@@ -2262,10 +2205,7 @@ impl Engine {
/// the cluster state did not load — then every stream is unknown and
/// every backing index would look orphaned.
fn warn_orphaned_backing_indices(&self) {
- if !self
- .cluster_state_loaded
- .load(std::sync::atomic::Ordering::Acquire)
- {
+ if !self.cluster_state_boot.is_writable() {
return;
}
// Snapshot both maps before comparing them: nothing here may hold a
@@ -2478,24 +2418,30 @@ impl Engine {
Ok(removed)
}
- /// Add an alias pointing to an index.
- pub fn add_alias(&self, alias: &str, index: &str) {
+ /// Add an alias pointing to an index. A blocked boot fails before the map
+ /// or alias file changes.
+ pub fn add_alias(&self, alias: &str, index: &str) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
let mut entry = self.aliases.entry(alias.to_string()).or_default();
if !entry.contains(&index.to_string()) {
entry.push(index.to_string());
}
drop(entry);
self.flush_aliases();
+ Ok(())
}
- /// Remove an alias's association with an index.
- pub fn remove_alias(&self, alias: &str, index: &str) {
+ /// Remove an alias's association with an index. A blocked boot fails
+ /// before the map or alias file changes.
+ pub fn remove_alias(&self, alias: &str, index: &str) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
if let Some(mut entry) = self.aliases.get_mut(alias) {
entry.retain(|i| i != index);
}
// Clean up empty alias entries.
self.aliases.retain(|_, v| !v.is_empty());
self.flush_aliases();
+ Ok(())
}
/// Resolve a name: if it's an alias, return the aliased index names;
@@ -2518,6 +2464,7 @@ impl Engine {
/// 404 `index_not_found`, which left an operator with no lever but
/// stopping the server and removing the directory by hand (issue #206).
pub async fn delete_index(&self, name: &str) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
// Per-index authorization backstop (issue #79) — "destroy the brain"
// is the loudest door, so it is checked before the index is removed
// from the map. See `crate::index_guard`.
@@ -2605,7 +2552,9 @@ impl Engine {
let had_lifecycle_state = self.managed_indices.remove(name).is_some()
| self.lifecycle_detached.remove(name).is_some();
if had_lifecycle_state {
- self.persist_managed_indices();
+ // `delete_index` checked the compatibility fence before reaching
+ // this private cleanup path.
+ let _ = self.persist_managed_indices();
}
}
@@ -2684,6 +2633,7 @@ impl Engine {
/// new error is returned — the operator gets the *current* reason, not the
/// one from boot.
pub fn retry_failed_index(&self, name: &str) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
if !crate::index_guard::visible(name) || !self.failed_indices.contains_key(name) {
return Err(EngineError::Common(
xerj_common::XerjError::index_not_found(name),
@@ -2736,6 +2686,14 @@ impl Engine {
/// the concrete name, so pointing an alias at someone else's index cannot
/// launder access to it.
pub fn get_index(&self, name: &str) -> Result<Arc<Index>> {
+ // A blocked boot deliberately did not open any index or replay any
+ // WAL. Report that unavailable state instead of turning every real
+ // on-disk index into a misleading 404.
+ if !self.cluster_state_boot.is_writable() {
+ return Err(EngineError::Common(
+ xerj_common::XerjError::cluster_state_unavailable(),
+ ));
+ }
// Check if name is an alias — if so, resolve to the first backing index.
if let Some(aliased) = self.aliases.get(name) {
if let Some(real_name) = aliased.first() {
@@ -3018,7 +2976,7 @@ impl Engine {
let backing_name = format!(".ds-{}-000001", name);
self.create_index(&backing_name, Schema::empty())?;
// Alias: writing to the stream name → first backing index.
- self.add_alias(name, &backing_name);
+ self.add_alias(name, &backing_name)?;
let ds = DataStream {
name: name.to_string(),
backing_indices: vec![backing_name],
@@ -3039,7 +2997,13 @@ impl Engine {
/// exists (#282, ported from #262's `detach_data_stream_backing_index`)
/// — and persist the shrunken list, or a restart would reload the
/// deleted generation from `cluster_state.json` as an orphan.
- pub(crate) fn detach_data_stream_backing_index(&self, index: &str) {
+ pub(crate) fn detach_data_stream_backing_index(&self, index: &str) -> Result<()> {
+ // This helper is crate-visible because lifecycle deletion calls it
+ // after removing the physical backing index. Keep the compatibility
+ // fence here as well: a future caller must not be able to edit the
+ // live stream map merely because it bypassed the lifecycle entry
+ // point's guard.
+ self.ensure_cluster_state_writable()?;
let mut changed = false;
for mut entry in self.data_streams.iter_mut() {
let before = entry.backing_indices.len();
@@ -3047,10 +3011,9 @@ impl Engine {
changed |= entry.backing_indices.len() != before;
}
if changed {
- if let Err(e) = self.flush_cluster_state() {
- warn!(error = %e, "failed to persist data-stream state after lifecycle delete");
- }
+ self.flush_cluster_state()?;
}
+ Ok(())
}
/// Roll over a data stream: create the next backing index and update the alias.
@@ -3078,7 +3041,7 @@ impl Engine {
drop(entry);
self.flush_aliases();
} else {
- self.add_alias(name, &new_backing);
+ self.add_alias(name, &new_backing)?;
}
if let Some(mut ds) = self.data_streams.get_mut(name) {
@@ -3117,7 +3080,7 @@ impl Engine {
// `cluster_state.json` still describing the stream while some of its
// `.ds-*` directories are already gone. The next boot restores the
// stream and warns `restored data stream references backing indices
- // that are not present in the data dir` (see `load_cluster_state`),
+ // that are not present in the data dir` (see `apply_cluster_state`),
// `GET /_data_stream/<name>` still answers, and re-issuing the
// DELETE finishes the job.
//
@@ -3203,6 +3166,11 @@ impl Engine {
&self.config
}
+ /// Immutable boot classification used by readiness and diagnostics.
+ pub fn cluster_state_boot_status(&self) -> &ClusterStateBootStatus {
+ &self.cluster_state_boot
+ }
+
/// Return the effective embedding identity without exposing configuration
/// secrets or local asset paths.
pub fn embedding_execution_identity(&self) -> Result<EmbeddingExecutionIdentity> {
@@ -3356,6 +3324,7 @@ impl Engine {
name: &str,
indices: Option<Vec<String>>,
) -> Result<Value> {
+ self.ensure_cluster_state_writable()?;
let snap_dir = std::path::Path::new(repo_path).join(name);
let snap_dir = validate_snapshot_path(
repo_path,
@@ -3454,6 +3423,7 @@ impl Engine {
name: &str,
indices: Option<Vec<String>>,
) -> Result<Vec<String>> {
+ self.ensure_cluster_state_writable()?;
let snap_dir = std::path::Path::new(repo_path).join(name);
let snap_dir = validate_snapshot_path(
repo_path,
diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rs
index f6ceca711..90c614bb7 100644
--- a/engine/crates/xerj-engine/src/lib.rs
+++ b/engine/crates/xerj-engine/src/lib.rs
@@ -19,6 +19,7 @@
pub mod aggs;
pub mod audit;
pub mod bulk;
+mod cluster_state;
mod collection_publication;
pub mod engine;
pub mod governor;
@@ -43,6 +44,7 @@ mod bounded_ingest_resources_tests;
// ── Re-exports ────────────────────────────────────────────────────────────────
+pub use cluster_state::{ClusterStateBlockKind, ClusterStateBootStatus};
pub use engine::{EmbeddingExecutionIdentity, Engine, HealthStatus, IndexInfo};
pub use index::{
detect_log_format, resolve_date_math, resolve_field_alias, EnrichTable, FieldEncodingInfo,
diff --git a/engine/crates/xerj-engine/src/lifecycle.rs b/engine/crates/xerj-engine/src/lifecycle.rs
index 163aa013b..128d67be4 100644
--- a/engine/crates/xerj-engine/src/lifecycle.rs
+++ b/engine/crates/xerj-engine/src/lifecycle.rs
@@ -539,9 +539,13 @@ async fn execute_action(
format!("refusing lifecycle delete of '{index_name}': {reason}"),
)));
}
+ // Before deleting the index: detach persists cluster metadata, so
+ // a blocked boot must refuse before any backing directory or live
+ // map changes.
+ engine.ensure_cluster_state_writable()?;
engine.delete_index(index_name).await?;
engine.managed_indices.remove(index_name);
- engine.detach_data_stream_backing_index(index_name);
+ engine.detach_data_stream_backing_index(index_name)?;
Ok(ActionOutcome::IndexDeleted)
}
LifecycleAction::ReadOnly(_) => {
@@ -688,7 +692,9 @@ pub async fn tick(engine: &Engine) {
}
if any_change {
- engine.persist_managed_indices();
+ if let Err(error) = engine.persist_managed_indices() {
+ tracing::warn!(%error, "lifecycle state was not persisted");
+ }
}
}
@@ -723,6 +729,117 @@ pub fn explain_json(index_name: &str, managed: &ManagedIndexState) -> Value {
mod tests {
use super::*;
+ #[tokio::test]
+ async fn cluster_state_format1_lifecycle_delete_and_detach_guard_before_side_effects() {
+ fn storage_snapshot(root: &std::path::Path) -> Vec<(String, Vec<u8>)> {
+ fn walk(
+ root: &std::path::Path,
+ path: &std::path::Path,
+ out: &mut Vec<(String, Vec<u8>)>,
+ ) {
+ let mut entries: Vec<_> = std::fs::read_dir(path)
+ .unwrap()
+ .map(|entry| entry.unwrap())
+ .collect();
+ entries.sort_by_key(|entry| entry.file_name());
+ for entry in entries {
+ let path = entry.path();
+ let relative = path.strip_prefix(root).unwrap().to_string_lossy();
+ let kind = entry.file_type().unwrap();
+ if kind.is_dir() {
+ out.push((format!("d:{relative}"), Vec::new()));
+ walk(root, &path, out);
+ } else {
+ out.push((format!("f:{relative}"), std::fs::read(path).unwrap()));
+ }
+ }
+ }
+ let mut out = Vec::new();
+ walk(root, root, &mut out);
+ out
+ }
+
+ let dir = tempfile::tempdir().unwrap();
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ {
+ let engine = Engine::new(config.clone()).unwrap();
+ engine
+ .create_index("logs-1", xerj_common::types::Schema::empty())
+ .unwrap();
+ }
+
+ let future =
+ include_bytes!("../tests/fixtures/cluster_state/v2-format-version-minimal.json");
+ let state_path = dir.path().join("cluster_state.json");
+ std::fs::write(&state_path, future).unwrap();
+ let engine = Engine::new(config).unwrap();
+ assert!(matches!(
+ engine.get_index("logs-1"),
+ Err(EngineError::Common(
+ xerj_common::XerjError::ClusterStateUnavailable
+ ))
+ ));
+ engine.data_streams.insert(
+ "events".to_owned(),
+ crate::engine::DataStream {
+ name: "events".to_owned(),
+ backing_indices: vec!["logs-1".to_owned(), "write-current".to_owned()],
+ timestamp_field: "@timestamp".to_owned(),
+ generation: 2,
+ },
+ );
+ let before_files = storage_snapshot(dir.path());
+ let before_stream = engine.data_streams.get("events").unwrap().clone();
+
+ let direct = engine.detach_data_stream_backing_index("logs-1");
+ assert!(matches!(
+ direct,
+ Err(EngineError::Common(
+ xerj_common::XerjError::ClusterStateUnavailable
+ ))
+ ));
+ assert_eq!(
+ engine.data_streams.get("events").unwrap().backing_indices,
+ before_stream.backing_indices,
+ "the direct detach mutator must fail before editing the live stream"
+ );
+ assert_eq!(
+ storage_snapshot(dir.path()),
+ before_files,
+ "the direct detach mutator must fail before rewriting any file"
+ );
+
+ let result = execute_action(
+ &engine,
+ "logs-1",
+ &LifecycleAction::Delete(EmptyParams {}),
+ 1,
+ now_ms().saturating_sub(1),
+ )
+ .await;
+ assert!(matches!(
+ result,
+ Err(EngineError::Common(
+ xerj_common::XerjError::ClusterStateUnavailable
+ ))
+ ));
+ assert!(dir.path().join("logs-1").is_dir());
+ assert_eq!(
+ engine.data_streams.get("events").unwrap().backing_indices,
+ before_stream.backing_indices,
+ "the detach half must not edit the live stream"
+ );
+ assert_eq!(
+ storage_snapshot(dir.path()),
+ before_files,
+ "blocked lifecycle delete/detach must not rewrite any file"
+ );
+
+ assert_eq!(std::fs::read(state_path).unwrap(), future.as_slice());
+ }
+
#[test]
fn parse_duration_handles_all_units() {
assert_eq!(parse_duration_ms("500ms").unwrap(), 500);
diff --git a/engine/crates/xerj-engine/tests/cluster_state_compatibility.rs b/engine/crates/xerj-engine/tests/cluster_state_compatibility.rs
new file mode 100644
index 000000000..ee8c1d6bd
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/cluster_state_compatibility.rs
@@ -0,0 +1,542 @@
+use sha2::{Digest, Sha256};
+use xerj_common::config::{Config, WalSync};
+use xerj_common::types::Schema;
+use xerj_common::XerjError;
+use xerj_engine::{ClusterStateBlockKind, ClusterStateBootStatus, Engine};
+
+/// Byte-level storage snapshot. Directory and symlink topology is included,
+/// and every regular file is represented by its length and SHA-256. This is
+/// intentionally stronger than an index-count assertion: WAL replay can rotate
+/// one file into another while leaving both the document count and directory
+/// count unchanged.
+fn recursive_storage_snapshot(root: &std::path::Path) -> Vec<String> {
+ fn walk(root: &std::path::Path, path: &std::path::Path, out: &mut Vec<String>) {
+ let mut entries: Vec<_> = std::fs::read_dir(path)
+ .expect("read snapshot directory")
+ .map(|entry| entry.expect("read snapshot entry"))
+ .collect();
+ entries.sort_by_key(|entry| entry.file_name());
+ for entry in entries {
+ let entry_path = entry.path();
+ let relative = entry_path
+ .strip_prefix(root)
+ .expect("snapshot path under root")
+ .to_string_lossy();
+ let kind = entry.file_type().expect("snapshot file type");
+ if kind.is_dir() {
+ out.push(format!("d:{relative}"));
+ walk(root, &entry_path, out);
+ } else if kind.is_symlink() {
+ out.push(format!(
+ "l:{relative}->{}",
+ std::fs::read_link(&entry_path)
+ .expect("read snapshot symlink")
+ .to_string_lossy()
+ ));
+ } else {
+ let bytes = std::fs::read(&entry_path).expect("read snapshot file");
+ out.push(format!(
+ "f:{relative}:{}:{}",
+ bytes.len(),
+ hex_digest(&bytes)
+ ));
+ }
+ }
+ }
+
+ let mut out = Vec::new();
+ walk(root, root, &mut out);
+ out
+}
+
+fn config_for(path: &std::path::Path) -> Config {
+ let mut config = Config::default();
+ config.server.data_dir = path.to_string_lossy().into_owned();
+ config.storage.wal_sync = WalSync::Async;
+ config
+}
+
+fn fixture(name: &str) -> Vec<u8> {
+ std::fs::read(
+ std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
+ .join("tests/fixtures/cluster_state")
+ .join(name),
+ )
+ .expect("read fixture")
+}
+
+fn write_fixture(path: &std::path::Path, name: &str) -> Vec<u8> {
+ let bytes = fixture(name);
+ std::fs::write(path.join("cluster_state.json"), &bytes).expect("write fixture");
+ bytes
+}
+
+fn hex_digest(bytes: &[u8]) -> String {
+ Sha256::digest(bytes)
+ .iter()
+ .map(|byte| format!("{byte:02x}"))
+ .collect()
+}
+
+#[test]
+fn cluster_state_format1_fixture_manifest_is_exact() {
+ let manifest: serde_json::Value = serde_json::from_slice(&fixture("MANIFEST.json"))
+ .expect("parse cluster-state fixture manifest");
+ assert_eq!(manifest["schema"], "xerj.cluster-state-format1-fixtures.v1");
+ assert_eq!(
+ manifest["format1_writer_commit"],
+ "ee29361bccce77de527baac3fe6ed387cd1f6dff"
+ );
+ assert_eq!(
+ manifest["format1_writer_tree"],
+ "7227e51db69f9910a355bfd6b30b6a64b52dac87"
+ );
+ assert_eq!(
+ manifest["provenance"]["method"],
+ "runtime-capture-through-public-apis"
+ );
+ assert_eq!(
+ manifest["provenance"]["writer_source"],
+ "engine/crates/xerj-engine/src/engine.rs"
+ );
+ assert_eq!(manifest["provenance"]["reproduction"], "REPRODUCE.txt");
+ let reproduction = fixture("REPRODUCE.txt");
+ assert_eq!(
+ hex_digest(&reproduction),
+ manifest["provenance"]["reproduction_sha256"]
+ .as_str()
+ .unwrap()
+ );
+ assert!(reproduction
+ .windows("/_index_template/logs".len())
+ .any(|window| window == b"/_index_template/logs"));
+ for (name, expected) in manifest["fixtures"]
+ .as_object()
+ .expect("fixture manifest object")
+ {
+ let bytes = fixture(name);
+ assert_eq!(
+ bytes.len() as u64,
+ expected["bytes"].as_u64().unwrap(),
+ "{name}"
+ );
+ assert_eq!(
+ hex_digest(&bytes),
+ expected["sha256"].as_str().unwrap(),
+ "{name}"
+ );
+ }
+}
+
+#[tokio::test]
+async fn cluster_state_format1_valid_v1_is_stable_and_remains_v1_after_mutation() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let original = write_fixture(dir.path(), "v1-ee29361-all-fields.json");
+
+ {
+ let engine = Engine::new(config_for(dir.path())).expect("open exact v1");
+ assert_eq!(
+ engine.cluster_state_boot_status(),
+ &ClusterStateBootStatus::LoadedV1
+ );
+ assert_eq!(engine.templates.len(), 1);
+ assert_eq!(engine.legacy_templates.len(), 1);
+ assert_eq!(engine.component_templates.len(), 1);
+ assert_eq!(engine.pipelines.len(), 1);
+ assert_eq!(engine.data_streams.len(), 1);
+ assert_eq!(engine.ilm_policies.len(), 1);
+ }
+ assert_eq!(
+ std::fs::read(dir.path().join("cluster_state.json")).unwrap(),
+ original,
+ "a no-op restart must not normalize valid v1 bytes"
+ );
+
+ {
+ let engine = Engine::new(config_for(dir.path())).expect("reopen exact v1");
+ engine
+ .put_component_template(
+ "extra".to_owned(),
+ serde_json::json!({"template":{"settings":{"refresh_interval":"1s"}}}),
+ )
+ .expect("intentional supported mutation");
+ }
+ let written: serde_json::Value =
+ serde_json::from_slice(&std::fs::read(dir.path().join("cluster_state.json")).unwrap())
+ .unwrap();
+ assert_eq!(written["version"], 1);
+ assert_eq!(written.as_object().unwrap().len(), 7);
+ for field in [
+ "index_templates",
+ "legacy_templates",
+ "component_templates",
+ "pipelines",
+ "data_streams",
+ "ilm_policies",
+ ] {
+ assert!(written[field].is_object(), "missing v1 map {field}");
+ }
+ assert_eq!(written["component_templates"].as_object().unwrap().len(), 2);
+ assert_eq!(
+ written["index_templates"]["logs"],
+ serde_json::json!({
+ "index_patterns": ["logs-*"],
+ "settings": {"number_of_replicas": 0},
+ "mappings": {"properties": {"host": {"type": "keyword"}}},
+ "priority": 200
+ })
+ );
+ assert_eq!(
+ written["legacy_templates"]["legacy-logs"],
+ serde_json::json!({
+ "index_patterns": ["legacy-*"],
+ "settings": {"index.number_of_shards": "1"}
+ })
+ );
+ assert_eq!(
+ written["component_templates"]["base-settings"],
+ serde_json::json!({"template": {"settings": {"number_of_replicas": 1}}})
+ );
+ assert_eq!(
+ written["component_templates"]["extra"],
+ serde_json::json!({"template": {"settings": {"refresh_interval": "1s"}}})
+ );
+ assert_eq!(
+ written["pipelines"]["tagger"],
+ serde_json::json!({
+ "description": "tag every doc",
+ "stages": [{"type": "set", "config": {"field": "tagged", "value": "yes"}}]
+ })
+ );
+ assert_eq!(
+ written["data_streams"]["metrics-app"],
+ serde_json::json!({
+ "name": "metrics-app",
+ "backing_indices": [".ds-metrics-app-000001"],
+ "timestamp_field": "@timestamp",
+ "generation": 1
+ })
+ );
+ assert_eq!(
+ written["ilm_policies"]["hot-warm"],
+ serde_json::json!({"policy": {"phases": {"hot": {"actions": {}}}}})
+ );
+
+ let engine = Engine::new(config_for(dir.path())).expect("reopen mutated v1");
+ assert_eq!(engine.templates.len(), 1);
+ assert_eq!(engine.legacy_templates.len(), 1);
+ assert_eq!(engine.component_templates.len(), 2);
+ assert_eq!(engine.pipelines.len(), 1);
+ assert_eq!(engine.data_streams.len(), 1);
+ assert_eq!(engine.ilm_policies.len(), 1);
+ assert_eq!(
+ engine.templates.get("logs").unwrap().index_patterns,
+ vec!["logs-*".to_owned()]
+ );
+ assert_eq!(
+ engine
+ .data_streams
+ .get("metrics-app")
+ .unwrap()
+ .backing_indices,
+ vec![".ds-metrics-app-000001".to_owned()]
+ );
+}
+
+#[tokio::test]
+async fn cluster_state_format1_incompatible_fixtures_latch_and_preserve_bytes() {
+ let cases = [
+ (
+ "v1-unknown-top-level.json",
+ ClusterStateBlockKind::IncompatibleSchema,
+ ),
+ (
+ "v1-unknown-index-template-field.json",
+ ClusterStateBlockKind::IncompatibleSchema,
+ ),
+ (
+ "v1-unknown-data-stream-field.json",
+ ClusterStateBlockKind::IncompatibleSchema,
+ ),
+ (
+ "v1-duplicate-version.json",
+ ClusterStateBlockKind::DuplicateKey,
+ ),
+ (
+ "v1-duplicate-nested-value-key.json",
+ ClusterStateBlockKind::DuplicateKey,
+ ),
+ (
+ "v2-format-version-minimal.json",
+ ClusterStateBlockKind::UnsupportedFormat,
+ ),
+ ];
+
+ for (name, expected_kind) in cases {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let original = write_fixture(dir.path(), name);
+ let original_digest = hex_digest(&original);
+ for boot in 1..=2 {
+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine still boots");
+ assert_eq!(
+ engine.cluster_state_boot_status().block_kind(),
+ Some(&expected_kind),
+ "{name}, boot {boot}"
+ );
+ let err = engine
+ .put_component_template("must-not-land".to_owned(), serde_json::json!({}))
+ .expect_err("blocked state must refuse mutation");
+ let common: XerjError = err.into();
+ assert!(matches!(&common, XerjError::ClusterStateUnavailable));
+ drop(engine);
+
+ let after = std::fs::read(dir.path().join("cluster_state.json")).unwrap();
+ assert_eq!(after, original, "{name}, boot {boot}");
+ assert_eq!(hex_digest(&after), original_digest, "{name}, boot {boot}");
+ assert!(
+ !dir.path().join("cluster_state.corrupt.json").exists(),
+ "unsupported, unknown and duplicate-keyed bytes are not malformed salvage"
+ );
+ }
+ }
+}
+
+#[tokio::test]
+async fn cluster_state_format1_malformed_bytes_are_untouched_and_never_authorize_write() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let malformed = br#"{"version":1,"index_templates":{"cut":true"#;
+ std::fs::write(dir.path().join("cluster_state.json"), malformed).unwrap();
+ for boot in 1..=2 {
+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine still boots");
+ assert_eq!(
+ engine.cluster_state_boot_status().block_kind(),
+ Some(&ClusterStateBlockKind::MalformedJson),
+ "boot {boot}"
+ );
+ assert!(engine
+ .put_component_template("must-not-land".to_owned(), serde_json::json!({}))
+ .is_err());
+ drop(engine);
+ assert_eq!(
+ std::fs::read(dir.path().join("cluster_state.json")).unwrap(),
+ malformed,
+ "boot {boot}"
+ );
+ assert!(
+ !dir.path().join("cluster_state.corrupt.json").exists(),
+ "a blocked diagnostic boot must not create recovery files"
+ );
+ }
+}
+
+#[cfg(unix)]
+#[tokio::test]
+async fn cluster_state_format1_read_failure_is_not_mislabeled_or_salvaged() {
+ use std::os::unix::fs::symlink;
+
+ let dir = tempfile::tempdir().expect("tempdir");
+ let state_path = dir.path().join("cluster_state.json");
+ symlink("cluster_state.json", &state_path).expect("create self-loop");
+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine still boots");
+ assert_eq!(
+ engine.cluster_state_boot_status().block_kind(),
+ Some(&ClusterStateBlockKind::ReadFailure)
+ );
+ assert!(engine
+ .put_component_template("must-not-land".to_owned(), serde_json::json!({}))
+ .is_err());
+ drop(engine);
+ assert_eq!(
+ std::fs::read_link(&state_path).expect("read self-loop"),
+ std::path::PathBuf::from("cluster_state.json")
+ );
+ assert!(
+ !dir.path().join("cluster_state.corrupt.json").exists(),
+ "read failures are never parse-corruption salvage"
+ );
+}
+
+#[tokio::test]
+async fn cluster_state_format1_future_state_cannot_reconcile_and_downgrade_on_boot() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ {
+ let engine = Engine::new(config_for(dir.path())).expect("fresh engine");
+ for name in ["user-events", ".xerj_dashboards", ".ds-events-000002"] {
+ engine
+ .create_index(name, Schema::empty())
+ .unwrap_or_else(|err| panic!("create {name}: {err}"));
+ engine
+ .get_index(name)
+ .expect("new index is open")
+ .index_document(
+ Some("pending-wal".to_owned()),
+ serde_json::json!({"body": format!("unflushed {name}")}),
+ )
+ .await
+ .unwrap_or_else(|err| panic!("append {name} WAL: {err}"));
+ }
+ }
+ let original = write_fixture(dir.path(), "v2-format-version-minimal.json");
+ let before = recursive_storage_snapshot(dir.path());
+
+ for boot in 1..=2 {
+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine still boots");
+ assert_eq!(
+ engine.cluster_state_boot_status().block_kind(),
+ Some(&ClusterStateBlockKind::UnsupportedFormat)
+ );
+ assert!(engine.data_streams.is_empty());
+ assert!(engine.index_name_list().is_empty(), "boot {boot}");
+ let get_err: XerjError = match engine.get_index("user-events") {
+ Ok(_) => panic!("blocked boot must not open a user index"),
+ Err(err) => err.into(),
+ };
+ assert!(matches!(get_err, XerjError::ClusterStateUnavailable));
+ let create_err: XerjError = engine
+ .create_index("must-not-exist", Schema::empty())
+ .expect_err("blocked boot must not activate a new WAL")
+ .into();
+ assert!(matches!(create_err, XerjError::ClusterStateUnavailable));
+ engine.audit.append(
+ "blocked-diagnostic",
+ "test",
+ "cluster-state",
+ "refused",
+ "must remain memory-only",
+ );
+ drop(engine);
+
+ assert_eq!(
+ recursive_storage_snapshot(dir.path()),
+ before,
+ "blocked boot {boot} must not open, replay, rotate or flush any user/system index, WAL, segment, or durable audit file"
+ );
+ assert!(!dir.path().join("must-not-exist").exists());
+ }
+ assert_eq!(
+ std::fs::read(dir.path().join("cluster_state.json")).unwrap(),
+ original,
+ "startup index discovery must not downgrade future cluster state"
+ );
+}
+
+#[tokio::test]
+async fn cluster_state_format1_direct_durable_mutators_fail_before_any_side_effect() {
+ fn unavailable<T>(result: xerj_engine::Result<T>, operation: &str) {
+ let error = match result {
+ Ok(_) => panic!("{operation}"),
+ Err(error) => error,
+ };
+ let common: XerjError = error.into();
+ assert!(
+ matches!(common, XerjError::ClusterStateUnavailable),
+ "{operation}: {common}"
+ );
+ }
+
+ let dir = tempfile::tempdir().expect("tempdir");
+ let future = write_fixture(dir.path(), "v2-format-version-minimal.json");
+ let engine = Engine::new(config_for(dir.path())).expect("blocked engine");
+ let policy: xerj_engine::lifecycle::LifecyclePolicy =
+ serde_json::from_value(serde_json::json!({
+ "default_state": "hot",
+ "states": [{"name":"hot","actions":[],"transitions":[]}]
+ }))
+ .unwrap();
+ engine.index_mappings.insert(
+ "seed".to_owned(),
+ serde_json::json!({"properties":{"kept":{"type":"keyword"}}}),
+ );
+ engine.api_keys.insert(
+ "seed-key".to_owned(),
+ xerj_engine::engine::ApiKeyRecord::new("seed", "secret", 1, None, Vec::new()),
+ );
+ engine
+ .ism_policies
+ .insert("seed-policy".to_owned(), policy.clone());
+ engine.managed_indices.insert(
+ "seed".to_owned(),
+ xerj_engine::lifecycle::ManagedIndexState::new(
+ "seed-policy".to_owned(),
+ "hot".to_owned(),
+ 1,
+ ),
+ );
+ engine.index_settings.insert(
+ "seed".to_owned(),
+ serde_json::json!({"index":{"lifecycle":{"name":"seed-policy"}}}),
+ );
+ engine
+ .aliases
+ .insert("seed-alias".to_owned(), vec!["seed".to_owned()]);
+
+ let before_storage = recursive_storage_snapshot(dir.path());
+ unavailable(
+ engine.put_index_mapping("seed", serde_json::json!({"replaced":true})),
+ "mapping mutation must be fenced",
+ );
+ unavailable(
+ engine.persist_api_key(
+ "new-key".to_owned(),
+ xerj_engine::engine::ApiKeyRecord::new("new", "secret", 2, None, Vec::new()),
+ ),
+ "API-key insert must be fenced",
+ );
+ unavailable(
+ engine
+ .invalidate_api_keys(&["seed-key".to_owned()], 3)
+ .map(|_| ()),
+ "API-key invalidation must be fenced",
+ );
+ unavailable(
+ engine.put_ism_policy("new-policy".to_owned(), policy),
+ "ISM insert must be fenced",
+ );
+ unavailable(
+ engine.remove_ism_policy("seed-policy").map(|_| ()),
+ "ISM remove must be fenced",
+ );
+ unavailable(
+ engine.persist_managed_indices(),
+ "managed lifecycle persistence must be fenced",
+ );
+ unavailable(
+ engine.detach_lifecycle("seed").map(|_| ()),
+ "lifecycle detach must be fenced",
+ );
+ unavailable(
+ engine.add_alias("new-alias", "seed"),
+ "alias insert must be fenced",
+ );
+ unavailable(
+ engine.remove_alias("seed-alias", "seed"),
+ "alias remove must be fenced",
+ );
+
+ assert_eq!(engine.index_mappings.len(), 1);
+ assert_eq!(
+ engine.index_mappings.get("seed").unwrap().value(),
+ &serde_json::json!({"properties":{"kept":{"type":"keyword"}}})
+ );
+ assert_eq!(engine.api_keys.len(), 1);
+ assert!(!engine.api_keys.get("seed-key").unwrap().invalidated);
+ assert_eq!(engine.ism_policies.len(), 1);
+ assert!(engine.ism_policies.contains_key("seed-policy"));
+ assert_eq!(engine.managed_indices.len(), 1);
+ assert!(engine.managed_indices.contains_key("seed"));
+ assert!(engine.lifecycle_detached.is_empty());
+ assert_eq!(
+ engine.index_settings.get("seed").unwrap().value(),
+ &serde_json::json!({"index":{"lifecycle":{"name":"seed-policy"}}})
+ );
+ assert_eq!(engine.aliases.len(), 1);
+ assert_eq!(
+ engine.aliases.get("seed-alias").unwrap().value(),
+ &vec!["seed".to_owned()]
+ );
+ assert_eq!(recursive_storage_snapshot(dir.path()), before_storage);
+ assert_eq!(
+ std::fs::read(dir.path().join("cluster_state.json")).unwrap(),
+ future
+ );
+}
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/MANIFEST.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/MANIFEST.json
new file mode 100644
index 000000000..d0a675e11
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/MANIFEST.json
@@ -0,0 +1,23 @@
+{
+ "schema": "xerj.cluster-state-format1-fixtures.v1",
+ "format1_writer_commit": "ee29361bccce77de527baac3fe6ed387cd1f6dff",
+ "format1_writer_tree": "7227e51db69f9910a355bfd6b30b6a64b52dac87",
+ "provenance": {
+ "method": "runtime-capture-through-public-apis",
+ "writer_source": "engine/crates/xerj-engine/src/engine.rs",
+ "reproduction": "REPRODUCE.txt",
+ "reproduction_sha256": "84a08f9fd38809fe25343b4ee503641ea1a3d25b83601066d10f4e494da10022",
+ "writer_build": "git worktree add ../xerj-format1-writer ee29361bccce77de527baac3fe6ed387cd1f6dff && cd ../xerj-format1-writer/engine && cargo build --locked --profile profiling -j 32 -p xerj-server",
+ "capture_contract": "six documented ES-compatible PUTs, six matching GETs, graceful shutdown, then copy cluster_state.json",
+ "note": "The all-fields fixture is byte-identical to captured-cluster-state.json emitted after public PUT and GET of all six managed map types and a clean shutdown."
+ },
+ "fixtures": {
+ "v1-duplicate-nested-value-key.json": {"bytes": 202, "sha256": "e815f8872638de2ce5fce2520dcf9ab38869cb6aceedb5baecbbdcf5837b121c"},
+ "v1-duplicate-version.json": {"bytes": 145, "sha256": "6e5e671200fdda04567d4201c9abed40695f33c5f36adab4ee5de41f55093153"},
+ "v1-ee29361-all-fields.json": {"bytes": 1275, "sha256": "ae9f96a1dbf70cc711eadfdf8cd2e6701004b66f39c197f1e74cde556d64bb7b"},
+ "v1-unknown-data-stream-field.json": {"bytes": 259, "sha256": "dda0121ed37ff4a8cff4ac17280f07eeb932f552c78bacacaa0e4aa129391f0f"},
+ "v1-unknown-index-template-field.json": {"bytes": 224, "sha256": "93a057bae38183ddda637f3c6d23a63153f8e9e77ae5387134dca189c0394f7b"},
+ "v1-unknown-top-level.json": {"bytes": 155, "sha256": "d0f5b5e976899a6e4d46d648c001a9867e3cbf52f4e74834fe9bd083e09fd2c6"},
+ "v2-format-version-minimal.json": {"bytes": 176, "sha256": "28894ecec5edf98baa3988d365013f6ba13b7024c9e4292bc5fe56dc92d0b7d3"}
+ }
+}
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/REPRODUCE.txt b/engine/crates/xerj-engine/tests/fixtures/cluster_state/REPRODUCE.txt
new file mode 100644
index 000000000..1ffcaf241
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/REPRODUCE.txt
@@ -0,0 +1,41 @@
+Reproduce the format-1 all-fields fixture
+=========================================
+
+`v1-ee29361-all-fields.json` is output captured from the released format-1
+writer. It is not a hand-authored approximation and does not depend on a
+private benchmark directory.
+
+1. Put writer commit `ee29361bccce77de527baac3fe6ed387cd1f6dff` in
+ a disposable worktree (do not rewrite the tree being reviewed) and build
+ only the server:
+
+ git worktree add ../xerj-format1-writer ee29361bccce77de527baac3fe6ed387cd1f6dff
+ cd ../xerj-format1-writer/engine
+ CARGO_PROFILE_PROFILING_LTO=thin cargo build --locked --profile profiling -j 32 -p xerj-server
+
+2. Start that binary with an empty data directory, authentication disabled,
+ and an unused loopback ES-compatible port. In the commands below,
+ `XERJ_URL` is that listener, for example `http://127.0.0.1:19200`.
+
+3. Drive every format-1 map through its public write endpoint:
+
+ curl -fsS -XPUT "$XERJ_URL/_index_template/logs" -H 'content-type: application/json' --data-binary '{"index_patterns":["logs-*"],"template":{"settings":{"number_of_replicas":0},"mappings":{"properties":{"host":{"type":"keyword"}}}},"priority":200}'
+ curl -fsS -XPUT "$XERJ_URL/_template/legacy-logs" -H 'content-type: application/json' --data-binary '{"index_patterns":["legacy-*"],"settings":{"index.number_of_shards":"1"}}'
+ curl -fsS -XPUT "$XERJ_URL/_component_template/base-settings" -H 'content-type: application/json' --data-binary '{"template":{"settings":{"number_of_replicas":1}}}'
+ curl -fsS -XPUT "$XERJ_URL/_ingest/pipeline/tagger" -H 'content-type: application/json' --data-binary '{"description":"tag every doc","stages":[{"type":"set","config":{"field":"tagged","value":"yes"}}]}'
+ curl -fsS -XPUT "$XERJ_URL/_data_stream/metrics-app"
+ curl -fsS -XPUT "$XERJ_URL/_ilm/policy/hot-warm" -H 'content-type: application/json' --data-binary '{"policy":{"phases":{"hot":{"actions":{}}}}}'
+
+4. Read the same six resources before accepting the capture:
+
+ curl -fsS "$XERJ_URL/_index_template/logs"
+ curl -fsS "$XERJ_URL/_template/legacy-logs"
+ curl -fsS "$XERJ_URL/_component_template/base-settings"
+ curl -fsS "$XERJ_URL/_ingest/pipeline/tagger"
+ curl -fsS "$XERJ_URL/_data_stream/metrics-app"
+ curl -fsS "$XERJ_URL/_ilm/policy/hot-warm"
+
+5. Stop the server with `SIGTERM`, require exit status 0, and copy
+ `<data_dir>/cluster_state.json`. The result is 1,275 bytes with SHA-256
+ `ae9f96a1dbf70cc711eadfdf8cd2e6701004b66f39c197f1e74cde556d64bb7b`.
+ `MANIFEST.json` pins that digest and the digests of every negative fixture.
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-nested-value-key.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-nested-value-key.json
new file mode 100644
index 000000000..a1592b159
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-nested-value-key.json
@@ -0,0 +1 @@
+{"version":1,"index_templates":{},"legacy_templates":{"logs":{"settings":{"refresh_interval":"1s","refresh_interval":"5s"}}},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{}}
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-version.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-version.json
new file mode 100644
index 000000000..7357e8068
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-duplicate-version.json
@@ -0,0 +1 @@
+{"version":1,"version":2,"index_templates":{},"legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{}}
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-ee29361-all-fields.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-ee29361-all-fields.json
new file mode 100644
index 000000000..0ee98963f
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-ee29361-all-fields.json
@@ -0,0 +1,75 @@
+{
+ "version": 1,
+ "index_templates": {
+ "logs": {
+ "index_patterns": [
+ "logs-*"
+ ],
+ "settings": {
+ "number_of_replicas": 0
+ },
+ "mappings": {
+ "properties": {
+ "host": {
+ "type": "keyword"
+ }
+ }
+ },
+ "priority": 200
+ }
+ },
+ "legacy_templates": {
+ "legacy-logs": {
+ "index_patterns": [
+ "legacy-*"
+ ],
+ "settings": {
+ "index.number_of_shards": "1"
+ }
+ }
+ },
+ "component_templates": {
+ "base-settings": {
+ "template": {
+ "settings": {
+ "number_of_replicas": 1
+ }
+ }
+ }
+ },
+ "pipelines": {
+ "tagger": {
+ "description": "tag every doc",
+ "stages": [
+ {
+ "type": "set",
+ "config": {
+ "field": "tagged",
+ "value": "yes"
+ }
+ }
+ ]
+ }
+ },
+ "data_streams": {
+ "metrics-app": {
+ "name": "metrics-app",
+ "backing_indices": [
+ ".ds-metrics-app-000001"
+ ],
+ "timestamp_field": "@timestamp",
+ "generation": 1
+ }
+ },
+ "ilm_policies": {
+ "hot-warm": {
+ "policy": {
+ "phases": {
+ "hot": {
+ "actions": {}
+ }
+ }
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-data-stream-field.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-data-stream-field.json
new file mode 100644
index 000000000..631f202ff
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-data-stream-field.json
@@ -0,0 +1 @@
+{"version":1,"index_templates":{},"legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{"events":{"name":"events","backing_indices":[".ds-events-000001"],"timestamp_field":"@timestamp","generation":1,"future":true}},"ilm_policies":{}}
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-index-template-field.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-index-template-field.json
new file mode 100644
index 000000000..f38945d4f
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-index-template-field.json
@@ -0,0 +1 @@
+{"version":1,"index_templates":{"logs":{"index_patterns":["logs-*"],"settings":{},"mappings":{},"priority":0,"future":true}},"legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{}}
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-top-level.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-top-level.json
new file mode 100644
index 000000000..928628a25
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v1-unknown-top-level.json
@@ -0,0 +1 @@
+{"version":1,"index_templates":{},"legacy_templates":{},"component_templates":{},"pipelines":{},"data_streams":{},"ilm_policies":{},"future_authority":{}}
diff --git a/engine/crates/xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json
new file mode 100644
index 000000000..602aa626e
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/fixtures/cluster_state/v2-format-version-minimal.json
@@ -0,0 +1 @@
+{"format_version":2,"data_streams":{"events":{"name":"events","backing_indices":[".ds-events-000001"],"timestamp_field":"@timestamp","generation":1}},"corpus_publications":{}}
diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rs
index 208918097..962ab78e6 100644
--- a/engine/crates/xerj-engine/tests/integration.rs
+++ b/engine/crates/xerj-engine/tests/integration.rs
@@ -1716,7 +1716,7 @@ async fn test_alias_search() {
.unwrap();
// Add alias "my_alias" → "real_index".
- engine.add_alias("my_alias", "real_index");
+ engine.add_alias("my_alias", "real_index").unwrap();
// Search via alias should return the same results as searching via the real name.
let idx_via_alias = engine.get_index("my_alias").unwrap();
@@ -1734,7 +1734,7 @@ async fn test_alias_search() {
assert!(ids.contains(&"a2"));
// Remove alias and verify it no longer resolves.
- engine.remove_alias("my_alias", "real_index");
+ engine.remove_alias("my_alias", "real_index").unwrap();
let resolved = engine.resolve_alias("my_alias");
assert_eq!(
resolved,
diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.toml
index d0a756590..d19165ddb 100644
--- a/engine/crates/xerj-server/Cargo.toml
+++ b/engine/crates/xerj-server/Cargo.toml
@@ -95,3 +95,4 @@ protox = { workspace = true }
reqwest = { workspace = true }
tempfile = { workspace = true }
flate2 = "1"
+tokio-stream = "0.1"
diff --git a/engine/crates/xerj-server/src/grpc.rs b/engine/crates/xerj-server/src/grpc.rs
index 56b83e582..703b15736 100644
--- a/engine/crates/xerj-server/src/grpc.rs
+++ b/engine/crates/xerj-server/src/grpc.rs
@@ -373,6 +373,17 @@ impl tonic::service::Interceptor for GrpcAuth {
));
}
request.extensions_mut().insert(principal);
+ // Authentication deliberately runs first: an unauthenticated caller
+ // must not learn whether this node rejected persisted metadata. Once
+ // identity is established, reject every RPC centrally before tonic
+ // dispatches to a handler. Individual handlers used to translate the
+ // same blocked Engine error into not-found, invalid-argument, or a
+ // partial bulk response depending on the method.
+ if !self.state.engine.cluster_state_boot_status().is_writable() {
+ return Err(Status::unavailable(
+ xerj_common::XerjError::cluster_state_unavailable().to_string(),
+ ));
+ }
Ok(request)
}
}
@@ -454,6 +465,17 @@ mod tests {
panic!("gRPC client could not connect to {addr}");
}
+ fn with_authorization<T>(
+ message: T,
+ authorization: &tonic::metadata::MetadataValue<tonic::metadata::Ascii>,
+ ) -> tonic::Request<T> {
+ let mut request = tonic::Request::new(message);
+ request
+ .metadata_mut()
+ .insert("authorization", authorization.clone());
+ request
+ }
+
#[tokio::test]
async fn grpc_health_index_get_search_roundtrip() {
let dir = TempDir::new().unwrap();
@@ -666,6 +688,158 @@ mod tests {
let _ = server.await;
}
+ /// A rejected persisted-metadata format leaves the process available only
+ /// for HTTP liveness/readiness diagnostics. The gRPC surface has no split
+ /// liveness service, so every generated XerjSearch method is unavailable.
+ /// Authentication stays outside that fence: missing credentials remain
+ /// UNAUTHENTICATED and a valid credential reaches the UNAVAILABLE latch.
+ #[tokio::test]
+ async fn grpc_cluster_state_fence_covers_all_methods_after_authentication() {
+ use tonic::Code;
+
+ let dir = TempDir::new().unwrap();
+ std::fs::write(
+ dir.path().join("cluster_state.json"),
+ br#"{"format_version":2}"#,
+ )
+ .unwrap();
+ let admin = "grpc-blocked-admin";
+ let state = app_state_with_auth(&dir, admin);
+ assert!(!state.engine.cluster_state_boot_status().is_writable());
+
+ let port = free_port();
+ let addr: SocketAddr = format!("127.0.0.1:{port}").parse().unwrap();
+ let (tx, rx) = tokio::sync::oneshot::channel::<()>();
+ let server = tokio::spawn(serve_grpc(addr, state, async move {
+ let _ = rx.await;
+ }));
+ let url = format!("http://127.0.0.1:{port}");
+ let mut client = connect(&url).await;
+
+ macro_rules! assert_code {
+ ($future:expr, $code:expr, $method:literal) => {{
+ let status = $future
+ .await
+ .expect_err(concat!($method, " must be rejected"));
+ assert_eq!(status.code(), $code, "{}: {status:?}", $method);
+ }};
+ }
+
+ // Authentication is evaluated before the storage-format latch for
+ // every unary and streaming method.
+ assert_code!(
+ client.health(pb::HealthRequest {}),
+ Code::Unauthenticated,
+ "health"
+ );
+ assert_code!(
+ client.search(pb::SearchRequest {
+ index: "blocked".into(),
+ query_json: String::new(),
+ size: 0,
+ from: 0,
+ }),
+ Code::Unauthenticated,
+ "search"
+ );
+ assert_code!(
+ client.index(pb::IndexRequest {
+ index: "blocked".into(),
+ id: "1".into(),
+ source_json: "{}".into(),
+ }),
+ Code::Unauthenticated,
+ "index"
+ );
+ assert_code!(
+ client.bulk_index(tokio_stream::empty::<pb::IndexRequest>()),
+ Code::Unauthenticated,
+ "bulk_index"
+ );
+ assert_code!(
+ client.get_document(pb::GetRequest {
+ index: "blocked".into(),
+ id: "1".into(),
+ }),
+ Code::Unauthenticated,
+ "get_document"
+ );
+ assert_code!(
+ client.delete_document(pb::DeleteRequest {
+ index: "blocked".into(),
+ id: "1".into(),
+ }),
+ Code::Unauthenticated,
+ "delete_document"
+ );
+
+ let authorization: tonic::metadata::MetadataValue<tonic::metadata::Ascii> =
+ format!("ApiKey {admin}").parse().unwrap();
+ assert_code!(
+ client.health(with_authorization(pb::HealthRequest {}, &authorization)),
+ Code::Unavailable,
+ "authenticated health"
+ );
+ assert_code!(
+ client.search(with_authorization(
+ pb::SearchRequest {
+ index: "blocked".into(),
+ query_json: String::new(),
+ size: 0,
+ from: 0,
+ },
+ &authorization,
+ )),
+ Code::Unavailable,
+ "authenticated search"
+ );
+ assert_code!(
+ client.index(with_authorization(
+ pb::IndexRequest {
+ index: "blocked".into(),
+ id: "1".into(),
+ source_json: "{}".into(),
+ },
+ &authorization,
+ )),
+ Code::Unavailable,
+ "authenticated index"
+ );
+ let mut bulk = tonic::Request::new(tokio_stream::empty::<pb::IndexRequest>());
+ bulk.metadata_mut()
+ .insert("authorization", authorization.clone());
+ assert_code!(
+ client.bulk_index(bulk),
+ Code::Unavailable,
+ "authenticated bulk_index"
+ );
+ assert_code!(
+ client.get_document(with_authorization(
+ pb::GetRequest {
+ index: "blocked".into(),
+ id: "1".into(),
+ },
+ &authorization,
+ )),
+ Code::Unavailable,
+ "authenticated get_document"
+ );
+ assert_code!(
+ client.delete_document(with_authorization(
+ pb::DeleteRequest {
+ index: "blocked".into(),
+ id: "1".into(),
+ },
+ &authorization,
+ )),
+ Code::Unavailable,
+ "authenticated delete_document"
+ );
+
+ let _ = tx.send(());
+ let _ = server.await;
+ }
+
/// Standard-alphabet base64, so the test can build the `id:secret`
/// credential a minted key presents. `xerj_api`'s encoder is crate-private.
fn b64(input: &str) -> String {
@@ -698,10 +872,13 @@ mod tests {
/// Mint a key straight into the engine's store and return the
/// `authorization` metadata value a client would present.
fn mint_key(state: &AppState, id: &str, roles: Vec<xerj_engine::rbac::Role>) -> String {
- state.engine.persist_api_key(
- id.to_string(),
- xerj_engine::engine::ApiKeyRecord::new(id, "s3cret", 0, None, roles),
- );
+ state
+ .engine
+ .persist_api_key(
+ id.to_string(),
+ xerj_engine::engine::ApiKeyRecord::new(id, "s3cret", 0, None, roles),
+ )
+ .unwrap();
format!("ApiKey {}", b64(&format!("{id}:s3cret")))
}
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index d9785adf8..0f13c4e02 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -1866,6 +1866,7 @@ async fn async_main() -> Result<()> {
// 8. Engine (opens existing indices from disk)
let engine = Engine::new(cfg.clone()).context("initialise engine")?;
+ let storage_available = engine.cluster_state_boot_status().is_writable();
// This node's own cluster address, composed from the parsed bind IP.
// Needed twice — by the transport (8b) and by the console's node identity
@@ -1874,7 +1875,7 @@ async fn async_main() -> Result<()> {
let cluster_listen_addr = cfg.socket_addr(cfg.cluster.port);
// 8b. Cluster runner (if cluster mode is enabled)
- let _cluster_shutdown = if cfg.cluster.enabled {
+ let _cluster_shutdown = if cfg.cluster.enabled && storage_available {
// Fail closed (issue #75). The cluster port carries Raft control
// messages; without a shared secret every frame on it is
// unauthenticated, so refuse to start rather than open it. This must
@@ -1945,6 +1946,11 @@ async fn async_main() -> Result<()> {
}
Some(shutdown_tx)
+ } else if cfg.cluster.enabled {
+ warn!(
+ "cluster transport is disabled while cluster-state storage is unavailable; restart with supported bytes before rejoining peers"
+ );
+ None
} else {
info!("Cluster mode disabled — running in single-node mode");
None
@@ -1954,6 +1960,13 @@ async fn async_main() -> Result<()> {
// boot, persists a 32-byte master key under data_dir/.xerj_master_key
// (mode 0600), and prints the first-launch magic-link banner to
// stderr if no active user exists yet. Idempotent on reboot.
+ //
+ // This is storage activation, not cosmetic UI setup. When the engine's
+ // cluster-state compatibility fence is blocked, boot must remain live
+ // only for diagnostics: opening and seeding Console indices can replay
+ // their WAL and flush a new segment. The Console surface is therefore
+ // absent until a supported restart instead of mounting with a made-up
+ // key or partially-initialized stores.
let xerj_console_bind_url = format!(
"http://{}:{}",
if cfg.server.bind_address == "0.0.0.0" || cfg.server.bind_address == "::" {
@@ -1966,14 +1979,6 @@ async fn async_main() -> Result<()> {
},
cfg.server.es_compat_port,
);
- let xerj_console_outcome = xerj_console_api::bootstrap::run(
- &engine,
- Path::new(&cfg.server.data_dir),
- &xerj_console_bind_url,
- )
- .await
- .context("xerj-console bootstrap")?;
-
let xerj_console_node_id: String = match (cfg.cluster.enabled, cluster_listen_addr) {
// The same string the cluster transport registered as this node's id.
(true, Some(addr)) => addr.to_string(),
@@ -2005,13 +2010,29 @@ async fn async_main() -> Result<()> {
}
);
}
- let xerj_console_state = ConsoleState::new(
- engine.clone(),
- xerj_console_node_id,
- xerj_console_outcome.master_key,
- xerj_console_cluster_mode,
- )
- .with_trusted_proxies(trusted_proxies);
+ let xerj_console_state = if storage_available {
+ let outcome = xerj_console_api::bootstrap::run(
+ &engine,
+ Path::new(&cfg.server.data_dir),
+ &xerj_console_bind_url,
+ )
+ .await
+ .context("xerj-console bootstrap")?;
+ Some(
+ ConsoleState::new(
+ engine.clone(),
+ xerj_console_node_id,
+ outcome.master_key,
+ xerj_console_cluster_mode,
+ )
+ .with_trusted_proxies(trusted_proxies),
+ )
+ } else {
+ warn!(
+ "Xerj Console bootstrap and routes are disabled while cluster-state storage is unavailable"
+ );
+ None
+ };
// 9b. Application state
let state = AppState::new(cfg.clone(), engine, metrics);
@@ -2025,7 +2046,9 @@ async fn async_main() -> Result<()> {
// / wal_size_bytes / memory_usage live (otherwise a flat zero even at
// millions of docs, hiding the RSS-runaway and WAL-growth signals).
xerj_engine::set_engine_metrics(state.metrics.clone());
- tokio::spawn(xerj_api::es_compat::run_metrics_gauge_loop(state.clone()));
+ if storage_available {
+ tokio::spawn(xerj_api::es_compat::run_metrics_gauge_loop(state.clone()));
+ }
// 9c. Routers — engine and Xerj Console are *peer* surfaces. Each crate
// builds a complete Router (routes + its own auth + its own
@@ -2035,9 +2058,13 @@ async fn async_main() -> Result<()> {
// /_xerj-console/api/v1/* routes. Yanking xerj-console-api out of this
// merge would leave xerj-api compiling and serving on its own
// — a property worth preserving.
- let xerj_console_router = xerj_console_api::xerj_console_router(xerj_console_state);
- let native_router = build_native_router(state.clone()).merge(xerj_console_router.clone());
- let es_router = build_es_compat_router(state.clone()).merge(xerj_console_router);
+ let mut native_router = build_native_router(state.clone());
+ let mut es_router = build_es_compat_router(state.clone());
+ if let Some(console_state) = xerj_console_state {
+ let xerj_console_router = xerj_console_api::xerj_console_router(console_state);
+ native_router = native_router.merge(xerj_console_router.clone());
+ es_router = es_router.merge(xerj_console_router);
+ }
// 10. Banner (includes total startup time)
let startup_ms = startup_start.elapsed().as_millis();
@@ -2065,24 +2092,30 @@ async fn async_main() -> Result<()> {
.with_context(|| bad_bind_address(&cfg))?;
// 12. Background flush timer
- let flusher = spawn_periodic_flusher(
- state.engine.clone(),
- std::time::Duration::from_secs(cfg.storage.flush_interval_secs),
- );
+ let flusher = storage_available.then(|| {
+ spawn_periodic_flusher(
+ state.engine.clone(),
+ std::time::Duration::from_secs(cfg.storage.flush_interval_secs),
+ )
+ });
// 12b. Resource governor sampler (RC4 W3 items 1 & 3): refreshes the
// process-wide memtable / RSS / disk-usage atomics that drive the
// parent circuit breaker and the disk flood-stage write block. This
// is the structural guard against the 112 GiB OOM class — writes get
// a 429 circuit_breaking_exception before the kernel OOM-kills us.
- state.engine.spawn_resource_sampler();
+ if storage_available {
+ state.engine.spawn_resource_sampler();
+ }
// 12c. ISM/ILM lifecycle-management background job: drives every
// managed index's state machine (actions, then transitions) on
// `lifecycle.tick_interval_secs` (default 5 minutes).
- state.engine.spawn_lifecycle_manager();
+ if storage_available {
+ state.engine.spawn_lifecycle_manager();
+ }
- let _ingest_memory_trace = ingest_memory_trace::spawn(&state.engine);
+ let _ingest_memory_trace = storage_available.then(|| ingest_memory_trace::spawn(&state.engine));
// 13. Start servers concurrently
let rest_tls = tls_config.clone();
let rest = tokio::spawn(async move {
@@ -2122,7 +2155,9 @@ async fn async_main() -> Result<()> {
// boot, flush-permit exhaustion on the last write) sat in memory —
// pinning its WAL generations on disk — until shutdown or the next
// explicit `_flush`.
- flusher.abort();
+ if let Some(flusher) = flusher {
+ flusher.abort();
+ }
// 15. Final synchronous flush across every index.
//
@@ -2135,10 +2170,12 @@ async fn async_main() -> Result<()> {
// sessions) loses 100 % of its data on restart because startup index
// discovery looks for segment-bearing directories first. See POV
// report 2026-04-24T23-58-00 (B-2) for the full failure mode.
- info!("flushing in-memory state before exit…");
- let flush_started = std::time::Instant::now();
- state.engine.flush_all_force().await;
- info!("final flush complete in {:.0?}", flush_started.elapsed());
+ if storage_available {
+ info!("flushing in-memory state before exit…");
+ let flush_started = std::time::Instant::now();
+ state.engine.flush_all_force().await;
+ info!("final flush complete in {:.0?}", flush_started.elapsed());
+ }
info!("xerj v{} stopped. Goodbye.", env!("CARGO_PKG_VERSION"));
Ok(())