AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(cluster): the REST layer stops fabricating topology — real membership, honest waits, load-time config checks
xerj-orgf06de7cbbe73
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 997d6ea8c..121e5b2e8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Fixed++- **Cluster endpoints fabricated topology** (issue+ [#1169](https://github.com/xerj-org/xerj/issues/1169)) —+ `GET /_cluster/health?wait_for_nodes=N` echoed `N` back as+ `number_of_nodes` with `status: green` and `timed_out: false`, so the+ multinode smoke suite "converged" on a single node that had no cluster+ at all (the handler comment said so out loud). Health now reports the+ real configured membership (`1` when cluster mode is off, else the+ distinct own + peer addresses), an unsatisfiable `wait_for_nodes` times+ out honestly (408, `timed_out: true`, count stays real), and upper+ bounds (`<=N`, `<N`, `le(N)`) are enforced instead of collapsed to+ "always satisfied". `_cat/nodes` lists every configured member (peers+ with `-` placeholders, no invented metrics), `_nodes` totals the+ membership while `successful: 1` reflects that only self answered, and+ `_cluster/state` reports this node's actual cluster address instead of+ the never-bound 9300 default and `master_node: null` on a configured+ ring rather than every node claiming mastership. The multinode smoke+ case now **skips with a recorded reason** instead of passing vacuously;+ `cluster.health/20_request_timeout` passes through the real timeout+ path. Cluster config that could only fail silently is refused at load+ (issue [#1171](https://github.com/xerj-org/xerj/issues/1171)): a peer+ label differing from its address (which made the node lose every+ election with no diagnostic of its own) and `cluster.port` colliding+ with this node's own rest/grpc/es_compat port (which degraded the node+ to single-node with green health).+## [1.0.0-rc.83] - 2026-10-06The corpus-hub hundred release. The engine window is small — one retrievaldiff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 0e161d572..207486930 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -219,6 +219,56 @@ fn unassigned_info_json(f: &xerj_engine::engine::FailedIndex) -> Value {info}+/// A parsed `wait_for_nodes` constraint. Upper bounds were previously+/// collapsed onto "required minimum 1" — always satisfied — which could not+/// express `wait_for_nodes<=1` against a larger cluster (#1169).+enum NodesWait {+ AtLeast(u32),+ AtMost(u32),+}++/// The declared cluster topology as this node knows it from config (#1169):+/// this node's own address (`server.bind_address` + `cluster.port`) followed+/// by every distinct peer address. Raft node ids in this cluster ARE socket+/// addresses (the server boot derives them that way), so the returned ids+/// are the addresses.+///+/// This is the configured floor, not live state: reachability and leader+/// identity arrive with the ring-to-engine wiring tracked in #1170, and a+/// node whose cluster transport failed to bind (degraded single-node) still+/// reports its configured peers until that failure is surfaced in health+/// (#1171). It is nonetheless the honest answer to "how many nodes does+/// this cluster declare" — and it is never a number echoed back out of the+/// request, which is what it replaces.+fn configured_members(config: &xerj_common::config::Config) -> Vec<(String, std::net::SocketAddr)> {+ let mut out: Vec<(String, std::net::SocketAddr)> = Vec::new();+ let mut seen = std::collections::HashSet::new();+ if let Some(own) = config.socket_addr(config.cluster.port) {+ seen.insert(own);+ out.push((own.to_string(), own));+ }+ for peer in &config.cluster.peers {+ if let Some((_, addr)) = peer.split_once('=') {+ if let Ok(addr) = addr.trim().parse::<std::net::SocketAddr>() {+ if seen.insert(addr) {+ out.push((addr.to_string(), addr));+ }+ }+ }+ }+ out+}++/// Node count for `/_cluster/health`: `1` when cluster mode is off, else the+/// distinct configured membership (own address + peers, so a peer list that+/// also names this node is not double-counted).+fn configured_node_count(config: &xerj_common::config::Config) -> u32 {+ if !config.cluster.enabled {+ return 1;+ }+ configured_members(config).len() as u32+}+async fn cluster_health_inner(state: AppState,index_filter: Option<String>,@@ -391,33 +441,47 @@ async fn cluster_health_inner(let active: u32 = selected.iter().map(|info| shard_count(&info.name)).sum();let _ = closed_count;- // Wait-for assertions: single-node cluster, nothing relocating /- // initializing, primaries always active. Most wait_for_* are- // satisfied immediately. The exceptions:- // wait_for_nodes: N (where N > 1) — timeout, we only have 1 node+ // Wait-for assertions: nothing relocating / initializing, primaries+ // always active, so most wait_for_* are satisfied immediately. The+ // exceptions:+ // wait_for_nodes — checked against the real configured membership+ // below (#1169)// wait_for_active_shards: all / N — if the cluster is yellow// (unassigned replicas), those `all` conditions never converge.let wait_for_nodes = params.wait_for_nodes.as_deref().map(|s| {- // Supports `N`, `>=N`, `>N`, `<=N`, `<N`. Returns the required- // minimum integer nodes.+ // Supports `N` (treated as >= N, ES's own reading of a bare+ // count), `>=N`, `>N`, `<=N`, `<N`, `ge(N)`, `le(N)`. Upper+ // bounds used to be collapsed onto "minimum 1" — i.e. always+ // satisfied — which could not express `wait_for_nodes<=1`+ // against a larger cluster (#1169).let t = s.trim();if let Some(n) = t.strip_prefix(">=") {- n.trim().parse::<u32>().ok()- } else if let Some(n) = t.strip_prefix("<=") {- n.trim().parse::<u32>().ok().map(|v| v.min(1))+ n.trim().parse::<u32>().ok().map(NodesWait::AtLeast)} else if let Some(n) = t.strip_prefix('>') {- n.trim().parse::<u32>().ok().map(|v| v.saturating_add(1))+ n.trim()+ .parse::<u32>()+ .ok()+ .map(|v| NodesWait::AtLeast(v.saturating_add(1)))+ } else if let Some(n) = t.strip_prefix("<=") {+ n.trim().parse::<u32>().ok().map(NodesWait::AtMost)} else if let Some(n) = t.strip_prefix('<') {- n.trim().parse::<u32>().ok().map(|_| 1)+ n.trim()+ .parse::<u32>()+ .ok()+ .map(|v| NodesWait::AtMost(v.saturating_sub(1)))+ } else if let Some(n) = t.strip_prefix("ge(").and_then(|r| r.strip_suffix(')')) {+ n.trim().parse::<u32>().ok().map(NodesWait::AtLeast)+ } else if let Some(n) = t.strip_prefix("le(").and_then(|r| r.strip_suffix(')')) {+ n.trim().parse::<u32>().ok().map(NodesWait::AtMost)} else {- t.parse::<u32>().ok()+ t.parse::<u32>().ok().map(NodesWait::AtLeast)}})- .unwrap_or(Some(1))- .unwrap_or(1);+ .unwrap_or(Some(NodesWait::AtLeast(1)))+ .unwrap_or(NodesWait::AtLeast(1));// wait_for_active_shards unmet: `all` when any shard is unassigned —// a replica, or (now that a failed index is a reachable state, issue// #206) an unopenable primary, which is the more serious of the two and@@ -467,43 +531,27 @@ async fn cluster_health_inner(.as_deref(),Some("green") | Some("yellow"));- // When the caller explicitly requests `wait_for_nodes>=N`, we- // satisfy it by reporting `N` as the declared cluster size so- // the multinode smoke suite converges. But if the caller also- // passes an aggressive `timeout` (≤ 100ms) they're explicitly- // testing the timeout path — report the original "single-node,- // didn't converge" behaviour in that case.- let aggressive_timeout = params- .timeout- .as_deref()- .map(|s| {- let s = s.trim();- if let Some(ms) = s.strip_suffix("ms") {- ms.parse::<u64>().ok().map(|v| v <= 100).unwrap_or(false)- } else {- s == "0"- }- })- .unwrap_or(false);- let declared_nodes: u32 = if aggressive_timeout {- 1- } else {- params- .wait_for_nodes- .as_deref()- .map(|_| wait_for_nodes)- .unwrap_or(1)- };- let timed_out = wait_for_active_shards_unmet- || wait_for_status_unmet- || (aggressive_timeout && wait_for_nodes > 1);+ // `wait_for_nodes` honored against the real configured membership+ // (#1169): a condition this cluster cannot satisfy (asking 2 nodes of+ // a 1-node cluster) times out immediately — `timed_out: true`, mapped+ // to 408 below, exactly like an unmet `wait_for_status`. The previous+ // behaviour reported the requested count back as `number_of_nodes` so+ // the multinode smoke suite would "converge" on a single node; that+ // suite now skips with a recorded reason instead of passing against a+ // fabricated topology (yaml/smoke/smoke_test_multinode/10_basic.yml).+ let nodes = configured_node_count(&state.config);+ let wait_for_nodes_unmet = match wait_for_nodes {+ NodesWait::AtLeast(n) => nodes < n,+ NodesWait::AtMost(n) => nodes > n,+ };+ let timed_out = wait_for_active_shards_unmet || wait_for_status_unmet || wait_for_nodes_unmet;let mut resp = json!({"cluster_name": "xerj","status": status,"timed_out": timed_out,- "number_of_nodes": declared_nodes,- "number_of_data_nodes": declared_nodes,+ "number_of_nodes": nodes,+ "number_of_data_nodes": nodes,"active_primary_shards": active,"active_shards": active,"relocating_shards": 0,@@ -20976,9 +21024,28 @@ pub async fn cat_nodes(let (l1, l5, l15) = read_loadavg();let name = state.engine.node_id.as_str();+ // Honest membership rows (#1169): one row per configured member. The+ // self row carries live metrics; peer rows — processes we cannot see —+ // carry "-" placeholders, exactly as ES renders values it does not know.+ // `master` is "*" only on a cluster-disabled node (where self is+ // trivially the whole cluster); on a configured ring the leader is not+ // known to this layer yet (see #1170), and stamping "*" on self would+ // be the same fabrication this endpoint is recovering from.+ let members = configured_members(&state.config);+ let own_addr = members+ .first()+ .map(|(_, a)| *a)+ .unwrap_or(std::net::SocketAddr::from(([127, 0, 0, 1], 9300)));+ let master_col = if state.config.cluster.enabled {+ "-"+ } else {+ "*"+ };+if params.format.as_deref() == Some("json") {- return Json(vec![json!({- "ip": "127.0.0.1",+ let mut rows: Vec<Value> = Vec::with_capacity(members.len());+ rows.push(json!({+ "ip": own_addr.ip().to_string(),"heap.percent": heap_percent.to_string(),"ram.percent": ram_percent.to_string(),"cpu": cpu.to_string(),@@ -20986,15 +21053,33 @@ pub async fn cat_nodes("load_5m": format!("{l5:.2}"),"load_15m": format!("{l15:.2}"),"node.role": "cdfhilmrstw",- "master": "*",+ "master": master_col,"name": name,- })])- .into_response();+ }));+ for (peer_id, addr) in members.iter().skip(1) {+ rows.push(json!({+ "ip": addr.ip().to_string(),+ "heap.percent": "-",+ "ram.percent": "-",+ "cpu": "-",+ "load_1m": "-",+ "load_5m": "-",+ "load_15m": "-",+ "node.role": "-",+ "master": "-",+ "name": peer_id,+ }));+ }+ return Json(rows).into_response();}- let body = format!(- "127.0.0.1 {heap_percent} {ram_percent} {cpu} {l1:.2} {l5:.2} {l15:.2} cdfhilmrstw * {name}\n"+ let mut body = format!(+ "{ip} {heap_percent} {ram_percent} {cpu} {l1:.2} {l5:.2} {l15:.2} cdfhilmrstw {master_col} {name}\n",+ ip = own_addr.ip(),);+ for (peer_id, addr) in members.iter().skip(1) {+ body.push_str(&format!("{ip} - - - - - - - - {peer_id}\n", ip = addr.ip()));+ }(StatusCode::OK,[(@@ -31731,20 +31816,44 @@ pub async fn cluster_state(State(state): State<AppState>) -> impl IntoResponse {unassigned_shards.push(shard);}+ // Honest topology (#1169): the self entry's transport address is this+ // node's real configured cluster address (previously the never-bound+ // default 9300), and configured peers appear as minimal entries. On a+ // configured ring `master_node` is null rather than self: the REST+ // layer does not know the elected leader (#1170), and claiming+ // mastership from every node is the same fabrication this fixes.+ let members = configured_members(&state.config);+ let own_addr = members+ .first()+ .map(|(_, a)| a.to_string())+ .unwrap_or_else(|| "127.0.0.1:9300".to_string());+ let mut state_nodes = serde_json::Map::new();+ state_nodes.insert(+ node_id.to_string(),+ json!({+ "name": node_id,+ "transport_address": own_addr,+ "roles": ["master", "data", "ingest"],+ }),+ );+ for (peer_id, addr) in members.iter().skip(1) {+ state_nodes.insert(+ peer_id.clone(),+ json!({+ "name": peer_id,+ "transport_address": addr.to_string(),+ }),+ );+ }+Json(json!({"cluster_name": "xerj","cluster_uuid": "xerj-cluster-1","version": 1,"state_uuid": uuid::Uuid::new_v4().to_string(),- "master_node": node_id,+ "master_node": if state.config.cluster.enabled { Value::Null } else { json!(node_id) },"blocks": {},- "nodes": {- node_id: {- "name": node_id,- "transport_address": "127.0.0.1:9300",- "roles": ["master", "data", "ingest"],- }- },+ "nodes": Value::Object(state_nodes),"metadata": {"cluster_uuid": "xerj-cluster-1","templates": {},@@ -33080,15 +33189,30 @@ pub async fn nodes_info(.as_millis() as u64;let node_id = state.engine.node_id.as_str();- Json(json!({- "_nodes": { "total": 1, "successful": 1, "failed": 0 },- "cluster_name": "xerj",- "nodes": {- node_id: {+ // Honest membership (#1169): `total` is the configured cluster size;+ // `successful` stays 1 because only THIS process's info is known —+ // peers are listed with name/transport only, never invented versions+ // or metrics. The transport fields report this node's real configured+ // cluster address instead of a phantom 9300.+ let members = configured_members(&state.config);+ let node_total = members.len().max(1);+ let own_addr = members+ .first()+ .map(|(_, a)| a.to_string())+ .unwrap_or_else(|| "127.0.0.1:9300".to_string());+ let own_ip = members+ .first()+ .map(|(_, a)| a.ip().to_string())+ .unwrap_or_else(|| "127.0.0.1".to_string());++ let mut nodes_map = serde_json::Map::new();+ nodes_map.insert(+ node_id.to_string(),+ json!({"name": node_id,- "transport_address": "127.0.0.1:9300",- "host": "127.0.0.1",- "ip": "127.0.0.1",+ "transport_address": own_addr.clone(),+ "host": own_ip.clone(),+ "ip": own_ip,"version": reported_version,"build_flavor": "default","build_type": "tar",@@ -33149,8 +33273,8 @@ pub async fn nodes_info("write": { "type": "fixed", "size": num_cpus, "queue_size": 200 },},"transport": {- "bound_address": ["127.0.0.1:9300"],- "publish_address": "127.0.0.1:9300",+ "bound_address": [own_addr.clone()],+ "publish_address": own_addr,"profiles": {},},"http": {@@ -33164,8 +33288,28 @@ pub async fn nodes_info("docs": { "count": total_docs, "deleted": 0 },"store": { "size_in_bytes": store_bytes },},- }- }+ }),+ );+ // Configured peers: name and transport address are known from config;+ // everything else about a remote process would be invented, so it is+ // omitted (#1169). On a cluster-disabled node the peer list is empty+ // and this response is byte-equivalent to the old single-entry one.+ for (peer_id, addr) in members.iter().skip(1) {+ nodes_map.insert(+ peer_id.clone(),+ json!({+ "name": peer_id,+ "transport_address": addr.to_string(),+ "host": addr.ip().to_string(),+ "ip": addr.ip().to_string(),+ }),+ );+ }++ Json(json!({+ "_nodes": { "total": node_total, "successful": 1, "failed": 0 },+ "cluster_name": "xerj",+ "nodes": Value::Object(nodes_map),})).into_response()}@@ -45117,3 +45261,203 @@ mod script_guard_symmetry_tests {);}}++#[cfg(test)]+mod cluster_topology_honesty_tests {+ use super::*;+ use axum::{body::Body, http::Request};+ use tower::ServiceExt;++ /// Issue #1169: the cluster endpoints used to fabricate topology —+ /// `wait_for_nodes` was echoed back as `number_of_nodes`, `_cat/nodes`+ /// was a one-row stub, `_nodes` reported total 1, and `_cluster/state`+ /// named a transport port nothing listens on. These tests pin the honest+ /// behaviour: configured membership, real wait semantics, and no+ /// invented peer details.+ fn state_with_cluster(enabled: bool, peers: &[&str], cluster_port: u16) -> AppState {+ let dir = tempfile::tempdir().expect("tempdir").keep();+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ config.server.bind_address = "127.0.0.1".to_string();+ if enabled {+ config.cluster.enabled = true;+ config.cluster.port = cluster_port;+ config.cluster.auth_secret = "test-secret-0123456789abcdef".to_string();+ config.cluster.peers = peers.iter().map(|s| s.to_string()).collect();+ }+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ AppState::new(config, engine, metrics)+ }++ async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {+ let response = app+ .clone()+ .oneshot(Request::get(path).body(Body::empty()).unwrap())+ .await+ .unwrap();+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .unwrap();+ (status, serde_json::from_slice(&bytes).unwrap())+ }++ fn ring3() -> AppState {+ state_with_cluster(+ true,+ &[+ "127.0.0.1:19301=127.0.0.1:19301",+ "127.0.0.1:19302=127.0.0.1:19302",+ ],+ 19300,+ )+ }++ #[tokio::test]+ async fn single_node_health_reports_one_and_times_out_unsatisfiable_waits() {+ let state = state_with_cluster(false, &[], 9300);+ let app = crate::router::build_es_compat_router(state);++ let (status, body) = get(&app, "/_cluster/health").await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(body["number_of_nodes"], 1);+ assert_eq!(body["timed_out"], false);++ // Asking 2 nodes of a 1-node cluster: 408 + timed_out, and the node+ // count stays the real 1 — never the requested 2 back (#1169).+ let (status, body) = get(&app, "/_cluster/health?wait_for_nodes=2").await;+ assert_eq!(status, StatusCode::REQUEST_TIMEOUT);+ assert_eq!(body["timed_out"], true);+ assert_eq!(body["number_of_nodes"], 1, "must not echo the request");++ let (status, body) = get(&app, "/_cluster/health?wait_for_nodes=1").await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(body["timed_out"], false);+ }++ #[tokio::test]+ async fn ring_health_reports_configured_membership_with_real_wait_semantics() {+ let state = ring3();+ let app = crate::router::build_es_compat_router(state);++ let (status, body) = get(&app, "/_cluster/health").await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(body["number_of_nodes"], 3);+ assert_eq!(body["number_of_data_nodes"], 3);++ let (status, _) = get(&app, "/_cluster/health?wait_for_nodes=3").await;+ assert_eq!(status, StatusCode::OK);+ let (status, _) = get(&app, "/_cluster/health?wait_for_nodes=4").await;+ assert_eq!(status, StatusCode::REQUEST_TIMEOUT);++ // Upper bounds are real constraints now: `<=3` holds, `<=2` does not+ // (previously every upper bound collapsed to "minimum 1" and passed).+ let (status, _) = get(&app, "/_cluster/health?wait_for_nodes=le(3)").await;+ assert_eq!(status, StatusCode::OK);+ let (status, body) = get(&app, "/_cluster/health?wait_for_nodes=le(2)").await;+ assert_eq!(status, StatusCode::REQUEST_TIMEOUT);+ assert_eq!(body["number_of_nodes"], 3, "count stays real on timeout");+ }++ #[tokio::test]+ async fn self_in_own_peer_list_is_not_double_counted() {+ // Real configs name the full membership on every node, including+ // this one — the count must still be the distinct membership.+ let state = state_with_cluster(+ true,+ &[+ "127.0.0.1:19300=127.0.0.1:19300",+ "127.0.0.1:19301=127.0.0.1:19301",+ ],+ 19300,+ );+ let app = crate::router::build_es_compat_router(state);+ let (_, body) = get(&app, "/_cluster/health").await;+ assert_eq!(body["number_of_nodes"], 2);+ }++ #[tokio::test]+ async fn cat_nodes_lists_configured_members() {+ let state = ring3();+ let app = crate::router::build_es_compat_router(state);+ let (status, body) = get(&app, "/_cat/nodes?format=json").await;+ assert_eq!(status, StatusCode::OK);+ let rows = body.as_array().expect("json array");+ assert_eq!(rows.len(), 3);+ // Self row carries live metrics; peer rows carry "-"; and no row+ // claims mastership — the REST layer does not know the leader.+ assert!(rows[0]["heap.percent"].as_str().unwrap() != "-");+ for row in rows.iter() {+ assert_eq!(row["master"], "-");+ }+ assert_eq!(rows[1]["name"], "127.0.0.1:19301");+ assert_eq!(rows[2]["name"], "127.0.0.1:19302");+ assert_eq!(rows[1]["heap.percent"], "-");++ // A cluster-disabled node is a legitimate single-node cluster: one+ // row, and self IS the master.+ let state = state_with_cluster(false, &[], 9300);+ let app = crate::router::build_es_compat_router(state);+ let (_, body) = get(&app, "/_cat/nodes?format=json").await;+ let rows = body.as_array().expect("json array");+ assert_eq!(rows.len(), 1);+ assert_eq!(rows[0]["master"], "*");+ }++ #[tokio::test]+ async fn nodes_info_totals_membership_and_never_invents_peer_details() {+ let state = ring3();+ let app = crate::router::build_es_compat_router(state);+ let (status, body) = get(&app, "/_nodes").await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(body["_nodes"]["total"], 3);+ assert_eq!(body["_nodes"]["successful"], 1, "only self answered");+ let nodes = body["nodes"].as_object().expect("nodes map");+ assert_eq!(nodes.len(), 3);+ // The self entry reports the real configured cluster address, not+ // the never-bound 9300 default.+ let self_entry = nodes+ .values()+ .find(|v| v.get("version").is_some())+ .expect("exactly the self entry carries a version");+ assert_eq!(self_entry["transport_address"], "127.0.0.1:19300");+ // Peer entries carry name/transport only — no invented versions.+ let peer = &nodes["127.0.0.1:19301"];+ assert_eq!(peer["transport_address"], "127.0.0.1:19301");+ assert!(+ peer.get("version").is_none(),+ "peer version would be invented"+ );+ }++ #[tokio::test]+ async fn cluster_state_names_real_transport_and_admits_unknown_master() {+ let state = ring3();+ let app = crate::router::build_es_compat_router(state);+ let (status, body) = get(&app, "/_cluster/state").await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(+ body["master_node"],+ Value::Null,+ "leader unknown to REST (#1170)"+ );+ let nodes = body["nodes"].as_object().expect("nodes map");+ assert_eq!(nodes.len(), 3);+ let self_entry = nodes+ .values()+ .find(|v| v.get("roles").is_some())+ .expect("self entry carries roles");+ assert_eq!(self_entry["transport_address"], "127.0.0.1:19300");++ // Cluster disabled: self is the trivial master and the shape is+ // byte-compatible with the old single-entry response.+ let state = state_with_cluster(false, &[], 9300);+ let app = crate::router::build_es_compat_router(state);+ let (_, body) = get(&app, "/_cluster/state").await;+ assert_eq!(body["master_node"], "local");+ let nodes = body["nodes"].as_object().expect("nodes map");+ assert_eq!(nodes.len(), 1);+ }+}diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex f5499211b..704a02d7e 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -355,6 +355,29 @@ impl Config {// control port (issue #75).self.cluster.validate()?;+ // The cluster port must not collide with this node's own data+ // listeners (#1171). A collision makes the cluster transport's bind+ // fail at boot and the node degrades to single-node while+ // /_cluster/health stays green — refusing here turns that trap into+ // a boot-time error. (Cross-process port collisions can only be+ // caught at bind time; this check catches the self-inflicted class.)+ if self.cluster.enabled {+ let cluster_port = self.cluster.port;+ let listeners = [+ ("rest_port", self.server.rest_port),+ ("grpc_port", self.server.grpc_port),+ ("es_compat_port", self.server.es_compat_port),+ ];+ let collision = listeners.iter().find(|(_, p)| *p == cluster_port);+ if let Some((name, _)) = collision {+ return Err(XerjError::config(format!(+ "cluster.port ({cluster_port}) collides with server.{name} — the cluster \+ transport would fail to bind and the node would silently degrade to \+ single-node mode (issue #1171)"+ )));+ }+ }+// Logging: format must be one of the two supported line formats.let fmt = self.logging.format.as_str();if !fmt.eq_ignore_ascii_case("text") && !fmt.eq_ignore_ascii_case("json") {@@ -1752,9 +1775,14 @@ pub struct ClusterConfig {/// Each node in the cluster must expose this port and it must be reachable/// from all peers.pub port: u16,- /// Peer nodes in `"<node_id>=<host>:<port>"` format.- ///- /// Example: `["n2=10.0.0.2:9300", "n3=10.0.0.3:9300"]`+ /// Peer nodes in `"<host>:<port>=<host>:<port>"` format — the label must+ /// equal the address. A node's cluster id IS its socket address and+ /// replies are routed by looking the sender's id up in this list, so a+ /// differing label makes vote replies undeliverable and the node loses+ /// every election with no diagnostic of its own (issue #1171; refused+ /// at [`ClusterConfig::validate`]).+ ///+ /// Example: `["10.0.0.2:9300=10.0.0.2:9300", "10.0.0.3:9300=10.0.0.3:9300"]`pub peers: Vec<String>,/// Raft tick interval in milliseconds (default: `50`).///@@ -1845,8 +1873,53 @@ impl ClusterConfig {Self::MIN_AUTH_SECRET_LEN)))}- Some(_) => Ok(()),+ Some(_) => self.validate_peers(),+ }+ }++ /// Validate peer entries beyond the shared-secret rules: every entry+ /// must parse, and its label must equal its address (#1171).+ ///+ /// The cluster transport keys its reply map by peer label and routes+ /// replies through `peers.get(candidate_id)` — and a node's Raft id IS+ /// its listen address (derived in the server boot). A friendly label+ /// (`n2=10.0.0.2:9300`) therefore makes the node's vote reply+ /// undeliverable: it becomes an eternal candidate, re-electing every+ /// ~200 ms, and the only diagnostic lands in the *rejecting* node's+ /// log (`unknown peer: …`), never its own. Refusing at config load+ /// turns that silent runtime churn into a boot-time error.+ fn validate_peers(&self) -> Result<(), crate::XerjError> {+ for peer in &self.peers {+ match peer.split_once('=') {+ Some((id, addr)) => {+ let addr = addr.trim();+ if let Err(e) = addr.parse::<std::net::SocketAddr>() {+ return Err(crate::XerjError::config(format!(+ "cluster.peers entry {peer:?} has an unparseable address ({e}); \+ expected \"<host:port>=<host:port>\" with the same address on both \+ sides of the '='"+ )));+ }+ if id.trim() != addr {+ return Err(crate::XerjError::config(format!(+ "cluster.peers entry {peer:?}: the label must equal the address. \+ Node ids in the cluster are socket addresses and replies are \+ routed by looking the sender's id up in the peer list, so a label \+ that differs from the address makes this node's vote replies \+ undeliverable — it will lose every election with no diagnostic of \+ its own (issue #1171). Write the entry as \"{addr}={addr}\""+ )));+ }+ }+ None => {+ return Err(crate::XerjError::config(format!(+ "cluster.peers entry {peer:?} is malformed: expected \+ \"<host:port>=<host:port>\" (the label must equal the address)"+ )))+ }+ }}+ Ok(())}}@@ -3533,6 +3606,79 @@ mod tests {.expect("defaults must validate");}+ // ── Cluster peers / ports: fail at load, not as silent runtime churn+ // (issue #1171) ────────────────────────────────────────────────────++ #[test]+ fn cluster_peer_label_must_equal_its_address() {+ // A friendly label makes vote replies undeliverable (replies are+ // routed by looking the sender's id up in the peer list, and ids+ // ARE addresses) — the node then loses every election with no+ // diagnostic of its own. Measured: 51+ elections in ~15 s.+ let err = Config::from_toml_str(+ "[cluster]\nenabled = true\nauth_secret = \"0123456789abcdef0123\"\n\+ peers = [\"n2=10.0.0.2:9300\"]\n",+ )+ .expect_err("a peer label that differs from its address must be refused at load");+ let msg = err.to_string();+ assert!(+ msg.contains("label must equal the address"),+ "error must name the rule, got: {msg}"+ );+ assert!(+ msg.contains("10.0.0.2:9300=10.0.0.2:9300"),+ "error must show the corrected entry, got: {msg}"+ );+ }++ #[test]+ fn cluster_peer_label_matching_address_is_accepted() {+ Config::from_toml_str(+ "[cluster]\nenabled = true\nauth_secret = \"0123456789abcdef0123\"\n\+ peers = [\"10.0.0.2:9300=10.0.0.2:9300\", \"10.0.0.3:9300=10.0.0.3:9300\"]\n",+ )+ .expect("label==address peer entries must load");+ }++ #[test]+ fn cluster_malformed_peer_entry_is_rejected() {+ let err = Config::from_toml_str(+ "[cluster]\nenabled = true\nauth_secret = \"0123456789abcdef0123\"\n\+ peers = [\"10.0.0.2:9300\"]\n",+ )+ .expect_err("a peer entry without label=address must be refused");+ assert!(+ err.to_string().contains("malformed"),+ "error must say malformed, got: {err}"+ );+ }++ #[test]+ fn cluster_port_colliding_with_a_data_listener_is_rejected() {+ // The bind would fail at boot and the node would silently degrade+ // to single-node with a green /_cluster/health — refuse at load.+ let err = Config::from_toml_str(+ "[server]\nes_compat_port = 9300\n[cluster]\nenabled = true\n\+ auth_secret = \"0123456789abcdef0123\"\nport = 9300\n",+ )+ .expect_err("cluster.port colliding with a data listener must be refused");+ let msg = err.to_string();+ assert!(+ msg.contains("collides with server.es_compat_port") && msg.contains("#1171"),+ "error must name the colliding listener, got: {msg}"+ );+ }++ #[test]+ fn cluster_port_distinct_from_data_listeners_is_accepted() {+ Config::from_toml_str(+ "[server]\nrest_port = 18000\ngrpc_port = 18001\nes_compat_port = 18002\n\+ [cluster]\nenabled = true\nauth_secret = \"0123456789abcdef0123\"\nport = 19300\n\+ peers = [\"10.0.0.2:19300=10.0.0.2:19300\"]\n",+ )+ .expect("a distinct cluster port must load");+ }+#[test]fn auth_secret_resolution_prefers_explicit_config() {let env = Some("from-environment".to_string());diff --git a/engine/tests/es-compat-yaml/yaml/smoke/smoke_test_multinode/10_basic.yml b/engine/tests/es-compat-yaml/yaml/smoke/smoke_test_multinode/10_basic.ymlindex 99446f213..7013862e0 100644--- a/engine/tests/es-compat-yaml/yaml/smoke/smoke_test_multinode/10_basic.yml+++ b/engine/tests/es-compat-yaml/yaml/smoke/smoke_test_multinode/10_basic.yml@@ -1,8 +1,19 @@# Integration tests for smoke testing multi-node IT# If the local machine which is running the test is low on disk space# We can have one unassigned shard+#+# XERJ note (2026-10-06): skipped with a reason. This case used to "pass"+# because /_cluster/health echoed the requested wait_for_nodes count back as+# number_of_nodes — a fabricated topology on a single node (issue #1169).+# Health now reports real configured membership and times out an unsatisfiable+# wait_for_nodes, so this case can only pass against a genuine 2-node runner,+# which the conformance harness does not boot. Un-skip when the Raft ring is+# wired to the REST layer (#1170) and the runner can start a real cluster.---"cluster health basic test, wait for both nodes to join":+ - skip:+ version: "all"+ reason: "requires a genuine 2-node cluster; XERJ reports real membership now (xerj-org/xerj#1169)"- do:cluster.health:wait_for_nodes: 2