AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
security(cluster): authenticate Raft control frames, fail closed without a secret (#113)
xerj-org837a34df3b0c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex cf1d357a9..5d85299db 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -73,6 +73,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0Operators terminating TLS or load-balancing in front of XERJ must set`server.trusted_proxies` to their proxy's address, otherwise every userbehind it shares one rate-limit bucket.+- **Cluster control frames are now authenticated (issue #75).** The inter-node+ transport on `cluster.port` accepted any TCP connection that spoke its wire+ format: anyone who could reach the port could send Raft `RequestVote` /+ `AppendEntries` frames and steer consensus. Every frame now carries an+ HMAC-SHA256 tag over a per-connection random challenge issued by the+ receiver, the sender's node id, the frame's position in the connection, and+ the payload; tags are compared in constant time, and the tag is verified+ before the payload reaches the JSON deserialiser. The challenge makes a+ captured frame non-replayable, including against a different node.+- **Cluster mode fails closed.** `[cluster] enabled = true` now requires+ `cluster.auth_secret` (or `XERJ_CLUSTER_AUTH_SECRET`), minimum 16+ characters. With cluster mode on and no secret from either source the node+ **refuses to start** instead of running an unauthenticated control port.+ Single-node mode — the default — is unaffected and needs no secret.++### Breaking++- **The cluster wire format changed and is not backward compatible.** A node+ running this version cannot talk to a node running rc.9 or earlier in either+ direction: the receiver now speaks first (magic, version, challenge) where+ the old protocol expected the sender to open with a length-prefixed JSON+ frame. Upgrading a cluster requires stopping every node and restarting them+ on the new version with a shared secret configured — a rolling restart will+ not interoperate. Only cluster mode is affected; it is off by default, so+ single-node deployments need no action.+- Scope note: the HMAC authenticates frames, it does not encrypt them. Cluster+ traffic is still plaintext and the secret is cluster-wide, so it proves+ cluster membership, not per-node identity. `cluster.port` still belongs on a+ trusted network segment; mTLS remains unimplemented.## [1.0.0-rc.9] - 2026-08-01diff --git a/docs/case-studies/xerj-self-audit/ISSUES.md b/docs/case-studies/xerj-self-audit/ISSUES.mdindex cd56c47db..d83f09cd7 100644--- a/docs/case-studies/xerj-self-audit/ISSUES.md+++ b/docs/case-studies/xerj-self-audit/ISSUES.md@@ -9,7 +9,7 @@ live-reproduction transcripts: [FINDINGS-V2.md](FINDINGS-V2.md).| [#72](https://github.com/xerj-org/xerj/issues/72) | INJ-01 search-template injection | High | fixed, tracking merge || [#73](https://github.com/xerj-org/xerj/issues/73) | F-PATH-02 snapshot location traversal | High | fixed, tracking merge || [#74](https://github.com/xerj-org/xerj/issues/74) | S5-1 session-revocation lost update | High | fixed, tracking merge |-| [#75](https://github.com/xerj-org/xerj/issues/75) | DESER-EGRESS-01 unauth Raft transport | High (gated) | open — Raft-auth Phase 2 |+| [#75](https://github.com/xerj-org/xerj/issues/75) | DESER-EGRESS-01 unauth Raft transport | High (gated) | fixed, tracking merge — HMAC frame auth + fail-closed config || [#76](https://github.com/xerj-org/xerj/issues/76) | Console hardening: S5-4, S5-3, S5-5, AUTHZ-2 | Med/Low/Info | open |The three Highs were filed publicly only *after* the fix was committed. Thediff --git a/docs/recipes/production-deployment.md b/docs/recipes/production-deployment.mdindex 0d5e9d19c..064f8f348 100644--- a/docs/recipes/production-deployment.md+++ b/docs/recipes/production-deployment.md@@ -499,22 +499,34 @@ inter-node consensus and search:```toml[cluster]-enabled = true-port = 9300-peers = ["n2=10.0.0.2:9300", "n3=10.0.0.3:9300"]-tick_ms = 50+enabled = true+port = 9300+peers = ["n2=10.0.0.2:9300", "n3=10.0.0.3:9300"]+tick_ms = 50+auth_secret = "…" # or XERJ_CLUSTER_AUTH_SECRET; same value on every node```-That inter-node transport is **plaintext and unauthenticated** — it is a plain-TCP listener with no TLS and no API-key check on the Raft/search messages. The-API-key auth and in-process TLS described above protect the *client-facing*-listeners (REST/ES/gRPC); they do **not** extend to `cluster.port`. Anyone who-can reach `:9300` can participate in the cluster protocol. So if you evaluate-cluster mode, keep every `cluster.port` on a fully trusted, isolated network-segment, and treat the feature as experimental — cross-cluster replication and-disaster recovery are out of scope. For production HA today, mirror via-snapshot/restore on a schedule, or dual-write to two independent single nodes-at the application layer.+`auth_secret` is **mandatory** when `enabled = true`. Every control frame on+`cluster.port` carries an HMAC-SHA256 tag over a per-connection random+challenge, so a peer that does not hold the secret cannot inject Raft messages+and a captured frame cannot be replayed. A node started with cluster mode on+and no secret — from either the config file or `XERJ_CLUSTER_AUTH_SECRET` —+**refuses to start** rather than open an unauthenticated control port. Generate+one with `openssl rand -hex 32` (minimum 16 characters) and use the identical+value on every node; a node with the wrong secret is simply not admitted.++Two limits to be clear about. First, the HMAC **authenticates, it does not+encrypt**: cluster traffic is still plaintext JSON, so `cluster.port` still+belongs on a trusted, isolated network segment, and the API-key auth and TLS+described above still cover only the client-facing listeners (REST/ES/gRPC).+Second, the secret is cluster-*wide*, so it proves "this peer is a cluster+member", not "this peer is node n2" — a compromised member can impersonate any+other member. Per-node identity needs mTLS, which is not implemented.++Cluster mode remains experimental — cross-cluster replication and disaster+recovery are out of scope. For production HA today, mirror via snapshot/restore+on a schedule, or dual-write to two independent single nodes at the application+layer.---diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 470426eba..3d98b8fbc 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -6449,9 +6449,12 @@ dependencies = ["bytes","crc32fast","futures",+ "hmac","rand 0.8.5","serde","serde_json",+ "sha2 0.10.9",+ "subtle","tempfile","thiserror 2.0.18","tokio",diff --git a/engine/crates/xerj-cluster/Cargo.toml b/engine/crates/xerj-cluster/Cargo.tomlindex f54087548..0842a0069 100644--- a/engine/crates/xerj-cluster/Cargo.toml+++ b/engine/crates/xerj-cluster/Cargo.toml@@ -22,6 +22,11 @@ rand.workspace = trueasync-trait = "0.1"xxhash-rust.workspace = truefutures = "0.3"+# Cluster control-frame authentication (issue #75): HMAC-SHA256 over every+# frame, compared in constant time.+hmac = "0.12"+sha2 = "0.10"+subtle = "2"[dev-dependencies]tokio = { version = "1", features = ["full", "test-util"] }diff --git a/engine/crates/xerj-cluster/src/auth.rs b/engine/crates/xerj-cluster/src/auth.rsnew file mode 100644index 000000000..cc8c77eaa--- /dev/null+++ b/engine/crates/xerj-cluster/src/auth.rs@@ -0,0 +1,271 @@+//! Shared-secret authentication for the cluster control transport.+//!+//! Every control frame on the cluster port carries an HMAC-SHA256 tag computed+//! over a domain-separated binding of:+//!+//! * a protocol context string (`hello` vs `frame` — domain separation),+//! * the wire version byte,+//! * the **per-connection challenge** the receiver generated and sent first,+//! * the sender's node id,+//! * the frame's position in the connection (`seq`, implicit — never on the wire),+//! * the frame payload.+//!+//! The challenge is what makes a captured frame useless: it is 32 fresh random+//! bytes chosen by the *receiver* for every accepted connection, so a recorded+//! frame replayed to the same node (or reflected at a different node) is+//! verified against a challenge it was never signed under and is rejected. The+//! implicit `seq` additionally pins each frame to its position, so frames+//! within a live connection cannot be reordered, dropped, or duplicated+//! without the tag failing.+//!+//! ## What this does *not* give you+//!+//! * **No confidentiality.** Frames are still plaintext JSON on the wire; the+//! HMAC authenticates, it does not encrypt. Cluster traffic still belongs on+//! a trusted network segment.+//! * **No per-node identity.** The secret is cluster-wide, so it proves+//! "the peer knows the cluster secret", not "the peer is node X". A+//! compromised member can impersonate any other member. Per-node identity+//! needs mTLS or per-node keys, which is a larger design change.++use hmac::{Hmac, Mac};+use sha2::Sha256;+use subtle::ConstantTimeEq;++type HmacSha256 = Hmac<Sha256>;++// ── Wire constants ────────────────────────────────────────────────────────────++/// Magic bytes that open the receiver's handshake.+pub const WIRE_MAGIC: &[u8; 8] = b"XERJCLUS";++/// Wire version. Bumped whenever the authenticated framing changes.+pub const WIRE_VERSION: u8 = 1;++/// Length of the per-connection challenge, in bytes.+pub const CHALLENGE_LEN: usize = 32;++/// Length of an HMAC-SHA256 tag, in bytes.+pub const TAG_LEN: usize = 32;++/// Minimum accepted length of a configured cluster secret, in characters.+///+/// Short secrets are brute-forceable offline from a single captured frame, so+/// they are rejected outright rather than accepted with a warning.+pub const MIN_SECRET_LEN: usize = 16;++/// Domain-separation context for the connection hello.+const CTX_HELLO: &[u8] = b"xerj-cluster/v1/hello";++/// Domain-separation context for a message frame.+const CTX_FRAME: &[u8] = b"xerj-cluster/v1/frame";++// ── Errors ────────────────────────────────────────────────────────────────────++/// Failure to build a [`ClusterSecret`] from operator-supplied material.+#[derive(Debug, thiserror::Error, PartialEq, Eq)]+pub enum SecretError {+ /// No secret was supplied at all.+ #[error(+ "cluster authentication secret is empty — cluster mode refuses to run \+ an unauthenticated control transport"+ )]+ Empty,+ /// The secret was supplied but is too short to resist offline guessing.+ #[error("cluster authentication secret is too short ({got} chars, minimum {MIN_SECRET_LEN})")]+ TooShort {+ /// The length that was supplied.+ got: usize,+ },+}++// ── ClusterSecret ─────────────────────────────────────────────────────────────++/// A validated cluster-wide shared secret.+///+/// Constructing one is the *only* way to obtain frame tags, which is how the+/// transport is made structurally incapable of running unauthenticated: there+/// is no "no secret" variant to fall back to.+#[derive(Clone)]+pub struct ClusterSecret {+ key: Vec<u8>,+}++impl std::fmt::Debug for ClusterSecret {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ // Never render key material, not even a prefix.+ f.write_str("ClusterSecret(<redacted>)")+ }+}++impl ClusterSecret {+ /// Validate and adopt an operator-supplied secret.+ ///+ /// Surrounding whitespace is trimmed (a trailing newline from+ /// `XERJ_CLUSTER_AUTH_SECRET=$(cat secret)` is a configuration accident,+ /// not key material), after which the secret must be at least+ /// [`MIN_SECRET_LEN`] characters.+ pub fn new(secret: &str) -> Result<Self, SecretError> {+ let trimmed = secret.trim();+ if trimmed.is_empty() {+ return Err(SecretError::Empty);+ }+ if trimmed.chars().count() < MIN_SECRET_LEN {+ return Err(SecretError::TooShort {+ got: trimmed.chars().count(),+ });+ }+ Ok(ClusterSecret {+ key: trimmed.as_bytes().to_vec(),+ })+ }++ /// Tag authenticating a connection hello (the sender's node id, bound to+ /// the receiver's challenge).+ pub fn hello_tag(&self, challenge: &[u8; CHALLENGE_LEN], sender: &str) -> [u8; TAG_LEN] {+ let mut mac = self.mac();+ mac.update(CTX_HELLO);+ mac.update(&[WIRE_VERSION]);+ mac.update(challenge);+ Self::update_lp(&mut mac, sender.as_bytes());+ mac.finalize().into_bytes().into()+ }++ /// Tag authenticating one message frame.+ ///+ /// `seq` is the zero-based index of this frame within the connection. It is+ /// never transmitted: both ends count independently, so a replayed,+ /// dropped, or reordered frame lands on the wrong `seq` and fails.+ pub fn frame_tag(+ &self,+ challenge: &[u8; CHALLENGE_LEN],+ sender: &str,+ seq: u64,+ payload: &[u8],+ ) -> [u8; TAG_LEN] {+ let mut mac = self.mac();+ mac.update(CTX_FRAME);+ mac.update(&[WIRE_VERSION]);+ mac.update(challenge);+ Self::update_lp(&mut mac, sender.as_bytes());+ mac.update(&seq.to_be_bytes());+ Self::update_lp(&mut mac, payload);+ mac.finalize().into_bytes().into()+ }++ fn mac(&self) -> HmacSha256 {+ // HMAC accepts a key of any length; this cannot fail.+ HmacSha256::new_from_slice(&self.key).expect("HMAC accepts keys of any length")+ }++ /// Feed a length-prefixed field so that concatenation is unambiguous+ /// (`"ab" || "c"` must not tag the same as `"a" || "bc"`).+ fn update_lp(mac: &mut HmacSha256, bytes: &[u8]) {+ mac.update(&(bytes.len() as u64).to_be_bytes());+ mac.update(bytes);+ }+}++// ── Constant-time comparison ──────────────────────────────────────────────────++/// Compare a computed tag against a tag read off the wire in constant time.+///+/// A plain `==` on the byte arrays would short-circuit on the first differing+/// byte and leak, through timing, how many leading bytes an attacker guessed+/// correctly — enough to forge a tag byte-by-byte. Length is not secret, so the+/// length check may short-circuit.+#[must_use]+pub fn tags_match(expected: &[u8; TAG_LEN], actual: &[u8]) -> bool {+ if actual.len() != TAG_LEN {+ return false;+ }+ expected.ct_eq(actual).into()+}++#[cfg(test)]+mod tests {+ use super::*;++ const SECRET: &str = "a-perfectly-fine-cluster-secret";++ fn challenge(b: u8) -> [u8; CHALLENGE_LEN] {+ [b; CHALLENGE_LEN]+ }++ #[test]+ fn rejects_empty_and_short_secrets() {+ assert_eq!(ClusterSecret::new("").unwrap_err(), SecretError::Empty);+ assert_eq!(ClusterSecret::new(" \n").unwrap_err(), SecretError::Empty);+ assert_eq!(+ ClusterSecret::new("short").unwrap_err(),+ SecretError::TooShort { got: 5 }+ );+ assert!(ClusterSecret::new(SECRET).is_ok());+ }++ #[test]+ fn secret_debug_never_leaks_key_material() {+ let s = ClusterSecret::new(SECRET).unwrap();+ let rendered = format!("{s:?}");+ assert!(+ !rendered.contains("cluster-secret"),+ "Debug leaked key material: {rendered}"+ );+ assert!(rendered.contains("redacted"));+ }++ #[test]+ fn tag_binds_challenge_sender_seq_and_payload() {+ let s = ClusterSecret::new(SECRET).unwrap();+ let base = s.frame_tag(&challenge(1), "node-a", 0, b"payload");++ assert_ne!(base, s.frame_tag(&challenge(2), "node-a", 0, b"payload"));+ assert_ne!(base, s.frame_tag(&challenge(1), "node-b", 0, b"payload"));+ assert_ne!(base, s.frame_tag(&challenge(1), "node-a", 1, b"payload"));+ assert_ne!(base, s.frame_tag(&challenge(1), "node-a", 0, b"payloa!"));++ // Different secret → different tag.+ let other = ClusterSecret::new("an-entirely-different-secret").unwrap();+ assert_ne!(+ base,+ other.frame_tag(&challenge(1), "node-a", 0, b"payload")+ );++ // Same inputs → same tag.+ assert_eq!(base, s.frame_tag(&challenge(1), "node-a", 0, b"payload"));+ }++ #[test]+ fn hello_and_frame_contexts_are_separated() {+ let s = ClusterSecret::new(SECRET).unwrap();+ let hello = s.hello_tag(&challenge(7), "node-a");+ let frame = s.frame_tag(&challenge(7), "node-a", 0, b"");+ assert_ne!(hello, frame, "hello and frame tags must not be confusable");+ }++ #[test]+ fn length_prefixing_prevents_field_smuggling() {+ let s = ClusterSecret::new(SECRET).unwrap();+ // "ab" + payload "c" must not collide with "a" + payload "bc".+ assert_ne!(+ s.frame_tag(&challenge(0), "ab", 0, b"c"),+ s.frame_tag(&challenge(0), "a", 0, b"bc")+ );+ }++ #[test]+ fn tags_match_accepts_equal_and_rejects_everything_else() {+ let s = ClusterSecret::new(SECRET).unwrap();+ let tag = s.frame_tag(&challenge(3), "n", 0, b"x");+ assert!(tags_match(&tag, &tag));++ let mut flipped = tag;+ flipped[TAG_LEN - 1] ^= 0x01;+ assert!(!tags_match(&tag, &flipped));++ // Truncated and over-long tags are rejected without panicking.+ assert!(!tags_match(&tag, &tag[..TAG_LEN - 1]));+ assert!(!tags_match(&tag, &[0u8; TAG_LEN + 1]));+ assert!(!tags_match(&tag, &[]));+ }+}diff --git a/engine/crates/xerj-cluster/src/lib.rs b/engine/crates/xerj-cluster/src/lib.rsindex 2100a0439..e106d4b58 100644--- a/engine/crates/xerj-cluster/src/lib.rs+++ b/engine/crates/xerj-cluster/src/lib.rs@@ -46,6 +46,7 @@//! }//! ```+pub mod auth;pub mod coordinator;pub mod metadata;pub mod node;@@ -60,6 +61,7 @@ pub mod transport;// ── Convenience re-exports ────────────────────────────────────────────────────+pub use auth::{ClusterSecret, SecretError};pub use coordinator::{IndexResponse, LocalSearcher, MergedSearchResult, SearchCoordinator, SearchTransport,};diff --git a/engine/crates/xerj-cluster/src/transport.rs b/engine/crates/xerj-cluster/src/transport.rsindex fc6d4bf06..b3f485b1a 100644--- a/engine/crates/xerj-cluster/src/transport.rs+++ b/engine/crates/xerj-cluster/src/transport.rs@@ -1,35 +1,202 @@//! TCP-based transport for inter-node communication.//!-//! Wire format: `[4-byte big-endian length][JSON-serialized RaftMessage]`+//! Every connection is authenticated with the cluster-wide shared secret+//! (HMAC-SHA256, see [`crate::auth`]). There is no unauthenticated mode: the+//! transport cannot be constructed without a validated [`ClusterSecret`].//!-//! Each connection carries messages from one peer to this node. The sender-//! writes length-prefixed JSON frames; the receiver decodes them and pushes-//! them onto the shared `incoming` channel.+//! ## Wire format (version 1)+//!+//! ```text+//! receiver → sender (handshake, sent immediately on accept)+//! [8] magic "XERJCLUS"+//! [1] wire version (1)+//! [32] challenge — fresh random bytes, per connection+//!+//! sender → receiver (hello, exactly once)+//! [4] node_id length (u32 BE, ≤ 256)+//! [n] node_id (UTF-8)+//! [32] HMAC tag over (hello-context, version, challenge, node_id)+//!+//! sender → receiver (message frame, repeated until EOF)+//! [4] payload length (u32 BE, ≤ 10 MiB)+//! [32] HMAC tag over (frame-context, version, challenge, node_id, seq, payload)+//! [n] payload — JSON-serialised RaftMessage+//! ```+//!+//! `seq` is the frame's zero-based index within the connection and is never+//! transmitted; both ends count it independently.+//!+//! A frame's tag is verified **before** its payload is handed to the JSON+//! deserialiser, so unauthenticated bytes never reach `serde_json`.+//!+//! ## Wire compatibility+//!+//! This framing is **not** compatible with the pre-authentication format. A+//! node running this version cannot talk to a node running an older one, in+//! either direction: the old sender writes a JSON frame where the new receiver+//! expects to write a challenge first, and the new sender waits for a handshake+//! an old receiver never sends. Upgrading a cluster requires a full stop/start,+//! not a rolling restart.use std::collections::HashMap;use std::net::SocketAddr;use std::sync::Arc;+use std::time::Duration;use anyhow::{Context, Result};use async_trait::async_trait;+use rand::RngCore;use tokio::io::{AsyncReadExt, AsyncWriteExt};use tokio::net::{TcpListener, TcpStream};use tokio::sync::{mpsc, Mutex};use tracing::{debug, info, warn};+use crate::auth::{tags_match, ClusterSecret, CHALLENGE_LEN, TAG_LEN, WIRE_MAGIC, WIRE_VERSION};use crate::node::ClusterTransport;use crate::raft::RaftMessage;+/// Largest accepted frame payload (10 MiB).+const MAX_FRAME_BYTES: usize = 10 * 1024 * 1024;++/// Largest accepted node-id length in the hello header.+const MAX_NODE_ID_BYTES: usize = 256;++/// How long the receiver will wait for a connected peer to complete its hello.+///+/// Without this, a peer that connects and then goes silent pins an accept task+/// (and its buffers) indefinitely — cheap for an attacker, expensive for us.+const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(10);++/// How long a send may take once the TCP connection is established.+///+/// Authentication makes the sender *read* before it writes (it needs the+/// receiver's challenge), so a peer that accepts connections and then goes+/// silent could otherwise stall the Raft loop indefinitely. The whole+/// post-connect exchange is bounded instead.+const SEND_TIMEOUT: Duration = Duration::from_secs(5);++/// Length of the receiver's handshake: magic ‖ version ‖ challenge.+const HANDSHAKE_LEN: usize = WIRE_MAGIC.len() + 1 + CHALLENGE_LEN;+// ── Wire helpers ──────────────────────────────────────────────────────────────-/// Write a single length-prefixed JSON frame to the given stream.-async fn write_frame(stream: &mut TcpStream, msg: &RaftMessage) -> Result<()> {+/// Write the receiver-side handshake and return the challenge it carried.+async fn write_handshake(stream: &mut TcpStream) -> Result<[u8; CHALLENGE_LEN]> {+ let mut challenge = [0u8; CHALLENGE_LEN];+ rand::rngs::OsRng.fill_bytes(&mut challenge);++ let mut buf = [0u8; HANDSHAKE_LEN];+ buf[..WIRE_MAGIC.len()].copy_from_slice(WIRE_MAGIC);+ buf[WIRE_MAGIC.len()] = WIRE_VERSION;+ buf[WIRE_MAGIC.len() + 1..].copy_from_slice(&challenge);++ stream.write_all(&buf).await.context("write handshake")?;+ stream.flush().await.context("flush handshake")?;+ Ok(challenge)+}++/// Read and validate the receiver's handshake, returning the challenge.+async fn read_handshake(stream: &mut TcpStream) -> Result<[u8; CHALLENGE_LEN]> {+ let mut buf = [0u8; HANDSHAKE_LEN];+ stream+ .read_exact(&mut buf)+ .await+ .context("read handshake")?;++ if &buf[..WIRE_MAGIC.len()] != WIRE_MAGIC {+ anyhow::bail!("peer did not send a xerj cluster handshake (bad magic)");+ }+ let version = buf[WIRE_MAGIC.len()];+ if version != WIRE_VERSION {+ anyhow::bail!(+ "cluster wire version mismatch: peer speaks v{version}, this node speaks v{WIRE_VERSION}"+ );+ }++ let mut challenge = [0u8; CHALLENGE_LEN];+ challenge.copy_from_slice(&buf[WIRE_MAGIC.len() + 1..]);+ Ok(challenge)+}++/// Write the authenticated hello identifying this node.+async fn write_hello(+ stream: &mut TcpStream,+ secret: &ClusterSecret,+ challenge: &[u8; CHALLENGE_LEN],+ node_id: &str,+) -> Result<()> {+ let id = node_id.as_bytes();+ if id.len() > MAX_NODE_ID_BYTES {+ anyhow::bail!("node_id too long: {} bytes", id.len());+ }+ let tag = secret.hello_tag(challenge, node_id);++ stream+ .write_all(&(id.len() as u32).to_be_bytes())+ .await+ .context("write hello length")?;+ stream.write_all(id).await.context("write hello node_id")?;+ stream.write_all(&tag).await.context("write hello tag")?;+ stream.flush().await.context("flush hello")?;+ Ok(())+}++/// Read and authenticate the peer's hello, returning its node id.+async fn read_hello(+ stream: &mut TcpStream,+ secret: &ClusterSecret,+ challenge: &[u8; CHALLENGE_LEN],+) -> Result<String> {+ let mut len_buf = [0u8; 4];+ stream+ .read_exact(&mut len_buf)+ .await+ .context("read hello length")?;+ let len = u32::from_be_bytes(len_buf) as usize;+ if len > MAX_NODE_ID_BYTES {+ anyhow::bail!("hello node_id too large: {len} bytes");+ }++ let mut id_buf = vec![0u8; len];+ stream+ .read_exact(&mut id_buf)+ .await+ .context("read hello node_id")?;+ let mut tag = [0u8; TAG_LEN];+ stream+ .read_exact(&mut tag)+ .await+ .context("read hello tag")?;++ // Authenticate the raw bytes before trusting them as a UTF-8 node id.+ let from = String::from_utf8(id_buf).context("decode hello node_id")?;+ let expected = secret.hello_tag(challenge, &from);+ if !tags_match(&expected, &tag) {+ anyhow::bail!("cluster authentication failed: bad hello tag");+ }+ Ok(from)+}++/// Write a single authenticated frame.+async fn write_frame(+ stream: &mut TcpStream,+ secret: &ClusterSecret,+ challenge: &[u8; CHALLENGE_LEN],+ node_id: &str,+ seq: u64,+ msg: &RaftMessage,+) -> Result<()> {let payload = serde_json::to_vec(msg).context("serialize RaftMessage")?;- let len = payload.len() as u32;+ if payload.len() > MAX_FRAME_BYTES {+ anyhow::bail!("frame too large to send: {} bytes", payload.len());+ }+ let tag = secret.frame_tag(challenge, node_id, seq, &payload);+stream- .write_all(&len.to_be_bytes())+ .write_all(&(payload.len() as u32).to_be_bytes()).await.context("write frame length")?;+ stream.write_all(&tag).await.context("write frame tag")?;stream.write_all(&payload).await@@ -38,38 +205,63 @@ async fn write_frame(stream: &mut TcpStream, msg: &RaftMessage) -> Result<()> {Ok(())}-/// Read a single length-prefixed JSON frame from the given stream.-async fn read_frame(stream: &mut TcpStream) -> Result<RaftMessage> {+/// Read, authenticate, and decode a single frame.+///+/// Returns `Ok(None)` on a clean end of stream (the peer closed after its last+/// frame, which is the normal case for the connection-per-send sender).+///+/// The tag is verified before the payload is deserialised, so a forged or+/// tampered frame never reaches `serde_json`.+async fn read_frame(+ stream: &mut TcpStream,+ secret: &ClusterSecret,+ challenge: &[u8; CHALLENGE_LEN],+ from: &str,+ seq: u64,+) -> Result<Option<RaftMessage>> {let mut len_buf = [0u8; 4];- stream- .read_exact(&mut len_buf)- .await- .context("read frame length")?;+ match stream.read_exact(&mut len_buf).await {+ Ok(_) => {}+ Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => return Ok(None),+ Err(e) => return Err(anyhow::Error::new(e).context("read frame length")),+ }let len = u32::from_be_bytes(len_buf) as usize;-- // Guard against absurdly large frames (e.g. 10 MiB max).- if len > 10 * 1024 * 1024 {- anyhow::bail!("frame too large: {} bytes", len);+ if len > MAX_FRAME_BYTES {+ anyhow::bail!("frame too large: {len} bytes");}+ let mut tag = [0u8; TAG_LEN];+ stream+ .read_exact(&mut tag)+ .await+ .context("read frame tag")?;let mut payload = vec![0u8; len];stream.read_exact(&mut payload).await.context("read frame payload")?;- let msg: RaftMessage = serde_json::from_slice(&payload).context("deserialize RaftMessage")?;- Ok(msg)++ let expected = secret.frame_tag(challenge, from, seq, &payload);+ if !tags_match(&expected, &tag) {+ anyhow::bail!("cluster authentication failed: bad frame tag (seq {seq})");+ }++ let msg = serde_json::from_slice(&payload).context("deserialize RaftMessage")?;+ Ok(Some(msg))}// ── TcpTransport ─────────────────────────────────────────────────────────────/// TCP-based transport for inter-node communication.///-/// Wire format: `[4-byte big-endian length][JSON-serialized RaftMessage]`-////// Incoming messages arrive via a background listener task and are delivered/// through an mpsc channel. Outgoing messages open a fresh TCP connection per/// send (connection pooling is a future optimisation).+///+/// Every connection is authenticated in both directions of setup: the receiver+/// proves nothing (it holds no identity beyond the secret) but issues a+/// challenge, and the sender proves knowledge of the shared secret on the hello+/// and on every frame.pub struct TcpTransport {/// This node's identifier.pub node_id: String,@@ -77,6 +269,8 @@ pub struct TcpTransport {listen_addr: SocketAddr,/// Map of peer node_id → socket address.peers: Arc<HashMap<String, SocketAddr>>,+ /// Cluster-wide shared secret used to authenticate every frame.+ secret: ClusterSecret,/// Receives `(sender_node_id, msg)` from the background listener.incoming: Arc<Mutex<mpsc::Receiver<(String, RaftMessage)>>>,// The sender half is kept alive so the channel is not closed when the@@ -88,12 +282,13 @@ pub struct TcpTransport {impl TcpTransport {/// Create a new TCP transport and begin listening for inbound connections.///- /// The listener task is spawned immediately; call [`start`] is not needed- /// (this constructor already binds and spawns).+ /// `secret` is the cluster-wide shared secret. It is required: there is no+ /// constructor that yields an unauthenticated transport.pub async fn new(node_id: String,listen_addr: SocketAddr,peers: HashMap<String, SocketAddr>,+ secret: ClusterSecret,) -> Result<Self> {let (tx, rx) = mpsc::channel::<(String, RaftMessage)>(1024);@@ -101,6 +296,7 @@ impl TcpTransport {node_id: node_id.clone(),listen_addr,peers: Arc::new(peers),+ secret,incoming: Arc::new(Mutex::new(rx)),sender: tx.clone(),};@@ -118,7 +314,8 @@ impl TcpTransport {.with_context(|| format!("bind TCP transport on {}", self.listen_addr))?;let node_id = self.node_id.clone();- info!(node = %node_id, addr = %self.listen_addr, "TCP transport listening");+ let secret = self.secret.clone();+ info!(node = %node_id, addr = %self.listen_addr, "TCP transport listening (authenticated)");tokio::spawn(async move {loop {@@ -127,9 +324,10 @@ impl TcpTransport {debug!(node = %node_id, %peer_addr, "Incoming TCP connection");let tx2 = tx.clone();let nid = node_id.clone();+ let secret = secret.clone();tokio::spawn(async move {- if let Err(e) = handle_connection(stream, nid.clone(), tx2).await {- debug!(node = %nid, error = %e, "TCP connection closed");+ if let Err(e) = handle_connection(stream, secret, tx2).await {+ debug!(node = %nid, %peer_addr, error = %e, "TCP connection closed");}});}@@ -145,7 +343,8 @@ impl TcpTransport {/// Send a message to a specific peer by node ID.///- /// Opens a fresh TCP connection, writes the frame, then closes.+ /// Opens a fresh TCP connection, completes the authenticated handshake,+ /// writes the frame, then closes.pub async fn send_to(&self, peer_id: &str, msg: &RaftMessage) -> Result<()> {let addr = self.peers@@ -156,14 +355,16 @@ impl TcpTransport {.await.with_context(|| format!("connect to peer {peer_id} at {addr}"))?;- // Send our node_id as a header frame so the receiver knows who sent this.- let header = self.node_id.as_bytes();- let hlen = header.len() as u32;- stream.write_all(&hlen.to_be_bytes()).await?;- stream.write_all(header).await?;-- write_frame(&mut stream, msg).await?;- Ok(())+ // The receiver speaks first: magic, version, challenge. Bound the whole+ // exchange so an accepting-but-silent peer cannot stall the Raft loop.+ tokio::time::timeout(SEND_TIMEOUT, async {+ let challenge = read_handshake(&mut stream).await?;+ write_hello(&mut stream, &self.secret, &challenge, &self.node_id).await?;+ write_frame(&mut stream, &self.secret, &challenge, &self.node_id, 0, msg).await+ })+ .await+ .with_context(|| format!("send to peer {peer_id} at {addr} timed out"))?+ .with_context(|| format!("send to peer {peer_id} at {addr}"))}}@@ -183,35 +384,44 @@ impl ClusterTransport for TcpTransport {// ── Connection handler ────────────────────────────────────────────────────────-/// Handle a single inbound TCP connection: read the sender ID header, then-/// drain all frames and push them into the shared channel.+/// Handle a single inbound TCP connection: issue a challenge, authenticate the+/// hello, then drain authenticated frames into the shared channel.+///+/// Any authentication failure aborts the whole connection — a peer that cannot+/// produce a valid tag does not get to retry on the same socket.async fn handle_connection(mut stream: TcpStream,- _local_node_id: String,+ secret: ClusterSecret,tx: mpsc::Sender<(String, RaftMessage)>,) -> Result<()> {- // Read sender header: [4-byte len][node_id bytes]- let mut hlen_buf = [0u8; 4];- stream- .read_exact(&mut hlen_buf)- .await- .context("read sender header length")?;- let hlen = u32::from_be_bytes(hlen_buf) as usize;- if hlen > 256 {- anyhow::bail!("sender header too large: {} bytes", hlen);- }- let mut hbuf = vec![0u8; hlen];- stream- .read_exact(&mut hbuf)- .await- .context("read sender header")?;- let from = String::from_utf8(hbuf).context("decode sender node_id")?;-- // Read all message frames from this connection.- // Loop exits on EOF or parse error (read_frame returns Err).- while let Ok(msg) = read_frame(&mut stream).await {- if tx.send((from.clone(), msg)).await.is_err() {- break; // receiver dropped+ let challenge = write_handshake(&mut stream).await?;++ let from = tokio::time::timeout(+ HANDSHAKE_TIMEOUT,+ read_hello(&mut stream, &secret, &challenge),+ )+ .await+ .context("peer did not complete the cluster handshake in time")??;++ // Read message frames until EOF, a decode error, or an authentication+ // failure. `seq` pins each frame to its position in the connection.+ let mut seq: u64 = 0;+ loop {+ match read_frame(&mut stream, &secret, &challenge, &from, seq).await {+ Ok(Some(msg)) => {+ if tx.send((from.clone(), msg)).await.is_err() {+ break; // receiver dropped+ }+ seq = seq.saturating_add(1);+ }+ // Clean end of stream — the normal close for a one-frame sender.+ Ok(None) => break,+ Err(e) => {+ // A rejected frame is a security-relevant event, not routine+ // connection churn: log it loudly and drop the connection.+ warn!(peer = %from, error = %e, "cluster frame rejected");+ break;+ }}}diff --git a/engine/crates/xerj-cluster/tests/transport_auth_tests.rs b/engine/crates/xerj-cluster/tests/transport_auth_tests.rsnew file mode 100644index 000000000..b330ec052--- /dev/null+++ b/engine/crates/xerj-cluster/tests/transport_auth_tests.rs@@ -0,0 +1,294 @@+//! Adversarial tests for cluster control-frame authentication (issue #75).+//!+//! Each test drives the real `TcpTransport` listener with a hand-rolled client+//! so it can lie about the things an attacker on the cluster port would lie+//! about: no secret, the wrong secret, a replayed capture, a tampered payload.++use std::collections::HashMap;+use std::net::SocketAddr;+use std::time::Duration;++use tokio::io::{AsyncReadExt, AsyncWriteExt};+use tokio::net::TcpStream;++use xerj_cluster::auth::{ClusterSecret, CHALLENGE_LEN, WIRE_MAGIC, WIRE_VERSION};+use xerj_cluster::node::ClusterTransport;+use xerj_cluster::raft::RaftMessage;+use xerj_cluster::transport::TcpTransport;++const SECRET: &str = "cluster-secret-for-tests-0123456789";+const OTHER_SECRET: &str = "a-completely-different-secret-9876";++/// How long a test waits before concluding "the listener did not deliver this".+const REJECT_WINDOW: Duration = Duration::from_millis(400);++// ── Helpers ───────────────────────────────────────────────────────────────────++async fn free_addr() -> SocketAddr {+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ listener.local_addr().unwrap()+}++fn secret(s: &str) -> ClusterSecret {+ ClusterSecret::new(s).expect("valid test secret")+}++/// Start a listening transport with no peers (receive-only).+async fn listener(addr: SocketAddr, s: &str) -> TcpTransport {+ TcpTransport::new("receiver".to_string(), addr, HashMap::new(), secret(s))+ .await+ .expect("bind receiver transport")+}++fn sample_msg() -> RaftMessage {+ RaftMessage::RequestVote {+ term: 7,+ candidate_id: "attacker".to_string(),+ last_log_index: 0,+ last_log_term: 0,+ }+}++/// Connect and read the receiver's handshake, returning the challenge it issued.+async fn connect_and_handshake(addr: SocketAddr) -> (TcpStream, [u8; CHALLENGE_LEN]) {+ let mut stream = TcpStream::connect(addr).await.expect("connect");+ let mut buf = [0u8; 8 + 1 + CHALLENGE_LEN];+ stream.read_exact(&mut buf).await.expect("read handshake");+ assert_eq!(&buf[..8], WIRE_MAGIC, "handshake magic");+ assert_eq!(buf[8], WIRE_VERSION, "handshake version");+ let mut challenge = [0u8; CHALLENGE_LEN];+ challenge.copy_from_slice(&buf[9..]);+ (stream, challenge)+}++/// The authenticated hello a legitimate sender writes first.+fn hello_bytes(s: &ClusterSecret, node_id: &str, challenge: &[u8; CHALLENGE_LEN]) -> Vec<u8> {+ let mut out = Vec::new();+ let id = node_id.as_bytes();+ out.extend_from_slice(&(id.len() as u32).to_be_bytes());+ out.extend_from_slice(id);+ out.extend_from_slice(&s.hello_tag(challenge, node_id));+ out+}++/// One authenticated message frame at position `seq`.+fn frame_bytes(+ s: &ClusterSecret,+ node_id: &str,+ challenge: &[u8; CHALLENGE_LEN],+ seq: u64,+ msg: &RaftMessage,+) -> Vec<u8> {+ let payload = serde_json::to_vec(msg).unwrap();+ let mut out = Vec::new();+ out.extend_from_slice(&(payload.len() as u32).to_be_bytes());+ out.extend_from_slice(&s.frame_tag(challenge, node_id, seq, &payload));+ out.extend_from_slice(&payload);+ out+}++/// Everything a legitimate sender writes for a single message: hello ‖ frame 0.+fn client_bytes(+ s: &ClusterSecret,+ node_id: &str,+ challenge: &[u8; CHALLENGE_LEN],+ msg: &RaftMessage,+) -> Vec<u8> {+ let mut out = hello_bytes(s, node_id, challenge);+ out.extend_from_slice(&frame_bytes(s, node_id, challenge, 0, msg));+ out+}++/// Assert the transport delivers nothing within the rejection window.+async fn assert_no_delivery(t: &TcpTransport, what: &str) {+ let got = tokio::time::timeout(REJECT_WINDOW, t.recv()).await;+ assert!(+ got.is_err(),+ "{what}: transport accepted a frame it must have rejected: {:?}",+ got.map(|r| r.map(|(from, m)| (from, format!("{m:?}"))))+ );+}++// ── Tests ─────────────────────────────────────────────────────────────────────++/// Baseline: two transports sharing a secret still talk to each other.+#[tokio::test]+async fn authenticated_peers_exchange_messages() {+ let addr_a = free_addr().await;+ let addr_b = free_addr().await;++ let mut peers_a = HashMap::new();+ peers_a.insert("node-b".to_string(), addr_b);++ let a = TcpTransport::new("node-a".to_string(), addr_a, peers_a, secret(SECRET))+ .await+ .unwrap();+ let b = listener(addr_b, SECRET).await;++ a.send_to("node-b", &sample_msg()).await.expect("send");++ let (from, msg) = tokio::time::timeout(Duration::from_secs(2), b.recv())+ .await+ .expect("timed out")+ .expect("recv");+ assert_eq!(from, "node-a");+ assert!(matches!(msg, RaftMessage::RequestVote { term: 7, .. }));+}++/// An unauthenticated client speaking the *old* pre-#75 wire format —+/// `[len][node_id]` then `[len][json]`, no tags — is rejected.+///+/// This is the exact shape of the issue: anyone who could reach the cluster+/// port used to be able to inject Raft control messages this way.+#[tokio::test]+async fn legacy_unauthenticated_client_is_rejected() {+ let addr = free_addr().await;+ let t = listener(addr, SECRET).await;++ let mut stream = TcpStream::connect(addr).await.unwrap();+ // The legacy client writes immediately and never reads the challenge.+ let id = b"evil-node";+ stream+ .write_all(&(id.len() as u32).to_be_bytes())+ .await+ .unwrap();+ stream.write_all(id).await.unwrap();+ let payload = serde_json::to_vec(&sample_msg()).unwrap();+ stream+ .write_all(&(payload.len() as u32).to_be_bytes())+ .await+ .unwrap();+ stream.write_all(&payload).await.unwrap();+ stream.flush().await.unwrap();++ assert_no_delivery(&t, "legacy unauthenticated client").await;+}++/// A client that completes the handshake but holds the wrong secret is rejected+/// at the hello — it never gets to send a frame that counts.+#[tokio::test]+async fn wrong_secret_is_rejected() {+ let addr = free_addr().await;+ let t = listener(addr, SECRET).await;++ let (mut stream, challenge) = connect_and_handshake(addr).await;+ let bytes = client_bytes(+ &secret(OTHER_SECRET),+ "evil-node",+ &challenge,+ &sample_msg(),+ );+ stream.write_all(&bytes).await.unwrap();+ stream.flush().await.unwrap();++ assert_no_delivery(&t, "wrong secret").await;+}++/// A frame captured off the wire cannot be replayed: the receiver issues a+/// fresh challenge per connection, so the captured tags no longer verify.+#[tokio::test]+async fn captured_frame_cannot_be_replayed() {+ let addr = free_addr().await;+ let t = listener(addr, SECRET).await;++ // Connection 1: a legitimate, correctly-signed exchange. Keep the bytes.+ let (mut stream, challenge) = connect_and_handshake(addr).await;+ let captured = client_bytes(&secret(SECRET), "node-a", &challenge, &sample_msg());+ stream.write_all(&captured).await.unwrap();+ stream.flush().await.unwrap();++ let (from, _) = tokio::time::timeout(Duration::from_secs(2), t.recv())+ .await+ .expect("legitimate frame timed out")+ .expect("recv");+ assert_eq!(from, "node-a");+ drop(stream);++ // Connection 2: replay the captured bytes verbatim.+ let (mut replay, _new_challenge) = connect_and_handshake(addr).await;+ replay.write_all(&captured).await.unwrap();+ replay.flush().await.unwrap();++ assert_no_delivery(&t, "replayed capture").await;+}++/// Swapping the payload under an otherwise-valid tag is rejected.+///+/// The substituted payload is *perfectly well-formed* Raft JSON — a vote+/// request at a much higher term, which a leader would honour by stepping+/// down — so nothing but the MAC can catch it.+#[tokio::test]+async fn payload_swapped_under_a_valid_tag_is_rejected() {+ let addr = free_addr().await;+ let t = listener(addr, SECRET).await;++ let (mut stream, challenge) = connect_and_handshake(addr).await;+ let s = secret(SECRET);++ let forged = RaftMessage::RequestVote {+ term: 9_999,+ candidate_id: "attacker".to_string(),+ last_log_index: 0,+ last_log_term: 0,+ };+ let forged_payload = serde_json::to_vec(&forged).unwrap();+ // The tag the attacker captured — computed over the *original* payload.+ let stolen_tag = s.frame_tag(+ &challenge,+ "node-a",+ 0,+ &serde_json::to_vec(&sample_msg()).unwrap(),+ );++ let mut bytes = hello_bytes(&s, "node-a", &challenge);+ bytes.extend_from_slice(&(forged_payload.len() as u32).to_be_bytes());+ bytes.extend_from_slice(&stolen_tag);+ bytes.extend_from_slice(&forged_payload);++ stream.write_all(&bytes).await.unwrap();+ stream.flush().await.unwrap();++ assert_no_delivery(&t, "payload swapped under a stolen tag").await;+}++/// A second frame replayed within the same connection is rejected: `seq` is+/// part of the signed material, so frame 0 cannot be re-sent as frame 1.+#[tokio::test]+async fn duplicated_frame_within_connection_is_rejected() {+ let addr = free_addr().await;+ let t = listener(addr, SECRET).await;++ let (mut stream, challenge) = connect_and_handshake(addr).await;+ let s = secret(SECRET);+ let frame0 = frame_bytes(&s, "node-a", &challenge, 0, &sample_msg());++ // Frame 0 (valid) followed by a byte-identical copy (which lands on seq 1).+ stream+ .write_all(&hello_bytes(&s, "node-a", &challenge))+ .await+ .unwrap();+ stream.write_all(&frame0).await.unwrap();+ stream.write_all(&frame0).await.unwrap();+ stream.flush().await.unwrap();++ // The first frame is legitimate and must arrive.+ let (from, _) = tokio::time::timeout(Duration::from_secs(2), t.recv())+ .await+ .expect("first frame timed out")+ .expect("recv");+ assert_eq!(from, "node-a");++ // The duplicate must not.+ assert_no_delivery(&t, "duplicated frame").await;+}++/// A peer that connects and then goes silent must not pin the accept task+/// forever, and must never be treated as authenticated.+#[tokio::test]+async fn silent_client_delivers_nothing() {+ let addr = free_addr().await;+ let t = listener(addr, SECRET).await;++ let (_stream, _challenge) = connect_and_handshake(addr).await;+ assert_no_delivery(&t, "silent client").await;+}diff --git a/engine/crates/xerj-cluster/tests/transport_tests.rs b/engine/crates/xerj-cluster/tests/transport_tests.rsindex 68323fd3d..b139a2db9 100644--- a/engine/crates/xerj-cluster/tests/transport_tests.rs+++ b/engine/crates/xerj-cluster/tests/transport_tests.rs@@ -6,6 +6,7 @@ use std::time::Duration;use tokio::sync::watch;+use xerj_cluster::auth::ClusterSecret;use xerj_cluster::node::{in_memory::{InMemoryBus, InMemoryTransport},ClusterNode,@@ -38,11 +39,13 @@ async fn test_tcp_transport_send_recv() {let mut peers_b = HashMap::new();peers_b.insert("node-a".to_string(), addr_a);- // Create both transports.- let transport_a = TcpTransport::new("node-a".to_string(), addr_a, peers_a)+ // Create both transports. Both sides must hold the same cluster secret —+ // there is no unauthenticated constructor.+ let secret = || ClusterSecret::new("shared-cluster-secret-for-tests").unwrap();+ let transport_a = TcpTransport::new("node-a".to_string(), addr_a, peers_a, secret()).await.expect("create transport A");- let transport_b = TcpTransport::new("node-b".to_string(), addr_b, peers_b)+ let transport_b = TcpTransport::new("node-b".to_string(), addr_b, peers_b, secret()).await.expect("create transport B");diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex e076debb8..bc7d828cd 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -237,6 +237,10 @@ impl Config {self.engine.validate()?;+ // Cluster: fail closed rather than expose an unauthenticated Raft+ // control port (issue #75).+ self.cluster.validate()?;+// Logging: format must be one of the two supported line formats.let fmt = self.logging.format.as_str();if !fmt.eq_ignore_ascii_case("text") && !fmt.eq_ignore_ascii_case("json") {@@ -1144,7 +1148,7 @@ impl EngineConfig {/// When `enabled = false` (the default), the node runs in single-node mode and/// no Raft or TCP transport is initialised.///-/// **4 settings.**+/// **5 settings.**#[derive(Debug, Clone, Serialize, Deserialize)]#[serde(default, deny_unknown_fields)]pub struct ClusterConfig {@@ -1167,6 +1171,25 @@ pub struct ClusterConfig {/// Controls how often the Raft state machine is driven forward. Lower values/// improve leader election latency at the cost of CPU.pub tick_ms: u64,+ /// Cluster-wide shared secret authenticating every control frame on+ /// `port` (default: empty — which is only legal while `enabled = false`).+ ///+ /// Every node in the cluster must be configured with the **same** secret.+ /// Frames are signed with HMAC-SHA256 over a per-connection challenge, so+ /// a peer that does not hold this secret cannot inject Raft messages, and a+ /// captured frame cannot be replayed.+ ///+ /// May be left empty here and supplied via the `XERJ_CLUSTER_AUTH_SECRET`+ /// environment variable instead — preferable when the config file is baked+ /// into an image. An explicit non-empty value here wins over the+ /// environment.+ ///+ /// **Fail-closed:** with `enabled = true` and no secret from either source,+ /// the node refuses to start. There is no unauthenticated cluster mode.+ ///+ /// Minimum length [`ClusterConfig::MIN_AUTH_SECRET_LEN`]; generate one with+ /// `openssl rand -hex 32`.+ pub auth_secret: String,}impl Default for ClusterConfig {@@ -1176,6 +1199,63 @@ impl Default for ClusterConfig {port: 9300,peers: Vec::new(),tick_ms: 50,+ auth_secret: String::new(),+ }+ }+}++impl ClusterConfig {+ /// Environment variable consulted when `auth_secret` is empty.+ pub const AUTH_SECRET_ENV: &'static str = "XERJ_CLUSTER_AUTH_SECRET";++ /// Minimum accepted secret length, in characters.+ ///+ /// Kept in step with `xerj_cluster::auth::MIN_SECRET_LEN`; the transport+ /// enforces the same floor independently, so a mismatch fails closed.+ pub const MIN_AUTH_SECRET_LEN: usize = 16;++ /// The secret actually in force, or `None` if neither source supplied one.+ ///+ /// Precedence: an explicit non-empty `auth_secret` wins; otherwise+ /// [`Self::AUTH_SECRET_ENV`]. Explicit configuration beats the ambient+ /// environment so that a stray env var cannot silently re-key a cluster.+ pub fn effective_auth_secret(&self) -> Option<String> {+ Self::resolve_auth_secret(&self.auth_secret, std::env::var(Self::AUTH_SECRET_ENV).ok())+ }++ /// Pure resolution rule behind [`Self::effective_auth_secret`], split out so+ /// it can be tested without mutating process-wide environment state.+ pub fn resolve_auth_secret(field: &str, env: Option<String>) -> Option<String> {+ let field = field.trim();+ if !field.is_empty() {+ return Some(field.to_string());+ }+ env.filter(|v| !v.trim().is_empty())+ .map(|v| v.trim().to_string())+ }++ /// Fail-closed validation: cluster mode requires a usable shared secret.+ pub fn validate(&self) -> Result<(), crate::XerjError> {+ if !self.enabled {+ return Ok(());+ }+ match self.effective_auth_secret() {+ None => Err(crate::XerjError::config(format!(+ "cluster.enabled = true requires a shared secret: set cluster.auth_secret \+ or the {} environment variable. Refusing to start an unauthenticated \+ cluster transport — anyone able to reach cluster.port could otherwise \+ inject Raft control messages.",+ Self::AUTH_SECRET_ENV+ ))),+ Some(s) if s.chars().count() < Self::MIN_AUTH_SECRET_LEN => {+ Err(crate::XerjError::config(format!(+ "cluster auth secret is too short ({} chars, minimum {}); \+ generate one with `openssl rand -hex 32`",+ s.chars().count(),+ Self::MIN_AUTH_SECRET_LEN+ )))+ }+ Some(_) => Ok(()),}}}@@ -1518,4 +1598,79 @@ mod tests {let total: usize = 6 + 3 + 3 + 10 + 5 + 3 + 1 + 6 + 2 + 4 + 12 + 3 + 2;assert_eq!(total, 60);}++ // ── Cluster auth: fail closed (issue #75) ────────────────────────────────++ #[test]+ fn cluster_enabled_without_secret_is_rejected() {+ let err = Config::from_toml_str(+ "[cluster]\nenabled = true\nport = 9300\npeers = [\"n2=10.0.0.2:9300\"]\n",+ )+ .expect_err("cluster mode without a shared secret must not load");+ let msg = err.to_string();+ assert!(+ msg.contains("auth_secret") && msg.contains("XERJ_CLUSTER_AUTH_SECRET"),+ "error must name both secret sources, got: {msg}"+ );+ }++ #[test]+ fn cluster_enabled_with_secret_is_accepted() {+ let cfg = Config::from_toml_str(+ "[cluster]\nenabled = true\nauth_secret = \"0123456789abcdef0123\"\n",+ )+ .expect("cluster mode with a secret must load");+ assert!(cfg.cluster.enabled);+ assert_eq!(+ cfg.cluster.effective_auth_secret().as_deref(),+ Some("0123456789abcdef0123")+ );+ }++ #[test]+ fn cluster_short_secret_is_rejected() {+ let err = Config::from_toml_str("[cluster]\nenabled = true\nauth_secret = \"hunter2\"\n")+ .expect_err("a 7-char cluster secret must be rejected");+ assert!(err.to_string().contains("too short"), "{err}");+ }++ #[test]+ fn cluster_disabled_needs_no_secret() {+ // The overwhelmingly common case: cluster mode off, nothing to check.+ Config::from_toml_str("[cluster]\nenabled = false\n").expect("single-node must still load");+ Config::default()+ .validate()+ .expect("defaults must validate");+ }++ #[test]+ fn auth_secret_resolution_prefers_explicit_config() {+ let env = Some("from-environment".to_string());+ // Explicit config wins; a stray env var cannot silently re-key a cluster.+ assert_eq!(+ ClusterConfig::resolve_auth_secret("from-config", env.clone()),+ Some("from-config".to_string())+ );+ // Empty / whitespace-only config falls back to the environment.+ assert_eq!(+ ClusterConfig::resolve_auth_secret("", env.clone()),+ Some("from-environment".to_string())+ );+ assert_eq!(+ ClusterConfig::resolve_auth_secret(" ", env),+ Some("from-environment".to_string())+ );+ // Neither source → None, which is what makes validation fail closed.+ assert_eq!(ClusterConfig::resolve_auth_secret("", None), None);+ assert_eq!(+ ClusterConfig::resolve_auth_secret(" ", Some(" ".to_string())),+ None+ );+ // Surrounding whitespace (e.g. a trailing newline from `$(cat file)`)+ // is configuration noise, not key material.+ assert_eq!(+ ClusterConfig::resolve_auth_secret("", Some(" padded-secret\n".to_string())),+ Some("padded-secret".to_string())+ );+ }}diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex d469d8fdb..0ad6ab8b7 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -1522,6 +1522,22 @@ async fn async_main() -> Result<()> {// 8b. Cluster runner (if cluster mode is enabled)let _cluster_shutdown = if cfg.cluster.enabled {+ // Fail closed (issue #75). The cluster port carries Raft control+ // messages; without a shared secret every frame on it is+ // unauthenticated, so refuse to start rather than open it. This must+ // abort startup — it is deliberately NOT folded into the degraded+ // single-node fallback below, which only covers bind failures.+ let secret_material = cfg.cluster.effective_auth_secret().ok_or_else(|| {+ anyhow::anyhow!(+ "cluster.enabled = true but no cluster auth secret is configured: set \+ cluster.auth_secret or {}. Refusing to start an unauthenticated cluster \+ transport.",+ xerj_common::config::ClusterConfig::AUTH_SECRET_ENV+ )+ })?;+ let cluster_secret = xerj_cluster::ClusterSecret::new(&secret_material)+ .context("cluster auth secret rejected")?;+let (shutdown_tx, shutdown_rx) = tokio::sync::watch::channel(false);// Parse peer list: "node_id=host:port" entries.@@ -1548,7 +1564,7 @@ async fn async_main() -> Result<()> {let tick = std::time::Duration::from_millis(cfg.cluster.tick_ms);- match TcpTransport::new(node_id.clone(), listen_addr, peers).await {+ match TcpTransport::new(node_id.clone(), listen_addr, peers, cluster_secret).await {Ok(transport) => {let peer_ids: Vec<String> = cfg.clusterdiff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex 2d36ab445..9ee6deabf 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -368,6 +368,30 @@ turbo_parallel = true# (e.g. "running" will not match "run").turbo_fast_analyzer = false+# ── Cluster (5 settings) ──────────────────────────────────────────────────────+#+# Multi-node cluster mode is OFF by default and is experimental. The whole+# block is commented out; uncomment it only if you are evaluating clustering.+#+# The cluster port carries Raft control messages. Every frame on it is+# authenticated with a cluster-wide shared secret (HMAC-SHA256 over a+# per-connection challenge), so `auth_secret` is mandatory whenever+# `enabled = true` — a node with cluster mode on and no secret refuses to+# start rather than expose an unauthenticated control port. Supply it here or+# via the XERJ_CLUSTER_AUTH_SECRET environment variable (this file wins if+# both are set), use the SAME value on every node, and generate it with+# `openssl rand -hex 32`. Minimum 16 characters.+#+# Note: the HMAC authenticates frames, it does not encrypt them. Cluster+# traffic is still plaintext — keep the cluster port on a trusted network.+#+# [cluster]+# enabled = true+# port = 9300+# peers = ["n2=10.0.0.2:9300", "n3=10.0.0.3:9300"]+# tick_ms = 50+# auth_secret = ""+# ── Logging (2 settings) ──────────────────────────────────────────────────────[logging]diff --git a/landing/docs/config.html b/landing/docs/config.htmlindex 603fc67eb..d6e9b7363 100644--- a/landing/docs/config.html+++ b/landing/docs/config.html@@ -124,7 +124,7 @@<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p><h2>Full key table</h2>- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"0.0.0.0"</div><div class="cell dsc">Interface to bind every listener to. Use "127.0.0.1" for dev or a private address in production.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">256</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">"size_tiered" or "log_structured". Size-tiered for write-heavy, log-structured for mixed workloads.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Minimum live segments before a merge pass is scheduled. Must be ≥ 2.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">5120</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments larger than this are never merged.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Merge I/O throttle (MiB/s). Set 0 to disable — not recommended in production.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Concurrent merge workers per index. Bump to 2–4 on fast NVMe.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"scalar8"</div><div class="cell dsc">"none" · "scalar8" · "scalar4" · "binary". scalar8 = 4× RAM saving, 1–2% recall loss.</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div></div>+ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"0.0.0.0"</div><div class="cell dsc">Interface to bind every listener to. Use "127.0.0.1" for dev or a private address in production.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">256</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">"size_tiered" or "log_structured". Size-tiered for write-heavy, log-structured for mixed workloads.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Minimum live segments before a merge pass is scheduled. Must be ≥ 2.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">5120</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments larger than this are never merged.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Merge I/O throttle (MiB/s). Set 0 to disable — not recommended in production.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Concurrent merge workers per index. Bump to 2–4 on fast NVMe.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"scalar8"</div><div class="cell dsc">"none" · "scalar8" · "scalar4" · "binary". scalar8 = 4× RAM saving, 1–2% recall loss.</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div><h2 id="server-section">[server]</h2>