Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

security(cluster): authenticate Raft control frames, fail closed without a secret (#113)

xerj-org837a34df3b0c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index cf1d357a9..5d85299db 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -73,6 +73,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Operators terminating TLS or load-balancing in front of XERJ must set
`server.trusted_proxies` to their proxy's address, otherwise every user
behind it shares one rate-limit bucket.
+- **Cluster control frames are now authenticated (issue #75).** The inter-node
+ transport on `cluster.port` accepted any TCP connection that spoke its wire
+ format: anyone who could reach the port could send Raft `RequestVote` /
+ `AppendEntries` frames and steer consensus. Every frame now carries an
+ HMAC-SHA256 tag over a per-connection random challenge issued by the
+ receiver, the sender's node id, the frame's position in the connection, and
+ the payload; tags are compared in constant time, and the tag is verified
+ before the payload reaches the JSON deserialiser. The challenge makes a
+ captured frame non-replayable, including against a different node.
+- **Cluster mode fails closed.** `[cluster] enabled = true` now requires
+ `cluster.auth_secret` (or `XERJ_CLUSTER_AUTH_SECRET`), minimum 16
+ characters. With cluster mode on and no secret from either source the node
+ **refuses to start** instead of running an unauthenticated control port.
+ Single-node mode — the default — is unaffected and needs no secret.
+
+### Breaking
+
+- **The cluster wire format changed and is not backward compatible.** A node
+ running this version cannot talk to a node running rc.9 or earlier in either
+ direction: the receiver now speaks first (magic, version, challenge) where
+ the old protocol expected the sender to open with a length-prefixed JSON
+ frame. Upgrading a cluster requires stopping every node and restarting them
+ on the new version with a shared secret configured — a rolling restart will
+ not interoperate. Only cluster mode is affected; it is off by default, so
+ single-node deployments need no action.
+- Scope note: the HMAC authenticates frames, it does not encrypt them. Cluster
+ traffic is still plaintext and the secret is cluster-wide, so it proves
+ cluster membership, not per-node identity. `cluster.port` still belongs on a
+ trusted network segment; mTLS remains unimplemented.
## [1.0.0-rc.9] - 2026-08-01
diff --git a/docs/case-studies/xerj-self-audit/ISSUES.md b/docs/case-studies/xerj-self-audit/ISSUES.md
index cd56c47db..d83f09cd7 100644
--- a/docs/case-studies/xerj-self-audit/ISSUES.md
+++ b/docs/case-studies/xerj-self-audit/ISSUES.md
@@ -9,7 +9,7 @@ live-reproduction transcripts: [FINDINGS-V2.md](FINDINGS-V2.md).
| [#72](https://github.com/xerj-org/xerj/issues/72) | INJ-01 search-template injection | High | fixed, tracking merge |
| [#73](https://github.com/xerj-org/xerj/issues/73) | F-PATH-02 snapshot location traversal | High | fixed, tracking merge |
| [#74](https://github.com/xerj-org/xerj/issues/74) | S5-1 session-revocation lost update | High | fixed, tracking merge |
-| [#75](https://github.com/xerj-org/xerj/issues/75) | DESER-EGRESS-01 unauth Raft transport | High (gated) | open — Raft-auth Phase 2 |
+| [#75](https://github.com/xerj-org/xerj/issues/75) | DESER-EGRESS-01 unauth Raft transport | High (gated) | fixed, tracking merge — HMAC frame auth + fail-closed config |
| [#76](https://github.com/xerj-org/xerj/issues/76) | Console hardening: S5-4, S5-3, S5-5, AUTHZ-2 | Med/Low/Info | open |
The three Highs were filed publicly only *after* the fix was committed. The
diff --git a/docs/recipes/production-deployment.md b/docs/recipes/production-deployment.md
index 0d5e9d19c..064f8f348 100644
--- a/docs/recipes/production-deployment.md
+++ b/docs/recipes/production-deployment.md
@@ -499,22 +499,34 @@ inter-node consensus and search:
```toml
[cluster]
-enabled = true
-port = 9300
-peers = ["n2=10.0.0.2:9300", "n3=10.0.0.3:9300"]
-tick_ms = 50
+enabled = true
+port = 9300
+peers = ["n2=10.0.0.2:9300", "n3=10.0.0.3:9300"]
+tick_ms = 50
+auth_secret = "…" # or XERJ_CLUSTER_AUTH_SECRET; same value on every node
```
-That inter-node transport is **plaintext and unauthenticated** — it is a plain
-TCP listener with no TLS and no API-key check on the Raft/search messages. The
-API-key auth and in-process TLS described above protect the *client-facing*
-listeners (REST/ES/gRPC); they do **not** extend to `cluster.port`. Anyone who
-can reach `:9300` can participate in the cluster protocol. So if you evaluate
-cluster mode, keep every `cluster.port` on a fully trusted, isolated network
-segment, and treat the feature as experimental — cross-cluster replication and
-disaster recovery are out of scope. For production HA today, mirror via
-snapshot/restore on a schedule, or dual-write to two independent single nodes
-at the application layer.
+`auth_secret` is **mandatory** when `enabled = true`. Every control frame on
+`cluster.port` carries an HMAC-SHA256 tag over a per-connection random
+challenge, so a peer that does not hold the secret cannot inject Raft messages
+and a captured frame cannot be replayed. A node started with cluster mode on
+and no secret — from either the config file or `XERJ_CLUSTER_AUTH_SECRET` —
+**refuses to start** rather than open an unauthenticated control port. Generate
+one with `openssl rand -hex 32` (minimum 16 characters) and use the identical
+value on every node; a node with the wrong secret is simply not admitted.
+
+Two limits to be clear about. First, the HMAC **authenticates, it does not
+encrypt**: cluster traffic is still plaintext JSON, so `cluster.port` still
+belongs on a trusted, isolated network segment, and the API-key auth and TLS
+described above still cover only the client-facing listeners (REST/ES/gRPC).
+Second, the secret is cluster-*wide*, so it proves "this peer is a cluster
+member", not "this peer is node n2" — a compromised member can impersonate any
+other member. Per-node identity needs mTLS, which is not implemented.
+
+Cluster mode remains experimental — cross-cluster replication and disaster
+recovery are out of scope. For production HA today, mirror via snapshot/restore
+on a schedule, or dual-write to two independent single nodes at the application
+layer.
---
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index 470426eba..3d98b8fbc 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -6449,9 +6449,12 @@ dependencies = [
"bytes",
"crc32fast",
"futures",
+ "hmac",
"rand 0.8.5",
"serde",
"serde_json",
+ "sha2 0.10.9",
+ "subtle",
"tempfile",
"thiserror 2.0.18",
"tokio",
diff --git a/engine/crates/xerj-cluster/Cargo.toml b/engine/crates/xerj-cluster/Cargo.toml
index f54087548..0842a0069 100644
--- a/engine/crates/xerj-cluster/Cargo.toml
+++ b/engine/crates/xerj-cluster/Cargo.toml
@@ -22,6 +22,11 @@ rand.workspace = true
async-trait = "0.1"
xxhash-rust.workspace = true
futures = "0.3"
+# Cluster control-frame authentication (issue #75): HMAC-SHA256 over every
+# frame, compared in constant time.
+hmac = "0.12"
+sha2 = "0.10"
+subtle = "2"
[dev-dependencies]
tokio = { version = "1", features = ["full", "test-util"] }
diff --git a/engine/crates/xerj-cluster/src/auth.rs b/engine/crates/xerj-cluster/src/auth.rs
new file mode 100644
index 000000000..cc8c77eaa
--- /dev/null
+++ b/engine/crates/xerj-cluster/src/auth.rs
@@ -0,0 +1,271 @@
+//! Shared-secret authentication for the cluster control transport.
+//!
+//! Every control frame on the cluster port carries an HMAC-SHA256 tag computed
+//! over a domain-separated binding of:
+//!
+//! * a protocol context string (`hello` vs `frame` — domain separation),
+//! * the wire version byte,
+//! * the **per-connection challenge** the receiver generated and sent first,
+//! * the sender's node id,
+//! * the frame's position in the connection (`seq`, implicit — never on the wire),
+//! * the frame payload.
+//!
+//! The challenge is what makes a captured frame useless: it is 32 fresh random
+//! bytes chosen by the *receiver* for every accepted connection, so a recorded
+//! frame replayed to the same node (or reflected at a different node) is
+//! verified against a challenge it was never signed under and is rejected. The
+//! implicit `seq` additionally pins each frame to its position, so frames
+//! within a live connection cannot be reordered, dropped, or duplicated
+//! without the tag failing.
+//!
+//! ## What this does *not* give you
+//!
+//! * **No confidentiality.** Frames are still plaintext JSON on the wire; the
+//! HMAC authenticates, it does not encrypt. Cluster traffic still belongs on
+//! a trusted network segment.
+//! * **No per-node identity.** The secret is cluster-wide, so it proves
+//! "the peer knows the cluster secret", not "the peer is node X". A
+//! compromised member can impersonate any other member. Per-node identity
+//! needs mTLS or per-node keys, which is a larger design change.
+
+use hmac::{Hmac, Mac};
+use sha2::Sha256;
+use subtle::ConstantTimeEq;
+
+type HmacSha256 = Hmac<Sha256>;
+
+// ── Wire constants ────────────────────────────────────────────────────────────
+
+/// Magic bytes that open the receiver's handshake.
+pub const WIRE_MAGIC: &[u8; 8] = b"XERJCLUS";
+
+/// Wire version. Bumped whenever the authenticated framing changes.
+pub const WIRE_VERSION: u8 = 1;
+
+/// Length of the per-connection challenge, in bytes.
+pub const CHALLENGE_LEN: usize = 32;
+
+/// Length of an HMAC-SHA256 tag, in bytes.
+pub const TAG_LEN: usize = 32;
+
+/// Minimum accepted length of a configured cluster secret, in characters.
+///
+/// Short secrets are brute-forceable offline from a single captured frame, so
+/// they are rejected outright rather than accepted with a warning.
+pub const MIN_SECRET_LEN: usize = 16;
+
+/// Domain-separation context for the connection hello.
+const CTX_HELLO: &[u8] = b"xerj-cluster/v1/hello";
+
+/// Domain-separation context for a message frame.
+const CTX_FRAME: &[u8] = b"xerj-cluster/v1/frame";
+
+// ── Errors ────────────────────────────────────────────────────────────────────
+
+/// Failure to build a [`ClusterSecret`] from operator-supplied material.
+#[derive(Debug, thiserror::Error, PartialEq, Eq)]
+pub enum SecretError {
+ /// No secret was supplied at all.
+ #[error(
+ "cluster authentication secret is empty — cluster mode refuses to run \
+ an unauthenticated control transport"
+ )]
+ Empty,
+ /// The secret was supplied but is too short to resist offline guessing.
+ #[error("cluster authentication secret is too short ({got} chars, minimum {MIN_SECRET_LEN})")]
+ TooShort {
+ /// The length that was supplied.
+ got: usize,
+ },
+}
+
+// ── ClusterSecret ─────────────────────────────────────────────────────────────
+
+/// A validated cluster-wide shared secret.
+///
+/// Constructing one is the *only* way to obtain frame tags, which is how the
+/// transport is made structurally incapable of running unauthenticated: there
+/// is no "no secret" variant to fall back to.
+#[derive(Clone)]
+pub struct ClusterSecret {
+ key: Vec<u8>,
+}
+
+impl std::fmt::Debug for ClusterSecret {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ // Never render key material, not even a prefix.
+ f.write_str("ClusterSecret(<redacted>)")
+ }
+}
+
+impl ClusterSecret {
+ /// Validate and adopt an operator-supplied secret.
+ ///
+ /// Surrounding whitespace is trimmed (a trailing newline from
+ /// `XERJ_CLUSTER_AUTH_SECRET=$(cat secret)` is a configuration accident,
+ /// not key material), after which the secret must be at least
+ /// [`MIN_SECRET_LEN`] characters.
+ pub fn new(secret: &str) -> Result<Self, SecretError> {
+ let trimmed = secret.trim();
+ if trimmed.is_empty() {
+ return Err(SecretError::Empty);
+ }
+ if trimmed.chars().count() < MIN_SECRET_LEN {
+ return Err(SecretError::TooShort {
+ got: trimmed.chars().count(),
+ });
+ }
+ Ok(ClusterSecret {
+ key: trimmed.as_bytes().to_vec(),
+ })
+ }
+
+ /// Tag authenticating a connection hello (the sender's node id, bound to
+ /// the receiver's challenge).
+ pub fn hello_tag(&self, challenge: &[u8; CHALLENGE_LEN], sender: &str) -> [u8; TAG_LEN] {
+ let mut mac = self.mac();
+ mac.update(CTX_HELLO);
+ mac.update(&[WIRE_VERSION]);
+ mac.update(challenge);
+ Self::update_lp(&mut mac, sender.as_bytes());
+ mac.finalize().into_bytes().into()
+ }
+
+ /// Tag authenticating one message frame.
+ ///
+ /// `seq` is the zero-based index of this frame within the connection. It is
+ /// never transmitted: both ends count independently, so a replayed,
+ /// dropped, or reordered frame lands on the wrong `seq` and fails.
+ pub fn frame_tag(
+ &self,
+ challenge: &[u8; CHALLENGE_LEN],
+ sender: &str,
+ seq: u64,
+ payload: &[u8],
+ ) -> [u8; TAG_LEN] {
+ let mut mac = self.mac();
+ mac.update(CTX_FRAME);
+ mac.update(&[WIRE_VERSION]);
+ mac.update(challenge);
+ Self::update_lp(&mut mac, sender.as_bytes());
+ mac.update(&seq.to_be_bytes());
+ Self::update_lp(&mut mac, payload);
+ mac.finalize().into_bytes().into()
+ }
+
+ fn mac(&self) -> HmacSha256 {
+ // HMAC accepts a key of any length; this cannot fail.
+ HmacSha256::new_from_slice(&self.key).expect("HMAC accepts keys of any length")
+ }
+
+ /// Feed a length-prefixed field so that concatenation is unambiguous
+ /// (`"ab" || "c"` must not tag the same as `"a" || "bc"`).
+ fn update_lp(mac: &mut HmacSha256, bytes: &[u8]) {
+ mac.update(&(bytes.len() as u64).to_be_bytes());
+ mac.update(bytes);
+ }
+}
+
+// ── Constant-time comparison ──────────────────────────────────────────────────
+
+/// Compare a computed tag against a tag read off the wire in constant time.
+///
+/// A plain `==` on the byte arrays would short-circuit on the first differing
+/// byte and leak, through timing, how many leading bytes an attacker guessed
+/// correctly — enough to forge a tag byte-by-byte. Length is not secret, so the
+/// length check may short-circuit.
+#[must_use]
+pub fn tags_match(expected: &[u8; TAG_LEN], actual: &[u8]) -> bool {
+ if actual.len() != TAG_LEN {
+ return false;
+ }
+ expected.ct_eq(actual).into()
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const SECRET: &str = "a-perfectly-fine-cluster-secret";
+
+ fn challenge(b: u8) -> [u8; CHALLENGE_LEN] {
+ [b; CHALLENGE_LEN]
+ }
+
+ #[test]
+ fn rejects_empty_and_short_secrets() {
+ assert_eq!(ClusterSecret::new("").unwrap_err(), SecretError::Empty);
+ assert_eq!(ClusterSecret::new(" \n").unwrap_err(), SecretError::Empty);
+ assert_eq!(
+ ClusterSecret::new("short").unwrap_err(),
+ SecretError::TooShort { got: 5 }
+ );
+ assert!(ClusterSecret::new(SECRET).is_ok());
+ }
+
+ #[test]
+ fn secret_debug_never_leaks_key_material() {
+ let s = ClusterSecret::new(SECRET).unwrap();
+ let rendered = format!("{s:?}");
+ assert!(
+ !rendered.contains("cluster-secret"),
+ "Debug leaked key material: {rendered}"
+ );
+ assert!(rendered.contains("redacted"));
+ }
+
+ #[test]
+ fn tag_binds_challenge_sender_seq_and_payload() {
+ let s = ClusterSecret::new(SECRET).unwrap();
+ let base = s.frame_tag(&challenge(1), "node-a", 0, b"payload");
+
+ assert_ne!(base, s.frame_tag(&challenge(2), "node-a", 0, b"payload"));
+ assert_ne!(base, s.frame_tag(&challenge(1), "node-b", 0, b"payload"));
+ assert_ne!(base, s.frame_tag(&challenge(1), "node-a", 1, b"payload"));
+ assert_ne!(base, s.frame_tag(&challenge(1), "node-a", 0, b"payloa!"));
+
+ // Different secret → different tag.
+ let other = ClusterSecret::new("an-entirely-different-secret").unwrap();
+ assert_ne!(
+ base,
+ other.frame_tag(&challenge(1), "node-a", 0, b"payload")
+ );
+
+ // Same inputs → same tag.
+ assert_eq!(base, s.frame_tag(&challenge(1), "node-a", 0, b"payload"));
+ }
+
+ #[test]
+ fn hello_and_frame_contexts_are_separated() {
+ let s = ClusterSecret::new(SECRET).unwrap();
+ let hello = s.hello_tag(&challenge(7), "node-a");
+ let frame = s.frame_tag(&challenge(7), "node-a", 0, b"");
+ assert_ne!(hello, frame, "hello and frame tags must not be confusable");
+ }
+
+ #[test]
+ fn length_prefixing_prevents_field_smuggling() {
+ let s = ClusterSecret::new(SECRET).unwrap();
+ // "ab" + payload "c" must not collide with "a" + payload "bc".
+ assert_ne!(
+ s.frame_tag(&challenge(0), "ab", 0, b"c"),
+ s.frame_tag(&challenge(0), "a", 0, b"bc")
+ );
+ }
+
+ #[test]
+ fn tags_match_accepts_equal_and_rejects_everything_else() {
+ let s = ClusterSecret::new(SECRET).unwrap();
+ let tag = s.frame_tag(&challenge(3), "n", 0, b"x");
+ assert!(tags_match(&tag, &tag));
+
+ let mut flipped = tag;
+ flipped[TAG_LEN - 1] ^= 0x01;
+ assert!(!tags_match(&tag, &flipped));
+
+ // Truncated and over-long tags are rejected without panicking.
+ assert!(!tags_match(&tag, &tag[..TAG_LEN - 1]));
+ assert!(!tags_match(&tag, &[0u8; TAG_LEN + 1]));
+ assert!(!tags_match(&tag, &[]));
+ }
+}
diff --git a/engine/crates/xerj-cluster/src/lib.rs b/engine/crates/xerj-cluster/src/lib.rs
index 2100a0439..e106d4b58 100644
--- a/engine/crates/xerj-cluster/src/lib.rs
+++ b/engine/crates/xerj-cluster/src/lib.rs
@@ -46,6 +46,7 @@
//! }
//! ```
+pub mod auth;
pub mod coordinator;
pub mod metadata;
pub mod node;
@@ -60,6 +61,7 @@ pub mod transport;
// ── Convenience re-exports ────────────────────────────────────────────────────
+pub use auth::{ClusterSecret, SecretError};
pub use coordinator::{
IndexResponse, LocalSearcher, MergedSearchResult, SearchCoordinator, SearchTransport,
};
diff --git a/engine/crates/xerj-cluster/src/transport.rs b/engine/crates/xerj-cluster/src/transport.rs
index fc6d4bf06..b3f485b1a 100644
--- a/engine/crates/xerj-cluster/src/transport.rs
+++ b/engine/crates/xerj-cluster/src/transport.rs
@@ -1,35 +1,202 @@
//! TCP-based transport for inter-node communication.
//!
-//! Wire format: `[4-byte big-endian length][JSON-serialized RaftMessage]`
+//! Every connection is authenticated with the cluster-wide shared secret
+//! (HMAC-SHA256, see [`crate::auth`]). There is no unauthenticated mode: the
+//! transport cannot be constructed without a validated [`ClusterSecret`].
//!
-//! Each connection carries messages from one peer to this node. The sender
-//! writes length-prefixed JSON frames; the receiver decodes them and pushes
-//! them onto the shared `incoming` channel.
+//! ## Wire format (version 1)
+//!
+//! ```text
+//! receiver → sender (handshake, sent immediately on accept)
+//! [8] magic "XERJCLUS"
+//! [1] wire version (1)
+//! [32] challenge — fresh random bytes, per connection
+//!
+//! sender → receiver (hello, exactly once)
+//! [4] node_id length (u32 BE, ≤ 256)
+//! [n] node_id (UTF-8)
+//! [32] HMAC tag over (hello-context, version, challenge, node_id)
+//!
+//! sender → receiver (message frame, repeated until EOF)
+//! [4] payload length (u32 BE, ≤ 10 MiB)
+//! [32] HMAC tag over (frame-context, version, challenge, node_id, seq, payload)
+//! [n] payload — JSON-serialised RaftMessage
+//! ```
+//!
+//! `seq` is the frame's zero-based index within the connection and is never
+//! transmitted; both ends count it independently.
+//!
+//! A frame's tag is verified **before** its payload is handed to the JSON
+//! deserialiser, so unauthenticated bytes never reach `serde_json`.
+//!
+//! ## Wire compatibility
+//!
+//! This framing is **not** compatible with the pre-authentication format. A
+//! node running this version cannot talk to a node running an older one, in
+//! either direction: the old sender writes a JSON frame where the new receiver
+//! expects to write a challenge first, and the new sender waits for a handshake
+//! an old receiver never sends. Upgrading a cluster requires a full stop/start,
+//! not a rolling restart.
use std::collections::HashMap;
use std::net::SocketAddr;
use std::sync::Arc;
+use std::time::Duration;
use anyhow::{Context, Result};
use async_trait::async_trait;
+use rand::RngCore;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
use tokio::sync::{mpsc, Mutex};
use tracing::{debug, info, warn};
+use crate::auth::{tags_match, ClusterSecret, CHALLENGE_LEN, TAG_LEN, WIRE_MAGIC, WIRE_VERSION};
use crate::node::ClusterTransport;
use crate::raft::RaftMessage;
+/// Largest accepted frame payload (10 MiB).
+const MAX_FRAME_BYTES: usize = 10 * 1024 * 1024;
+
+/// Largest accepted node-id length in the hello header.
+const MAX_NODE_ID_BYTES: usize = 256;
+
+/// How long the receiver will wait for a connected peer to complete its hello.
+///
+/// Without this, a peer that connects and then goes silent pins an accept task
+/// (and its buffers) indefinitely — cheap for an attacker, expensive for us.
+const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// How long a send may take once the TCP connection is established.
+///
+/// Authentication makes the sender *read* before it writes (it needs the
+/// receiver's challenge), so a peer that accepts connections and then goes
+/// silent could otherwise stall the Raft loop indefinitely. The whole
+/// post-connect exchange is bounded instead.
+const SEND_TIMEOUT: Duration = Duration::from_secs(5);
+
+/// Length of the receiver's handshake: magic ‖ version ‖ challenge.
+const HANDSHAKE_LEN: usize = WIRE_MAGIC.len() + 1 + CHALLENGE_LEN;
+
// ── Wire helpers ──────────────────────────────────────────────────────────────
-/// Write a single length-prefixed JSON frame to the given stream.
-async fn write_frame(stream: &mut TcpStream, msg: &RaftMessage) -> Result<()> {
+/// Write the receiver-side handshake and return the challenge it carried.
+async fn write_handshake(stream: &mut TcpStream) -> Result<[u8; CHALLENGE_LEN]> {
+ let mut challenge = [0u8; CHALLENGE_LEN];
+ rand::rngs::OsRng.fill_bytes(&mut challenge);
+
+ let mut buf = [0u8; HANDSHAKE_LEN];
+ buf[..WIRE_MAGIC.len()].copy_from_slice(WIRE_MAGIC);
+ buf[WIRE_MAGIC.len()] = WIRE_VERSION;
+ buf[WIRE_MAGIC.len() + 1..].copy_from_slice(&challenge);
+
+ stream.write_all(&buf).await.context("write handshake")?;
+ stream.flush().await.context("flush handshake")?;
+ Ok(challenge)
+}
+
+/// Read and validate the receiver's handshake, returning the challenge.
+async fn read_handshake(stream: &mut TcpStream) -> Result<[u8; CHALLENGE_LEN]> {
+ let mut buf = [0u8; HANDSHAKE_LEN];
+ stream
+ .read_exact(&mut buf)
+ .await
+ .context("read handshake")?;
+
+ if &buf[..WIRE_MAGIC.len()] != WIRE_MAGIC {
+ anyhow::bail!("peer did not send a xerj cluster handshake (bad magic)");
+ }
+ let version = buf[WIRE_MAGIC.len()];
+ if version != WIRE_VERSION {
+ anyhow::bail!(
+ "cluster wire version mismatch: peer speaks v{version}, this node speaks v{WIRE_VERSION}"
+ );
+ }
+
+ let mut challenge = [0u8; CHALLENGE_LEN];
+ challenge.copy_from_slice(&buf[WIRE_MAGIC.len() + 1..]);
+ Ok(challenge)
+}
+
+/// Write the authenticated hello identifying this node.
+async fn write_hello(
+ stream: &mut TcpStream,
+ secret: &ClusterSecret,
+ challenge: &[u8; CHALLENGE_LEN],
+ node_id: &str,
+) -> Result<()> {
+ let id = node_id.as_bytes();
+ if id.len() > MAX_NODE_ID_BYTES {
+ anyhow::bail!("node_id too long: {} bytes", id.len());
+ }
+ let tag = secret.hello_tag(challenge, node_id);
+
+ stream
+ .write_all(&(id.len() as u32).to_be_bytes())
+ .await
+ .context("write hello length")?;
+ stream.write_all(id).await.context("write hello node_id")?;
+ stream.write_all(&tag).await.context("write hello tag")?;
+ stream.flush().await.context("flush hello")?;
+ Ok(())
+}
+
+/// Read and authenticate the peer's hello, returning its node id.
+async fn read_hello(
+ stream: &mut TcpStream,
+ secret: &ClusterSecret,
+ challenge: &[u8; CHALLENGE_LEN],
+) -> Result<String> {
+ let mut len_buf = [0u8; 4];
+ stream
+ .read_exact(&mut len_buf)
+ .await
+ .context("read hello length")?;
+ let len = u32::from_be_bytes(len_buf) as usize;
+ if len > MAX_NODE_ID_BYTES {
+ anyhow::bail!("hello node_id too large: {len} bytes");
+ }
+
+ let mut id_buf = vec![0u8; len];
+ stream
+ .read_exact(&mut id_buf)
+ .await
+ .context("read hello node_id")?;
+ let mut tag = [0u8; TAG_LEN];
+ stream
+ .read_exact(&mut tag)
+ .await
+ .context("read hello tag")?;
+
+ // Authenticate the raw bytes before trusting them as a UTF-8 node id.
+ let from = String::from_utf8(id_buf).context("decode hello node_id")?;
+ let expected = secret.hello_tag(challenge, &from);
+ if !tags_match(&expected, &tag) {
+ anyhow::bail!("cluster authentication failed: bad hello tag");
+ }
+ Ok(from)
+}
+
+/// Write a single authenticated frame.
+async fn write_frame(
+ stream: &mut TcpStream,
+ secret: &ClusterSecret,
+ challenge: &[u8; CHALLENGE_LEN],
+ node_id: &str,
+ seq: u64,
+ msg: &RaftMessage,
+) -> Result<()> {
let payload = serde_json::to_vec(msg).context("serialize RaftMessage")?;
- let len = payload.len() as u32;
+ if payload.len() > MAX_FRAME_BYTES {
+ anyhow::bail!("frame too large to send: {} bytes", payload.len());
+ }
+ let tag = secret.frame_tag(challenge, node_id, seq, &payload);
+
stream
- .write_all(&len.to_be_bytes())
+ .write_all(&(payload.len() as u32).to_be_bytes())
.await
.context("write frame length")?;
+ stream.write_all(&tag).await.context("write frame tag")?;
stream
.write_all(&payload)
.await
@@ -38,38 +205,63 @@ async fn write_frame(stream: &mut TcpStream, msg: &RaftMessage) -> Result<()> {
Ok(())
}
-/// Read a single length-prefixed JSON frame from the given stream.
-async fn read_frame(stream: &mut TcpStream) -> Result<RaftMessage> {
+/// Read, authenticate, and decode a single frame.
+///
+/// Returns `Ok(None)` on a clean end of stream (the peer closed after its last
+/// frame, which is the normal case for the connection-per-send sender).
+///
+/// The tag is verified before the payload is deserialised, so a forged or
+/// tampered frame never reaches `serde_json`.
+async fn read_frame(
+ stream: &mut TcpStream,
+ secret: &ClusterSecret,
+ challenge: &[u8; CHALLENGE_LEN],
+ from: &str,
+ seq: u64,
+) -> Result<Option<RaftMessage>> {
let mut len_buf = [0u8; 4];
- stream
- .read_exact(&mut len_buf)
- .await
- .context("read frame length")?;
+ match stream.read_exact(&mut len_buf).await {
+ Ok(_) => {}
+ Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => return Ok(None),
+ Err(e) => return Err(anyhow::Error::new(e).context("read frame length")),
+ }
let len = u32::from_be_bytes(len_buf) as usize;
-
- // Guard against absurdly large frames (e.g. 10 MiB max).
- if len > 10 * 1024 * 1024 {
- anyhow::bail!("frame too large: {} bytes", len);
+ if len > MAX_FRAME_BYTES {
+ anyhow::bail!("frame too large: {len} bytes");
}
+ let mut tag = [0u8; TAG_LEN];
+ stream
+ .read_exact(&mut tag)
+ .await
+ .context("read frame tag")?;
let mut payload = vec![0u8; len];
stream
.read_exact(&mut payload)
.await
.context("read frame payload")?;
- let msg: RaftMessage = serde_json::from_slice(&payload).context("deserialize RaftMessage")?;
- Ok(msg)
+
+ let expected = secret.frame_tag(challenge, from, seq, &payload);
+ if !tags_match(&expected, &tag) {
+ anyhow::bail!("cluster authentication failed: bad frame tag (seq {seq})");
+ }
+
+ let msg = serde_json::from_slice(&payload).context("deserialize RaftMessage")?;
+ Ok(Some(msg))
}
// ── TcpTransport ─────────────────────────────────────────────────────────────
/// TCP-based transport for inter-node communication.
///
-/// Wire format: `[4-byte big-endian length][JSON-serialized RaftMessage]`
-///
/// Incoming messages arrive via a background listener task and are delivered
/// through an mpsc channel. Outgoing messages open a fresh TCP connection per
/// send (connection pooling is a future optimisation).
+///
+/// Every connection is authenticated in both directions of setup: the receiver
+/// proves nothing (it holds no identity beyond the secret) but issues a
+/// challenge, and the sender proves knowledge of the shared secret on the hello
+/// and on every frame.
pub struct TcpTransport {
/// This node's identifier.
pub node_id: String,
@@ -77,6 +269,8 @@ pub struct TcpTransport {
listen_addr: SocketAddr,
/// Map of peer node_id → socket address.
peers: Arc<HashMap<String, SocketAddr>>,
+ /// Cluster-wide shared secret used to authenticate every frame.
+ secret: ClusterSecret,
/// Receives `(sender_node_id, msg)` from the background listener.
incoming: Arc<Mutex<mpsc::Receiver<(String, RaftMessage)>>>,
// The sender half is kept alive so the channel is not closed when the
@@ -88,12 +282,13 @@ pub struct TcpTransport {
impl TcpTransport {
/// Create a new TCP transport and begin listening for inbound connections.
///
- /// The listener task is spawned immediately; call [`start`] is not needed
- /// (this constructor already binds and spawns).
+ /// `secret` is the cluster-wide shared secret. It is required: there is no
+ /// constructor that yields an unauthenticated transport.
pub async fn new(
node_id: String,
listen_addr: SocketAddr,
peers: HashMap<String, SocketAddr>,
+ secret: ClusterSecret,
) -> Result<Self> {
let (tx, rx) = mpsc::channel::<(String, RaftMessage)>(1024);
@@ -101,6 +296,7 @@ impl TcpTransport {
node_id: node_id.clone(),
listen_addr,
peers: Arc::new(peers),
+ secret,
incoming: Arc::new(Mutex::new(rx)),
sender: tx.clone(),
};
@@ -118,7 +314,8 @@ impl TcpTransport {
.with_context(|| format!("bind TCP transport on {}", self.listen_addr))?;
let node_id = self.node_id.clone();
- info!(node = %node_id, addr = %self.listen_addr, "TCP transport listening");
+ let secret = self.secret.clone();
+ info!(node = %node_id, addr = %self.listen_addr, "TCP transport listening (authenticated)");
tokio::spawn(async move {
loop {
@@ -127,9 +324,10 @@ impl TcpTransport {
debug!(node = %node_id, %peer_addr, "Incoming TCP connection");
let tx2 = tx.clone();
let nid = node_id.clone();
+ let secret = secret.clone();
tokio::spawn(async move {
- if let Err(e) = handle_connection(stream, nid.clone(), tx2).await {
- debug!(node = %nid, error = %e, "TCP connection closed");
+ if let Err(e) = handle_connection(stream, secret, tx2).await {
+ debug!(node = %nid, %peer_addr, error = %e, "TCP connection closed");
}
});
}
@@ -145,7 +343,8 @@ impl TcpTransport {
/// Send a message to a specific peer by node ID.
///
- /// Opens a fresh TCP connection, writes the frame, then closes.
+ /// Opens a fresh TCP connection, completes the authenticated handshake,
+ /// writes the frame, then closes.
pub async fn send_to(&self, peer_id: &str, msg: &RaftMessage) -> Result<()> {
let addr = self
.peers
@@ -156,14 +355,16 @@ impl TcpTransport {
.await
.with_context(|| format!("connect to peer {peer_id} at {addr}"))?;
- // Send our node_id as a header frame so the receiver knows who sent this.
- let header = self.node_id.as_bytes();
- let hlen = header.len() as u32;
- stream.write_all(&hlen.to_be_bytes()).await?;
- stream.write_all(header).await?;
-
- write_frame(&mut stream, msg).await?;
- Ok(())
+ // The receiver speaks first: magic, version, challenge. Bound the whole
+ // exchange so an accepting-but-silent peer cannot stall the Raft loop.
+ tokio::time::timeout(SEND_TIMEOUT, async {
+ let challenge = read_handshake(&mut stream).await?;
+ write_hello(&mut stream, &self.secret, &challenge, &self.node_id).await?;
+ write_frame(&mut stream, &self.secret, &challenge, &self.node_id, 0, msg).await
+ })
+ .await
+ .with_context(|| format!("send to peer {peer_id} at {addr} timed out"))?
+ .with_context(|| format!("send to peer {peer_id} at {addr}"))
}
}
@@ -183,35 +384,44 @@ impl ClusterTransport for TcpTransport {
// ── Connection handler ────────────────────────────────────────────────────────
-/// Handle a single inbound TCP connection: read the sender ID header, then
-/// drain all frames and push them into the shared channel.
+/// Handle a single inbound TCP connection: issue a challenge, authenticate the
+/// hello, then drain authenticated frames into the shared channel.
+///
+/// Any authentication failure aborts the whole connection — a peer that cannot
+/// produce a valid tag does not get to retry on the same socket.
async fn handle_connection(
mut stream: TcpStream,
- _local_node_id: String,
+ secret: ClusterSecret,
tx: mpsc::Sender<(String, RaftMessage)>,
) -> Result<()> {
- // Read sender header: [4-byte len][node_id bytes]
- let mut hlen_buf = [0u8; 4];
- stream
- .read_exact(&mut hlen_buf)
- .await
- .context("read sender header length")?;
- let hlen = u32::from_be_bytes(hlen_buf) as usize;
- if hlen > 256 {
- anyhow::bail!("sender header too large: {} bytes", hlen);
- }
- let mut hbuf = vec![0u8; hlen];
- stream
- .read_exact(&mut hbuf)
- .await
- .context("read sender header")?;
- let from = String::from_utf8(hbuf).context("decode sender node_id")?;
-
- // Read all message frames from this connection.
- // Loop exits on EOF or parse error (read_frame returns Err).
- while let Ok(msg) = read_frame(&mut stream).await {
- if tx.send((from.clone(), msg)).await.is_err() {
- break; // receiver dropped
+ let challenge = write_handshake(&mut stream).await?;
+
+ let from = tokio::time::timeout(
+ HANDSHAKE_TIMEOUT,
+ read_hello(&mut stream, &secret, &challenge),
+ )
+ .await
+ .context("peer did not complete the cluster handshake in time")??;
+
+ // Read message frames until EOF, a decode error, or an authentication
+ // failure. `seq` pins each frame to its position in the connection.
+ let mut seq: u64 = 0;
+ loop {
+ match read_frame(&mut stream, &secret, &challenge, &from, seq).await {
+ Ok(Some(msg)) => {
+ if tx.send((from.clone(), msg)).await.is_err() {
+ break; // receiver dropped
+ }
+ seq = seq.saturating_add(1);
+ }
+ // Clean end of stream — the normal close for a one-frame sender.
+ Ok(None) => break,
+ Err(e) => {
+ // A rejected frame is a security-relevant event, not routine
+ // connection churn: log it loudly and drop the connection.
+ warn!(peer = %from, error = %e, "cluster frame rejected");
+ break;
+ }
}
}
diff --git a/engine/crates/xerj-cluster/tests/transport_auth_tests.rs b/engine/crates/xerj-cluster/tests/transport_auth_tests.rs
new file mode 100644
index 000000000..b330ec052
--- /dev/null
+++ b/engine/crates/xerj-cluster/tests/transport_auth_tests.rs
@@ -0,0 +1,294 @@
+//! Adversarial tests for cluster control-frame authentication (issue #75).
+//!
+//! Each test drives the real `TcpTransport` listener with a hand-rolled client
+//! so it can lie about the things an attacker on the cluster port would lie
+//! about: no secret, the wrong secret, a replayed capture, a tampered payload.
+
+use std::collections::HashMap;
+use std::net::SocketAddr;
+use std::time::Duration;
+
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::net::TcpStream;
+
+use xerj_cluster::auth::{ClusterSecret, CHALLENGE_LEN, WIRE_MAGIC, WIRE_VERSION};
+use xerj_cluster::node::ClusterTransport;
+use xerj_cluster::raft::RaftMessage;
+use xerj_cluster::transport::TcpTransport;
+
+const SECRET: &str = "cluster-secret-for-tests-0123456789";
+const OTHER_SECRET: &str = "a-completely-different-secret-9876";
+
+/// How long a test waits before concluding "the listener did not deliver this".
+const REJECT_WINDOW: Duration = Duration::from_millis(400);
+
+// ── Helpers ───────────────────────────────────────────────────────────────────
+
+async fn free_addr() -> SocketAddr {
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ listener.local_addr().unwrap()
+}
+
+fn secret(s: &str) -> ClusterSecret {
+ ClusterSecret::new(s).expect("valid test secret")
+}
+
+/// Start a listening transport with no peers (receive-only).
+async fn listener(addr: SocketAddr, s: &str) -> TcpTransport {
+ TcpTransport::new("receiver".to_string(), addr, HashMap::new(), secret(s))
+ .await
+ .expect("bind receiver transport")
+}
+
+fn sample_msg() -> RaftMessage {
+ RaftMessage::RequestVote {
+ term: 7,
+ candidate_id: "attacker".to_string(),
+ last_log_index: 0,
+ last_log_term: 0,
+ }
+}
+
+/// Connect and read the receiver's handshake, returning the challenge it issued.
+async fn connect_and_handshake(addr: SocketAddr) -> (TcpStream, [u8; CHALLENGE_LEN]) {
+ let mut stream = TcpStream::connect(addr).await.expect("connect");
+ let mut buf = [0u8; 8 + 1 + CHALLENGE_LEN];
+ stream.read_exact(&mut buf).await.expect("read handshake");
+ assert_eq!(&buf[..8], WIRE_MAGIC, "handshake magic");
+ assert_eq!(buf[8], WIRE_VERSION, "handshake version");
+ let mut challenge = [0u8; CHALLENGE_LEN];
+ challenge.copy_from_slice(&buf[9..]);
+ (stream, challenge)
+}
+
+/// The authenticated hello a legitimate sender writes first.
+fn hello_bytes(s: &ClusterSecret, node_id: &str, challenge: &[u8; CHALLENGE_LEN]) -> Vec<u8> {
+ let mut out = Vec::new();
+ let id = node_id.as_bytes();
+ out.extend_from_slice(&(id.len() as u32).to_be_bytes());
+ out.extend_from_slice(id);
+ out.extend_from_slice(&s.hello_tag(challenge, node_id));
+ out
+}
+
+/// One authenticated message frame at position `seq`.
+fn frame_bytes(
+ s: &ClusterSecret,
+ node_id: &str,
+ challenge: &[u8; CHALLENGE_LEN],
+ seq: u64,
+ msg: &RaftMessage,
+) -> Vec<u8> {
+ let payload = serde_json::to_vec(msg).unwrap();
+ let mut out = Vec::new();
+ out.extend_from_slice(&(payload.len() as u32).to_be_bytes());
+ out.extend_from_slice(&s.frame_tag(challenge, node_id, seq, &payload));
+ out.extend_from_slice(&payload);
+ out
+}
+
+/// Everything a legitimate sender writes for a single message: hello ‖ frame 0.
+fn client_bytes(
+ s: &ClusterSecret,
+ node_id: &str,
+ challenge: &[u8; CHALLENGE_LEN],
+ msg: &RaftMessage,
+) -> Vec<u8> {
+ let mut out = hello_bytes(s, node_id, challenge);
+ out.extend_from_slice(&frame_bytes(s, node_id, challenge, 0, msg));
+ out
+}
+
+/// Assert the transport delivers nothing within the rejection window.
+async fn assert_no_delivery(t: &TcpTransport, what: &str) {
+ let got = tokio::time::timeout(REJECT_WINDOW, t.recv()).await;
+ assert!(
+ got.is_err(),
+ "{what}: transport accepted a frame it must have rejected: {:?}",
+ got.map(|r| r.map(|(from, m)| (from, format!("{m:?}"))))
+ );
+}
+
+// ── Tests ─────────────────────────────────────────────────────────────────────
+
+/// Baseline: two transports sharing a secret still talk to each other.
+#[tokio::test]
+async fn authenticated_peers_exchange_messages() {
+ let addr_a = free_addr().await;
+ let addr_b = free_addr().await;
+
+ let mut peers_a = HashMap::new();
+ peers_a.insert("node-b".to_string(), addr_b);
+
+ let a = TcpTransport::new("node-a".to_string(), addr_a, peers_a, secret(SECRET))
+ .await
+ .unwrap();
+ let b = listener(addr_b, SECRET).await;
+
+ a.send_to("node-b", &sample_msg()).await.expect("send");
+
+ let (from, msg) = tokio::time::timeout(Duration::from_secs(2), b.recv())
+ .await
+ .expect("timed out")
+ .expect("recv");
+ assert_eq!(from, "node-a");
+ assert!(matches!(msg, RaftMessage::RequestVote { term: 7, .. }));
+}
+
+/// An unauthenticated client speaking the *old* pre-#75 wire format —
+/// `[len][node_id]` then `[len][json]`, no tags — is rejected.
+///
+/// This is the exact shape of the issue: anyone who could reach the cluster
+/// port used to be able to inject Raft control messages this way.
+#[tokio::test]
+async fn legacy_unauthenticated_client_is_rejected() {
+ let addr = free_addr().await;
+ let t = listener(addr, SECRET).await;
+
+ let mut stream = TcpStream::connect(addr).await.unwrap();
+ // The legacy client writes immediately and never reads the challenge.
+ let id = b"evil-node";
+ stream
+ .write_all(&(id.len() as u32).to_be_bytes())
+ .await
+ .unwrap();
+ stream.write_all(id).await.unwrap();
+ let payload = serde_json::to_vec(&sample_msg()).unwrap();
+ stream
+ .write_all(&(payload.len() as u32).to_be_bytes())
+ .await
+ .unwrap();
+ stream.write_all(&payload).await.unwrap();
+ stream.flush().await.unwrap();
+
+ assert_no_delivery(&t, "legacy unauthenticated client").await;
+}
+
+/// A client that completes the handshake but holds the wrong secret is rejected
+/// at the hello — it never gets to send a frame that counts.
+#[tokio::test]
+async fn wrong_secret_is_rejected() {
+ let addr = free_addr().await;
+ let t = listener(addr, SECRET).await;
+
+ let (mut stream, challenge) = connect_and_handshake(addr).await;
+ let bytes = client_bytes(
+ &secret(OTHER_SECRET),
+ "evil-node",
+ &challenge,
+ &sample_msg(),
+ );
+ stream.write_all(&bytes).await.unwrap();
+ stream.flush().await.unwrap();
+
+ assert_no_delivery(&t, "wrong secret").await;
+}
+
+/// A frame captured off the wire cannot be replayed: the receiver issues a
+/// fresh challenge per connection, so the captured tags no longer verify.
+#[tokio::test]
+async fn captured_frame_cannot_be_replayed() {
+ let addr = free_addr().await;
+ let t = listener(addr, SECRET).await;
+
+ // Connection 1: a legitimate, correctly-signed exchange. Keep the bytes.
+ let (mut stream, challenge) = connect_and_handshake(addr).await;
+ let captured = client_bytes(&secret(SECRET), "node-a", &challenge, &sample_msg());
+ stream.write_all(&captured).await.unwrap();
+ stream.flush().await.unwrap();
+
+ let (from, _) = tokio::time::timeout(Duration::from_secs(2), t.recv())
+ .await
+ .expect("legitimate frame timed out")
+ .expect("recv");
+ assert_eq!(from, "node-a");
+ drop(stream);
+
+ // Connection 2: replay the captured bytes verbatim.
+ let (mut replay, _new_challenge) = connect_and_handshake(addr).await;
+ replay.write_all(&captured).await.unwrap();
+ replay.flush().await.unwrap();
+
+ assert_no_delivery(&t, "replayed capture").await;
+}
+
+/// Swapping the payload under an otherwise-valid tag is rejected.
+///
+/// The substituted payload is *perfectly well-formed* Raft JSON — a vote
+/// request at a much higher term, which a leader would honour by stepping
+/// down — so nothing but the MAC can catch it.
+#[tokio::test]
+async fn payload_swapped_under_a_valid_tag_is_rejected() {
+ let addr = free_addr().await;
+ let t = listener(addr, SECRET).await;
+
+ let (mut stream, challenge) = connect_and_handshake(addr).await;
+ let s = secret(SECRET);
+
+ let forged = RaftMessage::RequestVote {
+ term: 9_999,
+ candidate_id: "attacker".to_string(),
+ last_log_index: 0,
+ last_log_term: 0,
+ };
+ let forged_payload = serde_json::to_vec(&forged).unwrap();
+ // The tag the attacker captured — computed over the *original* payload.
+ let stolen_tag = s.frame_tag(
+ &challenge,
+ "node-a",
+ 0,
+ &serde_json::to_vec(&sample_msg()).unwrap(),
+ );
+
+ let mut bytes = hello_bytes(&s, "node-a", &challenge);
+ bytes.extend_from_slice(&(forged_payload.len() as u32).to_be_bytes());
+ bytes.extend_from_slice(&stolen_tag);
+ bytes.extend_from_slice(&forged_payload);
+
+ stream.write_all(&bytes).await.unwrap();
+ stream.flush().await.unwrap();
+
+ assert_no_delivery(&t, "payload swapped under a stolen tag").await;
+}
+
+/// A second frame replayed within the same connection is rejected: `seq` is
+/// part of the signed material, so frame 0 cannot be re-sent as frame 1.
+#[tokio::test]
+async fn duplicated_frame_within_connection_is_rejected() {
+ let addr = free_addr().await;
+ let t = listener(addr, SECRET).await;
+
+ let (mut stream, challenge) = connect_and_handshake(addr).await;
+ let s = secret(SECRET);
+ let frame0 = frame_bytes(&s, "node-a", &challenge, 0, &sample_msg());
+
+ // Frame 0 (valid) followed by a byte-identical copy (which lands on seq 1).
+ stream
+ .write_all(&hello_bytes(&s, "node-a", &challenge))
+ .await
+ .unwrap();
+ stream.write_all(&frame0).await.unwrap();
+ stream.write_all(&frame0).await.unwrap();
+ stream.flush().await.unwrap();
+
+ // The first frame is legitimate and must arrive.
+ let (from, _) = tokio::time::timeout(Duration::from_secs(2), t.recv())
+ .await
+ .expect("first frame timed out")
+ .expect("recv");
+ assert_eq!(from, "node-a");
+
+ // The duplicate must not.
+ assert_no_delivery(&t, "duplicated frame").await;
+}
+
+/// A peer that connects and then goes silent must not pin the accept task
+/// forever, and must never be treated as authenticated.
+#[tokio::test]
+async fn silent_client_delivers_nothing() {
+ let addr = free_addr().await;
+ let t = listener(addr, SECRET).await;
+
+ let (_stream, _challenge) = connect_and_handshake(addr).await;
+ assert_no_delivery(&t, "silent client").await;
+}
diff --git a/engine/crates/xerj-cluster/tests/transport_tests.rs b/engine/crates/xerj-cluster/tests/transport_tests.rs
index 68323fd3d..b139a2db9 100644
--- a/engine/crates/xerj-cluster/tests/transport_tests.rs
+++ b/engine/crates/xerj-cluster/tests/transport_tests.rs
@@ -6,6 +6,7 @@ use std::time::Duration;
use tokio::sync::watch;
+use xerj_cluster::auth::ClusterSecret;
use xerj_cluster::node::{
in_memory::{InMemoryBus, InMemoryTransport},
ClusterNode,
@@ -38,11 +39,13 @@ async fn test_tcp_transport_send_recv() {
let mut peers_b = HashMap::new();
peers_b.insert("node-a".to_string(), addr_a);
- // Create both transports.
- let transport_a = TcpTransport::new("node-a".to_string(), addr_a, peers_a)
+ // Create both transports. Both sides must hold the same cluster secret —
+ // there is no unauthenticated constructor.
+ let secret = || ClusterSecret::new("shared-cluster-secret-for-tests").unwrap();
+ let transport_a = TcpTransport::new("node-a".to_string(), addr_a, peers_a, secret())
.await
.expect("create transport A");
- let transport_b = TcpTransport::new("node-b".to_string(), addr_b, peers_b)
+ let transport_b = TcpTransport::new("node-b".to_string(), addr_b, peers_b, secret())
.await
.expect("create transport B");
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index e076debb8..bc7d828cd 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -237,6 +237,10 @@ impl Config {
self.engine.validate()?;
+ // Cluster: fail closed rather than expose an unauthenticated Raft
+ // control port (issue #75).
+ self.cluster.validate()?;
+
// Logging: format must be one of the two supported line formats.
let fmt = self.logging.format.as_str();
if !fmt.eq_ignore_ascii_case("text") && !fmt.eq_ignore_ascii_case("json") {
@@ -1144,7 +1148,7 @@ impl EngineConfig {
/// When `enabled = false` (the default), the node runs in single-node mode and
/// no Raft or TCP transport is initialised.
///
-/// **4 settings.**
+/// **5 settings.**
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct ClusterConfig {
@@ -1167,6 +1171,25 @@ pub struct ClusterConfig {
/// Controls how often the Raft state machine is driven forward. Lower values
/// improve leader election latency at the cost of CPU.
pub tick_ms: u64,
+ /// Cluster-wide shared secret authenticating every control frame on
+ /// `port` (default: empty — which is only legal while `enabled = false`).
+ ///
+ /// Every node in the cluster must be configured with the **same** secret.
+ /// Frames are signed with HMAC-SHA256 over a per-connection challenge, so
+ /// a peer that does not hold this secret cannot inject Raft messages, and a
+ /// captured frame cannot be replayed.
+ ///
+ /// May be left empty here and supplied via the `XERJ_CLUSTER_AUTH_SECRET`
+ /// environment variable instead — preferable when the config file is baked
+ /// into an image. An explicit non-empty value here wins over the
+ /// environment.
+ ///
+ /// **Fail-closed:** with `enabled = true` and no secret from either source,
+ /// the node refuses to start. There is no unauthenticated cluster mode.
+ ///
+ /// Minimum length [`ClusterConfig::MIN_AUTH_SECRET_LEN`]; generate one with
+ /// `openssl rand -hex 32`.
+ pub auth_secret: String,
}
impl Default for ClusterConfig {
@@ -1176,6 +1199,63 @@ impl Default for ClusterConfig {
port: 9300,
peers: Vec::new(),
tick_ms: 50,
+ auth_secret: String::new(),
+ }
+ }
+}
+
+impl ClusterConfig {
+ /// Environment variable consulted when `auth_secret` is empty.
+ pub const AUTH_SECRET_ENV: &'static str = "XERJ_CLUSTER_AUTH_SECRET";
+
+ /// Minimum accepted secret length, in characters.
+ ///
+ /// Kept in step with `xerj_cluster::auth::MIN_SECRET_LEN`; the transport
+ /// enforces the same floor independently, so a mismatch fails closed.
+ pub const MIN_AUTH_SECRET_LEN: usize = 16;
+
+ /// The secret actually in force, or `None` if neither source supplied one.
+ ///
+ /// Precedence: an explicit non-empty `auth_secret` wins; otherwise
+ /// [`Self::AUTH_SECRET_ENV`]. Explicit configuration beats the ambient
+ /// environment so that a stray env var cannot silently re-key a cluster.
+ pub fn effective_auth_secret(&self) -> Option<String> {
+ Self::resolve_auth_secret(&self.auth_secret, std::env::var(Self::AUTH_SECRET_ENV).ok())
+ }
+
+ /// Pure resolution rule behind [`Self::effective_auth_secret`], split out so
+ /// it can be tested without mutating process-wide environment state.
+ pub fn resolve_auth_secret(field: &str, env: Option<String>) -> Option<String> {
+ let field = field.trim();
+ if !field.is_empty() {
+ return Some(field.to_string());
+ }
+ env.filter(|v| !v.trim().is_empty())
+ .map(|v| v.trim().to_string())
+ }
+
+ /// Fail-closed validation: cluster mode requires a usable shared secret.
+ pub fn validate(&self) -> Result<(), crate::XerjError> {
+ if !self.enabled {
+ return Ok(());
+ }
+ match self.effective_auth_secret() {
+ None => Err(crate::XerjError::config(format!(
+ "cluster.enabled = true requires a shared secret: set cluster.auth_secret \
+ or the {} environment variable. Refusing to start an unauthenticated \
+ cluster transport — anyone able to reach cluster.port could otherwise \
+ inject Raft control messages.",
+ Self::AUTH_SECRET_ENV
+ ))),
+ Some(s) if s.chars().count() < Self::MIN_AUTH_SECRET_LEN => {
+ Err(crate::XerjError::config(format!(
+ "cluster auth secret is too short ({} chars, minimum {}); \
+ generate one with `openssl rand -hex 32`",
+ s.chars().count(),
+ Self::MIN_AUTH_SECRET_LEN
+ )))
+ }
+ Some(_) => Ok(()),
}
}
}
@@ -1518,4 +1598,79 @@ mod tests {
let total: usize = 6 + 3 + 3 + 10 + 5 + 3 + 1 + 6 + 2 + 4 + 12 + 3 + 2;
assert_eq!(total, 60);
}
+
+ // ── Cluster auth: fail closed (issue #75) ────────────────────────────────
+
+ #[test]
+ fn cluster_enabled_without_secret_is_rejected() {
+ let err = Config::from_toml_str(
+ "[cluster]\nenabled = true\nport = 9300\npeers = [\"n2=10.0.0.2:9300\"]\n",
+ )
+ .expect_err("cluster mode without a shared secret must not load");
+ let msg = err.to_string();
+ assert!(
+ msg.contains("auth_secret") && msg.contains("XERJ_CLUSTER_AUTH_SECRET"),
+ "error must name both secret sources, got: {msg}"
+ );
+ }
+
+ #[test]
+ fn cluster_enabled_with_secret_is_accepted() {
+ let cfg = Config::from_toml_str(
+ "[cluster]\nenabled = true\nauth_secret = \"0123456789abcdef0123\"\n",
+ )
+ .expect("cluster mode with a secret must load");
+ assert!(cfg.cluster.enabled);
+ assert_eq!(
+ cfg.cluster.effective_auth_secret().as_deref(),
+ Some("0123456789abcdef0123")
+ );
+ }
+
+ #[test]
+ fn cluster_short_secret_is_rejected() {
+ let err = Config::from_toml_str("[cluster]\nenabled = true\nauth_secret = \"hunter2\"\n")
+ .expect_err("a 7-char cluster secret must be rejected");
+ assert!(err.to_string().contains("too short"), "{err}");
+ }
+
+ #[test]
+ fn cluster_disabled_needs_no_secret() {
+ // The overwhelmingly common case: cluster mode off, nothing to check.
+ Config::from_toml_str("[cluster]\nenabled = false\n").expect("single-node must still load");
+ Config::default()
+ .validate()
+ .expect("defaults must validate");
+ }
+
+ #[test]
+ fn auth_secret_resolution_prefers_explicit_config() {
+ let env = Some("from-environment".to_string());
+ // Explicit config wins; a stray env var cannot silently re-key a cluster.
+ assert_eq!(
+ ClusterConfig::resolve_auth_secret("from-config", env.clone()),
+ Some("from-config".to_string())
+ );
+ // Empty / whitespace-only config falls back to the environment.
+ assert_eq!(
+ ClusterConfig::resolve_auth_secret("", env.clone()),
+ Some("from-environment".to_string())
+ );
+ assert_eq!(
+ ClusterConfig::resolve_auth_secret(" ", env),
+ Some("from-environment".to_string())
+ );
+ // Neither source → None, which is what makes validation fail closed.
+ assert_eq!(ClusterConfig::resolve_auth_secret("", None), None);
+ assert_eq!(
+ ClusterConfig::resolve_auth_secret(" ", Some(" ".to_string())),
+ None
+ );
+ // Surrounding whitespace (e.g. a trailing newline from `$(cat file)`)
+ // is configuration noise, not key material.
+ assert_eq!(
+ ClusterConfig::resolve_auth_secret("", Some(" padded-secret\n".to_string())),
+ Some("padded-secret".to_string())
+ );
+ }
}
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index d469d8fdb..0ad6ab8b7 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -1522,6 +1522,22 @@ async fn async_main() -> Result<()> {
// 8b. Cluster runner (if cluster mode is enabled)
let _cluster_shutdown = if cfg.cluster.enabled {
+ // Fail closed (issue #75). The cluster port carries Raft control
+ // messages; without a shared secret every frame on it is
+ // unauthenticated, so refuse to start rather than open it. This must
+ // abort startup — it is deliberately NOT folded into the degraded
+ // single-node fallback below, which only covers bind failures.
+ let secret_material = cfg.cluster.effective_auth_secret().ok_or_else(|| {
+ anyhow::anyhow!(
+ "cluster.enabled = true but no cluster auth secret is configured: set \
+ cluster.auth_secret or {}. Refusing to start an unauthenticated cluster \
+ transport.",
+ xerj_common::config::ClusterConfig::AUTH_SECRET_ENV
+ )
+ })?;
+ let cluster_secret = xerj_cluster::ClusterSecret::new(&secret_material)
+ .context("cluster auth secret rejected")?;
+
let (shutdown_tx, shutdown_rx) = tokio::sync::watch::channel(false);
// Parse peer list: "node_id=host:port" entries.
@@ -1548,7 +1564,7 @@ async fn async_main() -> Result<()> {
let tick = std::time::Duration::from_millis(cfg.cluster.tick_ms);
- match TcpTransport::new(node_id.clone(), listen_addr, peers).await {
+ match TcpTransport::new(node_id.clone(), listen_addr, peers, cluster_secret).await {
Ok(transport) => {
let peer_ids: Vec<String> = cfg
.cluster
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index 2d36ab445..9ee6deabf 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -368,6 +368,30 @@ turbo_parallel = true
# (e.g. "running" will not match "run").
turbo_fast_analyzer = false
+# ── Cluster (5 settings) ──────────────────────────────────────────────────────
+#
+# Multi-node cluster mode is OFF by default and is experimental. The whole
+# block is commented out; uncomment it only if you are evaluating clustering.
+#
+# The cluster port carries Raft control messages. Every frame on it is
+# authenticated with a cluster-wide shared secret (HMAC-SHA256 over a
+# per-connection challenge), so `auth_secret` is mandatory whenever
+# `enabled = true` — a node with cluster mode on and no secret refuses to
+# start rather than expose an unauthenticated control port. Supply it here or
+# via the XERJ_CLUSTER_AUTH_SECRET environment variable (this file wins if
+# both are set), use the SAME value on every node, and generate it with
+# `openssl rand -hex 32`. Minimum 16 characters.
+#
+# Note: the HMAC authenticates frames, it does not encrypt them. Cluster
+# traffic is still plaintext — keep the cluster port on a trusted network.
+#
+# [cluster]
+# enabled = true
+# port = 9300
+# peers = ["n2=10.0.0.2:9300", "n3=10.0.0.3:9300"]
+# tick_ms = 50
+# auth_secret = ""
+
# ── Logging (2 settings) ──────────────────────────────────────────────────────
[logging]
diff --git a/landing/docs/config.html b/landing/docs/config.html
index 603fc67eb..d6e9b7363 100644
--- a/landing/docs/config.html
+++ b/landing/docs/config.html
@@ -124,7 +124,7 @@
<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p>
<h2>Full key table</h2>
- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"0.0.0.0"</div><div class="cell dsc">Interface to bind every listener to. Use "127.0.0.1" for dev or a private address in production.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">256</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">"size_tiered" or "log_structured". Size-tiered for write-heavy, log-structured for mixed workloads.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Minimum live segments before a merge pass is scheduled. Must be ≥ 2.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">5120</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments larger than this are never merged.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Merge I/O throttle (MiB/s). Set 0 to disable — not recommended in production.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Concurrent merge workers per index. Bump to 2–4 on fast NVMe.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"scalar8"</div><div class="cell dsc">"none" · "scalar8" · "scalar4" · "binary". scalar8 = 4× RAM saving, 1–2% recall loss.</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div></div>
+ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"0.0.0.0"</div><div class="cell dsc">Interface to bind every listener to. Use "127.0.0.1" for dev or a private address in production.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">256</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">"size_tiered" or "log_structured". Size-tiered for write-heavy, log-structured for mixed workloads.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Minimum live segments before a merge pass is scheduled. Must be ≥ 2.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">5120</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments larger than this are never merged.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Merge I/O throttle (MiB/s). Set 0 to disable — not recommended in production.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Concurrent merge workers per index. Bump to 2–4 on fast NVMe.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"scalar8"</div><div class="cell dsc">"none" · "scalar8" · "scalar4" · "binary". scalar8 = 4× RAM saving, 1–2% recall loss.</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div>
<h2 id="server-section">[server]</h2>