AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(cluster): replay the on-disk Raft log when a node restarts (#2313)
xerj-org57b8a5058c5f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b9e8eaeb3..24ea8ad2d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -21,11 +21,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0confirm. Fewer confirmations than the quorum is still HTTP 503`replica_not_acknowledged` — the primary has the write, and retrying theindex copies it to the replicas still in sync. There is still no- `wait_for_active_shards`, no replica promotion, and no snapshot catch-up- for a node that missed the log (part of+ `wait_for_active_shards` and no replica promotion. A restarted member+ reloads metadata from the on-disk Raft log; document backfill is still+ open (part of[#1170](https://github.com/xerj-org/xerj/issues/1170)). Single-node modedoes not take this path.+- With `[cluster] enabled = true`, the server opens an on-disk Raft log+ at `{data_dir}/_raft` and replays its committed prefix before it+ serves. A restart restores the shard plan and the in-sync set, so a+ later document write is routed to the primary again. A member that+ starts with an empty log copies the leader's log and applies it, which+ installs index metadata (and an empty replica store when this node is+ a replica). It does not copy documents written before it joined. A+ replica removed from the in-sync set is not filled in. The log is not+ compacted; there is no InstallSnapshot. Term and vote are stored with+ the log, so a restarted node does not grant a second vote in the same+ term. `[cluster] enabled = false` does not open the log (part of+ [#1170](https://github.com/xerj-org/xerj/issues/1170)).+- A DeepSeek Harness profile bundle in `dsh/` (`xerj-dsh`, Apache-2.0)inserts the README's `mcp-xerj` stdio row via`@deepseek-ai/dsh-mcp-client`. The row is still untested on DSH.diff --git a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rsindex bf7033549..70e8dde3c 100644--- a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs+++ b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs@@ -25,6 +25,7 @@ struct Member {handle: ClusterHandle,/// Held when the runner is not started, so the proposal channel stays open._node: Option<ClusterNode>,+ runner: Option<tokio::task::JoinHandle<()>>,_dir: tempfile::TempDir,}@@ -55,16 +56,16 @@ async fn ring(start_runners: bool) -> Vec<Member> {handle.set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(state.clone())));let engine = std::sync::Arc::clone(&state.engine);let (shutdown, shutdown_rx) = watch::channel(false);- let kept = if start_runners {+ let (kept, runner) = if start_runners {let mut runner = ClusterRunner::new(node, Duration::from_millis(5), shutdown_rx);- tokio::spawn(async move {+ let task = tokio::spawn(async move {runner.run().await;});- None+ (None, Some(task))} else {handle.set_timeout(Duration::from_millis(400));let _ = shutdown_rx;- Some(node)+ (Some(node), None)};let app = xerj_api::router::build_es_compat_router(state.clone());members.push(Member {@@ -74,6 +75,7 @@ async fn ring(start_runners: bool) -> Vec<Member> {shutdown,handle,_node: kept,+ runner,_dir: dir,});}@@ -860,3 +862,156 @@ async fn a_document_is_stored_on_its_shard_owner_and_search_sees_it_from_any_memlet _ = member.shutdown.send(true);}}++async fn shutdown_and_keep(members: Vec<Member>) -> Vec<tempfile::TempDir> {+ let mut dirs = Vec::with_capacity(members.len());+ for mut member in members {+ let _ = member.shutdown.send(true);+ if let Some(task) = member.runner.take() {+ match tokio::time::timeout(Duration::from_secs(5), task).await {+ Ok(Ok(())) => {}+ other => panic!("cluster runner did not stop: {other:?}"),+ }+ }+ // ApiShardHandler holds AppState, which holds Engine, which holds+ // this handle. The engine's data-dir node.lock stays taken until+ // that cycle breaks, so a later Engine::new on the same directory+ // sees this process as another running node.+ member.handle.clear_shard_handler();+ drop(member.app);+ drop(member.state);+ drop(member.engine);+ drop(member.handle);+ dirs.push(member._dir);+ }+ dirs+}++/// Same ring as [`ring`], but each member opens `{data_dir}/_raft` and+/// replays the committed prefix before the runner starts.+async fn durable_ring(dirs: Vec<tempfile::TempDir>) -> Vec<Member> {+ let bus = InMemoryBus::new();+ let ids = ["n1", "n2", "n3"];+ let mut members = Vec::new();+ for (id, dir) in ids.into_iter().zip(dirs) {+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config, engine, metrics);++ let peers: Vec<String> = ids+ .iter()+ .filter(|peer| **peer != id)+ .map(|peer| (*peer).to_string())+ .collect();+ let transport = InMemoryTransport::new(id.to_string(), bus.clone()).await;+ let raft_dir = xerj_cluster::raft_log::raft_storage_dir(dir.path());+ let mut node =+ ClusterNode::with_storage(id.to_string(), peers, Box::new(transport), &raft_dir)+ .expect("open raft log");+ node.set_applier(std::sync::Arc::new(EngineMetadataApplier {+ engine: std::sync::Arc::clone(&state.engine),+ }));+ let handle = node.handle();+ state.engine.attach_cluster(handle.clone());+ handle.set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(state.clone())));+ node.replay_committed().await;+ let engine = std::sync::Arc::clone(&state.engine);+ let (shutdown, shutdown_rx) = watch::channel(false);+ let mut runner = ClusterRunner::new(node, Duration::from_millis(5), shutdown_rx);+ let task = tokio::spawn(async move {+ runner.run().await;+ });+ let app = xerj_api::router::build_es_compat_router(state.clone());+ members.push(Member {+ app,+ state,+ engine,+ shutdown,+ handle,+ _node: None,+ runner: Some(task),+ _dir: dir,+ });+ }+ members+}++#[tokio::test(flavor = "multi_thread", worker_threads = 4)]+async fn restart_reloads_the_shard_plan_and_still_routes_the_document() {+ let dirs = (0..3)+ .map(|_| tempfile::tempdir().expect("tempdir"))+ .collect();+ let members = durable_ring(dirs).await;+ let dead = [false, false, false];+ let _leader = wait_leader(&members, &dead).await;++ let (status, body) = put_json(&members[0].app, "/orders", json!({})).await;+ assert!(status.is_success(), "create: {status} {body}");+ for member in &members {+ wait_status(&member.app, "/orders", StatusCode::OK).await;+ }+ let (status, body) = put_json(+ &members[0].app,+ "/orders/_doc/1",+ json!({ "title": "kept" }),+ )+ .await;+ assert!(status.is_success(), "write: {status} {body}");++ let owner = members+ .iter()+ .position(|member| member.engine.owns_shard("orders", "1"))+ .expect("one primary");+ let owner_id = members[owner].handle.node_id().to_string();+ let other = (0..3).find(|i| *i != owner).expect("a non-owner");+ assert_eq!(+ members[other].engine.route_write("orders", "1").as_deref(),+ Some(owner_id.as_str()),+ "before restart the non-owner forwards"+ );+ let (status, body) = post_json(&members[owner].app, "/orders/_refresh", json!({})).await;+ assert!(+ status.is_success(),+ "refresh before restart: {status} {body}"+ );++ let dirs = shutdown_and_keep(members).await;+ let members = durable_ring(dirs).await;++ assert!(+ members[owner].engine.owns_shard("orders", "1"),+ "replay must put the primary back on {owner_id}"+ );+ assert_eq!(+ members[other].engine.route_write("orders", "1").as_deref(),+ Some(owner_id.as_str()),+ "restart must not treat the index as unassigned"+ );+ let (status, body) = get(&members[other].app, "/orders/_doc/1").await;+ assert!(+ status.is_success(),+ "GET from a non-owner after restart: {status} {body}"+ );++ let _leader = wait_leader(&members, &dead).await;+ let (status, body) = put_json(+ &members[other].app,+ "/orders/_doc/2",+ json!({ "title": "after" }),+ )+ .await;+ assert!(status.is_success(), "write after restart: {status} {body}");+ for member in &members {+ let (status, body) = get(&member.app, "/orders/_doc/2").await;+ assert!(+ status.is_success(),+ "doc 2 readable after restart: {status} {body}"+ );+ }++ for member in &members {+ let _ = member.shutdown.send(true);+ }+}diff --git a/engine/crates/xerj-cluster/src/node.rs b/engine/crates/xerj-cluster/src/node.rsindex 257b77aab..3c8aae835 100644--- a/engine/crates/xerj-cluster/src/node.rs+++ b/engine/crates/xerj-cluster/src/node.rs@@ -114,6 +114,40 @@ impl ClusterNode {}}+ /// Like [`ClusterNode::new`], opening the Raft log at `storage_dir`.+ ///+ /// The committed entries are loaded into the Raft log and are not applied+ /// yet. Call [`ClusterNode::replay_committed`] after [`ClusterNode::set_applier`]+ /// so the engine sees the same entries the metadata store does. Applying+ /// here would run before the applier exists and a restarted index plan+ /// would stay in metadata only.+ pub fn with_storage(+ id: String,+ peers: Vec<String>,+ transport: Box<dyn ClusterTransport>,+ storage_dir: impl AsRef<std::path::Path>,+ ) -> Result<Self> {+ let mut node = Self::new(id.clone(), peers, transport);+ let peer_ids = node.raft.peers().to_vec();+ node.raft = RaftNode::with_storage(id, peer_ids, storage_dir)?;+ Ok(node)+ }++ /// Apply every committed log entry that this process has not applied yet.+ ///+ /// On a fresh open that is the whole committed prefix. The applier, when+ /// one is installed, runs for each entry before this returns. Safe to call+ /// again: a second call finds nothing pending.+ pub async fn replay_committed(&mut self) {+ let commit = self.raft.commit_index();+ if commit == 0 {+ info!(node = %self.raft.id, "raft log has no committed entries to replay");+ return;+ }+ info!(node = %self.raft.id, commit, "replaying committed raft log");+ self.apply_ready().await;+ }+/// Handle the engine layer uses to propose metadata and read leadership.pub fn handle(&self) -> ClusterHandle {self.handle.clone()diff --git a/engine/crates/xerj-cluster/src/raft.rs b/engine/crates/xerj-cluster/src/raft.rsindex 4b24c9ced..88a34e01e 100644--- a/engine/crates/xerj-cluster/src/raft.rs+++ b/engine/crates/xerj-cluster/src/raft.rs@@ -99,8 +99,8 @@ pub enum ClusterCommand {/// `request_id` is unique on the node that accepted the caller. That node/// completes the waiting request when this entry is applied. Other nodes/// apply `inner` and ignore the id. The wrapper is persisted with the- /// log entry so a restarted node (once the server opens the file log)- /// can still tell client entries from internal ones.+ /// log entry so a restarted node, which opens the file log, can still+ /// tell client entries from internal ones.ClientMetadata {/// Node that accepted the caller. Only that node completes a waiter.origin: String,@@ -356,6 +356,15 @@ impl RaftNode {}}node.commit_index = flog.commit_index();+ // A vote is only valid in the term it was cast. A hard-state term+ // ahead of the log (we campaigned or stepped down without appending)+ // wins; a missing hard state leaves the term reconstructed from the log.+ if flog.hard_term() > node.current_term {+ node.current_term = flog.hard_term();+ }+ if flog.hard_term() == node.current_term && flog.hard_term() > 0 {+ node.voted_for = flog.hard_vote().map(str::to_string);+ }node.persistent = Some(flog);Ok(node)}@@ -399,6 +408,14 @@ impl RaftNode {new_term = msg_term,"Higher term observed — stepping down to follower");+ // The vote from the previous term must not survive into this one.+ if !self.write_hard_state(msg_term, None) {+ warn!(+ node = %self.id,+ term = msg_term,+ "failed to persist the higher term; stepping down anyway"+ );+ }self.current_term = msg_term;self.voted_for = None;self.become_follower(None);@@ -521,6 +538,11 @@ impl RaftNode {.map_err(|e| anyhow::anyhow!("append raft log: {e}"))?;flog.fsync().map_err(|e| anyhow::anyhow!("fsync raft log: {e}"))?;+ if let Err(e) = flog.set_log_end(index) {+ flog.truncate_from(index);+ let _ = flog.set_log_end(index.saturating_sub(1));+ return Err(anyhow::anyhow!("persist raft log end: {e}"));+ }}self.log.push(entry);@@ -626,6 +648,97 @@ impl RaftNode {}}+ /// Persist `term` and `voted_for` before they take effect.+ ///+ /// Returns `true` when there is no file log, or the write replaced+ /// `hard.meta`. A `false` means the caller must not grant a vote or+ /// start an election: a crash would otherwise forget the vote and+ /// grant a second one in the same term.+ fn write_hard_state(&mut self, term: u64, voted_for: Option<String>) -> bool {+ let Some(flog) = self.persistent.as_mut() else {+ return true;+ };+ match flog.set_hard_state(term, voted_for.as_deref()) {+ Ok(()) => true,+ Err(e) => {+ warn!(node = %self.id, error = %e, term, "failed to persist raft term and vote");+ false+ }+ }+ }++ /// Copy the in-memory log onto the file log and fsync it.+ ///+ /// Heartbeats that do not change the log do not fsync. Returns `false`+ /// when the file log could not be updated; the caller must not tell the+ /// leader the append succeeded.+ fn mirror_log(&mut self) -> bool {+ if self.persistent.is_none() {+ return true;+ }+ let node_id = self.id.clone();+ let mem_last = self.last_log_index();+ let encoded: Vec<(u64, u64, Vec<u8>)> = match self+ .log+ .iter()+ .map(|entry| {+ serde_json::to_vec(&entry.command).map(|payload| (entry.index, entry.term, payload))+ })+ .collect()+ {+ Ok(entries) => entries,+ Err(e) => {+ warn!(node = %node_id, error = %e, "encode raft entry");+ return false;+ }+ };+ let flog = self+ .persistent+ .as_mut()+ .expect("persistent log checked above");+ let mut dirty = false;+ if flog.last_index() > mem_last {+ flog.truncate_from(mem_last + 1);+ dirty = true;+ }+ for (index, term, payload) in &encoded {+ match flog.term_of(*index) {+ Some(have) if have == *term => {}+ Some(_) => {+ flog.truncate_from(*index);+ if let Err(e) = flog.append(*term, *index, payload) {+ warn!(node = %node_id, error = %e, index, "append raft log");+ return false;+ }+ dirty = true;+ }+ None => {+ if let Err(e) = flog.append(*term, *index, payload) {+ warn!(node = %node_id, error = %e, index, "append raft log");+ return false;+ }+ dirty = true;+ }+ }+ }+ if !dirty {+ match flog.persisted_log_end() {+ Some(end) if end == mem_last => return true,+ None if flog.last_index() == mem_last => return true,+ _ => {}+ }+ }+ if let Err(e) = flog.fsync() {+ warn!(node = %node_id, error = %e, "fsync raft log");+ return false;+ }+ if let Err(e) = flog.set_log_end(mem_last) {+ warn!(node = %node_id, error = %e, "persist raft log end");+ return false;+ }+ true+ }+fn cluster_size(&self) -> usize {self.peers.len() + 1 // peers + self}@@ -691,9 +804,17 @@ impl RaftNode {}fn start_election(&mut self) -> Vec<OutboundMessage> {- self.current_term += 1;+ let new_term = self.current_term + 1;+ let vote = self.id.clone();+ // The self-vote has to be durable before the RequestVote goes out.+ // A failed disk stays a follower and tries again after the next timeout.+ if !self.write_hard_state(new_term, Some(vote.clone())) {+ self.last_heartbeat = Instant::now();+ return vec![];+ }+ self.current_term = new_term;self.state = RaftState::Candidate;- self.voted_for = Some(self.id.clone()); // vote for self+ self.voted_for = Some(vote);self.votes_received = 1; // count self-voteself.election_timeout = Self::random_election_timeout();self.last_heartbeat = Instant::now();@@ -819,8 +940,11 @@ impl RaftNode {Some(e) if e.term != prev_log_term => {// 3. If an existing entry conflicts with a new one (same index,// different terms), delete the existing entry and all that- // follow it (§5.3)+ // follow it (§5.3). The shortened log is durable before+ // the rejection, so a crash cannot bring the tail back+ // and win an election with it.self.log.retain(|e| e.index < prev_log_index);+ let _ = self.mirror_log();return vec![OutboundMessage {to: leader_id,msg: RaftMessage::AppendEntriesResponse {@@ -852,6 +976,21 @@ impl RaftNode {// Ensure log is sorted by index (should already be, but be safe)self.log.sort_by_key(|e| e.index);+ // Durable before the success reply. A heartbeat that changes nothing+ // does not fsync. A failed mirror is a rejection: the leader retries,+ // and this process still has the entries in memory.+ if !self.mirror_log() {+ return vec![OutboundMessage {+ to: leader_id,+ msg: RaftMessage::AppendEntriesResponse {+ from: self.id.clone(),+ term: self.current_term,+ success: false,+ match_index: 0,+ },+ }];+ }+// 5. If leaderCommit > commitIndex, set commitIndex =// min(leaderCommit, index of last new entry) (§5.3)if leader_commit > self.commit_index {@@ -965,10 +1104,14 @@ impl RaftNode {&& last_log_index >= self.last_log_index());if can_vote && candidate_log_ok {- self.voted_for = Some(candidate_id.clone());- self.last_heartbeat = Instant::now(); // reset election timer- vote_granted = true;- debug!(node = %self.id, candidate = %candidate_id, "Granting vote");+ // Grant only after the vote is durable. Otherwise a restart+ // in this term would vote for someone else.+ if self.write_hard_state(self.current_term, Some(candidate_id.clone())) {+ self.voted_for = Some(candidate_id.clone());+ self.last_heartbeat = Instant::now(); // reset election timer+ vote_granted = true;+ debug!(node = %self.id, candidate = %candidate_id, "Granting vote");+ }}}diff --git a/engine/crates/xerj-cluster/src/raft_log.rs b/engine/crates/xerj-cluster/src/raft_log.rsindex 199d8962e..b55b9ef44 100644--- a/engine/crates/xerj-cluster/src/raft_log.rs+++ b/engine/crates/xerj-cluster/src/raft_log.rs@@ -25,9 +25,14 @@//! `BTreeMap<index, (term, file_offset, payload_length)>`, and re-derive//! `commit_index` + `last_applied` from a sidecar `commit.meta` file.//!-//! `fsync` is batched: by default every 100 ms or whenever 256 entries-//! accumulate in the OS page cache, whichever comes first. Both values-//! are tunable via `[cluster] raft_fsync_ms = 100` / `raft_fsync_batch = 256`.+//! The server opens this log at `{data_dir}/_raft` (`RAFT_DIR_NAME`).+//! `_raft` is not a valid index name, and the directory has no `wal`+//! child, so the engine's index scan does not open it.+//!+//! `hard.meta` stores the current term, the vote, and the logical end of+//! the log. Truncation does not rewrite `raft.log`; entries past that end+//! stay in the file and are ignored on the next open. A missing `hard.meta`+//! (a log written before this file existed) keeps every CRC-valid record.//!//! Exit criterion for M5.1 (verified by `raft_log_persist_test`)://!@@ -55,8 +60,19 @@ use crc32fast::Hasher as Crc32;/// Magic for the raft log file format. Distinct from the WAL magic so/// we never confuse one for the other on recovery.const MAGIC: u32 = 0x5A52_4C32; // "ZRL2" (ASCII Z R L 2)+const HARD_MAGIC: u32 = 0x5A52_4853; // "ZRHS"const LOG_FILE: &str = "raft.log";const COMMIT_FILE: &str = "commit.meta";+const HARD_FILE: &str = "hard.meta";++/// Directory name under the node's data dir. Not a valid index name, so a+/// user index cannot occupy the same path and take the log with it on delete.+pub const RAFT_DIR_NAME: &str = "_raft";++/// `{data_dir}/_raft`, where the server opens [`FileRaftLog`].+pub fn raft_storage_dir(data_dir: impl AsRef<Path>) -> PathBuf {+ data_dir.as_ref().join(RAFT_DIR_NAME)+}/// Raft payload kinds. We only use `ClusterCommand` today but leave room/// for future out-of-band messages (config changes, snapshots).@@ -96,6 +112,13 @@ pub struct FileRaftLog {last_index: u64,/// Commit index persisted to `commit.meta`.commit_index: u64,+ /// Current term persisted in `hard.meta`. 0 when that file is absent.+ hard_term: u64,+ /// Vote in `hard_term`, when one was persisted.+ hard_vote: Option<String>,+ /// Logical end of the log. `None` means `hard.meta` did not record one+ /// and the scan is authoritative. `Some(0)` is an empty log.+ log_end: Option<u64>,}impl FileRaftLog {@@ -197,6 +220,11 @@ impl FileRaftLog {// Load committed index (best-effort).let commit_index = Self::load_commit_index(&dir).unwrap_or(0);+ let (hard_term, hard_vote, log_end) = Self::load_hard_state(&dir)?;+ if let Some(end) = log_end {+ seek_table.retain(|&idx, _| idx <= end);+ last_index = seek_table.keys().next_back().copied().unwrap_or(0);+ }Ok(Self {dir,@@ -204,6 +232,9 @@ impl FileRaftLog {seek_table,last_index,commit_index,+ hard_term,+ hard_vote,+ log_end,})}@@ -295,14 +326,13 @@ impl FileRaftLog {Ok(Some((term, payload)))}- /// Truncate the log by removing every entry with `index >= from`.+ /// Drop every seek-table entry with `index >= from`.///- /// This is called by Raft when a leader steps down and the new leader- /// instructs the follower to overwrite conflicting tail entries.- /// We don't physically truncate the file — the seek table evicts the- /// removed entries, the CRC of later writes will differ, and on- /// recovery we stop reading at the first CRC mismatch. That keeps- /// this method O(log n) instead of O(file_size).+ /// The file is not rewritten. The drop becomes durable when+ /// [`FileRaftLog::set_log_end`] records the new end; until that file+ /// is replaced, a crash reopens the previous tail. Callers that have+ /// already decided the tail is gone must set the end before they+ /// acknowledge the append that conflicted.pub fn truncate_from(&mut self, from: u64) {self.seek_table.retain(|&idx, _| idx < from);self.last_index = self@@ -324,6 +354,140 @@ impl FileRaftLog {Ok(())}+ /// Term and vote from `hard.meta`. Term 0 and a missing vote mean the+ /// file was absent.+ pub fn hard_term(&self) -> u64 {+ self.hard_term+ }++ pub fn hard_vote(&self) -> Option<&str> {+ self.hard_vote.as_deref()+ }++ /// Logical end recorded in `hard.meta`, if that file had one.+ pub fn persisted_log_end(&self) -> Option<u64> {+ self.log_end+ }++ /// Record the current term and vote. The logical log end is left as it is.+ ///+ /// Memory is updated only after `hard.meta` is replaced. A failed write+ /// leaves the previous term and vote in place, so a later log-end save+ /// cannot publish an election this process aborted.+ pub fn set_hard_state(&mut self, term: u64, voted_for: Option<&str>) -> Result<()> {+ self.persist_hard(term, voted_for, self.log_end)?;+ self.hard_term = term;+ self.hard_vote = voted_for.map(str::to_string);+ Ok(())+ }++ /// Record the highest index that is still part of the log.+ ///+ /// The file is replaced before the seek table drops indexes above `end`.+ /// A failed write leaves those entries readable. The next open ignores+ /// them once this call has succeeded.+ pub fn set_log_end(&mut self, end: u64) -> Result<()> {+ self.persist_hard(self.hard_term, self.hard_vote.as_deref(), Some(end))?;+ if end < self.last_index {+ self.truncate_from(end + 1);+ }+ self.log_end = Some(end);+ Ok(())+ }++ /// Replace `hard.meta`. The caller updates its fields only after this+ /// returns `Ok`.+ fn persist_hard(&self, term: u64, vote: Option<&str>, log_end: Option<u64>) -> Result<()> {+ let path = self.dir.join(HARD_FILE);+ let tmp = self.dir.join("hard.meta.tmp");+ let mut buf = Vec::with_capacity(64);+ buf.extend_from_slice(&HARD_MAGIC.to_le_bytes());+ buf.extend_from_slice(&1u32.to_le_bytes());+ buf.extend_from_slice(&term.to_le_bytes());+ match log_end {+ Some(end) => {+ buf.push(1);+ buf.extend_from_slice(&end.to_le_bytes());+ }+ None => {+ buf.push(0);+ buf.extend_from_slice(&0u64.to_le_bytes());+ }+ }+ match vote {+ Some(vote) => {+ let bytes = vote.as_bytes();+ let len = u32::try_from(bytes.len()).context("raft vote longer than u32")?;+ buf.extend_from_slice(&len.to_le_bytes());+ buf.extend_from_slice(bytes);+ }+ None => buf.extend_from_slice(&u32::MAX.to_le_bytes()),+ }+ {+ let mut file = OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(&tmp)+ .with_context(|| format!("writing {tmp:?}"))?;+ file.write_all(&buf)+ .with_context(|| format!("writing {tmp:?}"))?;+ file.sync_all()+ .with_context(|| format!("syncing {tmp:?}"))?;+ }+ std::fs::rename(&tmp, &path).with_context(|| format!("replacing {path:?}"))?;+ // The rename is the publication. Sync the directory so a crash cannot+ // lose the directory entry that names the new file.+ let dir =+ File::open(&self.dir).with_context(|| format!("opening {:?} to sync", self.dir))?;+ dir.sync_all()+ .with_context(|| format!("syncing raft dir {:?}", self.dir))?;+ Ok(())+ }++ fn load_hard_state(dir: &Path) -> Result<(u64, Option<String>, Option<u64>)> {+ let path = dir.join(HARD_FILE);+ if !path.exists() {+ return Ok((0, None, None));+ }+ let bytes = std::fs::read(&path).with_context(|| format!("reading {path:?}"))?;+ if bytes.len() < 4 + 4 + 8 + 1 + 8 + 4 {+ return Err(anyhow!("raft hard state {path:?} is truncated"));+ }+ let magic = u32::from_le_bytes(bytes[0..4].try_into().expect("4 bytes"));+ if magic != HARD_MAGIC {+ return Err(anyhow!("raft hard state {path:?} has a bad magic"));+ }+ let version = u32::from_le_bytes(bytes[4..8].try_into().expect("4 bytes"));+ if version != 1 {+ return Err(anyhow!(+ "raft hard state {path:?} has version {version}, want 1"+ ));+ }+ let term = u64::from_le_bytes(bytes[8..16].try_into().expect("8 bytes"));+ let has_end = bytes[16];+ let end = u64::from_le_bytes(bytes[17..25].try_into().expect("8 bytes"));+ let log_end = match has_end {+ 0 => None,+ 1 => Some(end),+ other => return Err(anyhow!("raft hard state {path:?} has log-end flag {other}")),+ };+ let vote_len = u32::from_le_bytes(bytes[25..29].try_into().expect("4 bytes"));+ let vote = if vote_len == u32::MAX {+ None+ } else {+ let start = 29;+ let end_at = start + vote_len as usize;+ if bytes.len() < end_at {+ return Err(anyhow!("raft hard state {path:?} vote is truncated"));+ }+ let text = std::str::from_utf8(&bytes[start..end_at])+ .with_context(|| format!("raft vote in {path:?} is not utf-8"))?;+ Some(text.to_string())+ };+ Ok((term, vote, log_end))+ }+/// Highest log index appended so far (0 if the log is empty).pub fn last_index(&self) -> u64 {self.last_index@@ -424,4 +588,56 @@ mod tests {assert!(log2.len() >= 4);assert!(log2.len() <= 5);}++ #[test]+ fn log_end_hides_a_truncated_tail_on_reopen() {+ let dir = tempdir().unwrap();+ let mut log = FileRaftLog::open(dir.path()).unwrap();+ for i in 1..=5 {+ log.append(1, i, format!("{i}").as_bytes()).unwrap();+ }+ log.fsync().unwrap();+ log.truncate_from(4);+ log.set_log_end(3).unwrap();+ drop(log);++ let mut log2 = FileRaftLog::open(dir.path()).unwrap();+ assert_eq!(log2.last_index(), 3);+ assert!(log2.read(4).unwrap().is_none());+ assert!(log2.read(5).unwrap().is_none());+ assert_eq!(log2.read(3).unwrap().unwrap().1, b"3");++ log2.append(2, 4, b"new").unwrap();+ log2.fsync().unwrap();+ log2.set_log_end(4).unwrap();+ drop(log2);++ let mut log3 = FileRaftLog::open(dir.path()).unwrap();+ assert_eq!(log3.last_index(), 4);+ assert_eq!(log3.read(4).unwrap().unwrap().1, b"new");+ assert!(log3.read(5).unwrap().is_none());+ }++ #[test]+ fn hard_state_roundtrip_and_a_truncated_file_refuses_to_open() {+ let dir = tempdir().unwrap();+ let mut log = FileRaftLog::open(dir.path()).unwrap();+ log.set_hard_state(4, Some("n2")).unwrap();+ log.set_log_end(0).unwrap();+ drop(log);++ let log2 = FileRaftLog::open(dir.path()).unwrap();+ assert_eq!(log2.hard_term(), 4);+ assert_eq!(log2.hard_vote(), Some("n2"));+ assert_eq!(log2.persisted_log_end(), Some(0));++ std::fs::write(dir.path().join(HARD_FILE), [1u8, 2, 3]).unwrap();+ match FileRaftLog::open(dir.path()) {+ Err(err) => assert!(+ err.to_string().contains("truncated"),+ "corrupt hard state must not look like no vote: {err}"+ ),+ Ok(_) => panic!("corrupt hard state opened"),+ }+ }}diff --git a/engine/crates/xerj-cluster/tests/log_catchup.rs b/engine/crates/xerj-cluster/tests/log_catchup.rsnew file mode 100644index 000000000..b7bc59d72--- /dev/null+++ b/engine/crates/xerj-cluster/tests/log_catchup.rs@@ -0,0 +1,236 @@+//! The file Raft log is what a restarted member and a member that starts+//! with an empty directory use to install committed metadata.+//!+//! Document bytes are not in this log. These tests stop at the metadata+//! state machine: the index and the in-sync set.++use std::path::PathBuf;+use std::sync::{Arc, Mutex};+use std::time::{Duration, Instant};++use async_trait::async_trait;+use tempfile::TempDir;+use xerj_cluster::node::in_memory::{InMemoryBus, InMemoryTransport};+use xerj_cluster::{ApplyEffect, ClusterCommand, ClusterNode, MetadataApplier};++struct RecordingApplier {+ events: Arc<Mutex<Vec<String>>>,+}++#[async_trait]+impl MetadataApplier for RecordingApplier {+ async fn apply_metadata(&self, command: &ClusterCommand) -> ApplyEffect {+ let label = match command.as_metadata() {+ ClusterCommand::CreateIndex { name, .. } => format!("create:{name}"),+ ClusterCommand::SetShardInSync { index, shard, .. } => {+ format!("insync:{index}:{shard}")+ }+ other => format!("other:{other:?}"),+ };+ self.events.lock().expect("events").push(label);+ ApplyEffect::Applied+ }+}++async fn open_node(id: &str, ids: &[&str], bus: &InMemoryBus, dir: &PathBuf) -> ClusterNode {+ let peers: Vec<String> = ids+ .iter()+ .filter(|peer| **peer != id)+ .map(|peer| (*peer).to_string())+ .collect();+ let transport = InMemoryTransport::new(id.to_string(), bus.clone()).await;+ ClusterNode::with_storage(id.to_string(), peers, Box::new(transport), dir)+ .expect("open raft log")+}++async fn boot(ids: &[&str], dirs: &[PathBuf]) -> Vec<ClusterNode> {+ let bus = InMemoryBus::new();+ let mut nodes = Vec::new();+ for (id, dir) in ids.iter().zip(dirs.iter()) {+ nodes.push(open_node(id, ids, &bus, dir).await);+ }+ nodes+}++fn fresh_dirs(n: usize) -> Vec<(TempDir, PathBuf)> {+ (0..n)+ .map(|_| {+ let dir = tempfile::tempdir().expect("tempdir");+ let path = dir.path().to_path_buf();+ (dir, path)+ })+ .collect()+}++async fn pump_until(nodes: &mut [ClusterNode], mut done: impl FnMut(&[ClusterNode]) -> bool) {+ let deadline = Instant::now() + Duration::from_secs(10);+ while Instant::now() < deadline {+ for node in nodes.iter_mut() {+ node.step(Duration::from_millis(5))+ .await+ .expect("raft step");+ }+ if done(nodes) {+ return;+ }+ }+ let summary: Vec<String> = nodes+ .iter()+ .map(|node| {+ format!(+ "{} state={} leader={:?} commit={} log={} indices={:?}",+ node.raft.id,+ node.raft.state(),+ node.leader_id(),+ node.raft.commit_index(),+ node.raft.log_len(),+ node.metadata.indices.keys().collect::<Vec<_>>()+ )+ })+ .collect();+ panic!("timed out: {summary:?}");+}++fn has_orders(node: &ClusterNode) -> bool {+ node.metadata.indices.contains_key("orders")+ && node+ .metadata+ .config+ .get("shard_in_sync:orders:0")+ .map(String::as_str)+ == Some("n2")+}++#[tokio::test]+async fn restart_replays_the_committed_index_and_in_sync_set() {+ let ids = ["n1", "n2", "n3"];+ let held = fresh_dirs(3);+ let paths: Vec<PathBuf> = held.iter().map(|(_, path)| path.clone()).collect();+ let mut nodes = boot(&ids, &paths).await;++ pump_until(&mut nodes, |nodes| {+ nodes.iter().filter(|n| n.is_leader()).count() == 1+ })+ .await;+ let leader = nodes.iter().position(|n| n.is_leader()).expect("leader");+ nodes[leader]+ .propose(ClusterCommand::CreateIndex {+ name: "orders".into(),+ schema_json: "{}".into(),+ })+ .expect("propose create");+ nodes[leader]+ .propose(ClusterCommand::SetShardInSync {+ index: "orders".into(),+ shard: 0,+ nodes: vec!["n2".into()],+ })+ .expect("propose in-sync");+ pump_until(&mut nodes, |nodes| nodes.iter().all(has_orders)).await;+ drop(nodes);++ let mut restored = boot(&ids, &paths).await;+ for node in &restored {+ assert!(+ !node.metadata.indices.contains_key("orders"),+ "opening the log must not apply it before replay"+ );+ assert!(+ node.raft.commit_index() >= 2,+ "commit index must be on disk"+ );+ }+ let events = Arc::new(Mutex::new(Vec::new()));+ for node in &mut restored {+ node.set_applier(Arc::new(RecordingApplier {+ events: Arc::clone(&events),+ }));+ node.replay_committed().await;+ assert!(+ has_orders(node),+ "replay must restore metadata on {}",+ node.raft.id+ );+ }+ let events = events.lock().expect("events").clone();+ assert_eq!(+ events+ .iter()+ .filter(|e| e.as_str() == "create:orders")+ .count(),+ 3+ );+ assert_eq!(+ events+ .iter()+ .filter(|e| e.as_str() == "insync:orders:0")+ .count(),+ 3+ );+}++#[tokio::test]+async fn an_empty_member_installs_the_leader_log_and_keeps_it_across_restart() {+ let ids = ["n1", "n2", "n3"];+ let held = fresh_dirs(3);+ let paths: Vec<PathBuf> = held.iter().map(|(_, path)| path.clone()).collect();+ let bus = InMemoryBus::new();++ // n3 is in the membership but not started. Two of three is a majority.+ let mut nodes = vec![+ open_node("n1", &ids, &bus, &paths[0]).await,+ open_node("n2", &ids, &bus, &paths[1]).await,+ ];+ pump_until(&mut nodes, |nodes| {+ nodes.iter().filter(|n| n.is_leader()).count() == 1+ })+ .await;+ let leader = nodes.iter().position(|n| n.is_leader()).expect("leader");+ nodes[leader]+ .propose(ClusterCommand::CreateIndex {+ name: "orders".into(),+ schema_json: "{}".into(),+ })+ .expect("propose create");+ nodes[leader]+ .propose(ClusterCommand::SetShardInSync {+ index: "orders".into(),+ shard: 0,+ nodes: vec!["n2".into()],+ })+ .expect("propose in-sync");+ pump_until(&mut nodes, |nodes| nodes.iter().all(has_orders)).await;++ let events = Arc::new(Mutex::new(Vec::new()));+ let mut late = open_node("n3", &ids, &bus, &paths[2]).await;+ late.set_applier(Arc::new(RecordingApplier {+ events: Arc::clone(&events),+ }));+ nodes.push(late);+ pump_until(&mut nodes, |nodes| nodes.iter().all(has_orders)).await;+ assert!(+ nodes[2].raft.log_len() >= 2,+ "the empty member must have stored the leader's entries"+ );+ let caught_up = events.lock().expect("events").clone();+ assert!(+ caught_up.iter().any(|e| e == "create:orders"),+ "the applier runs as the empty member catches up: {caught_up:?}"+ );++ // Drop the late member and reopen only its directory. The other two+ // stay up so a missing disk log cannot be refilled before the assert.+ let late = nodes.pop().expect("n3");+ assert_eq!(late.raft.id, "n3");+ drop(late);+ let mut restarted = open_node("n3", &ids, &bus, &paths[2]).await;+ assert!(+ !restarted.metadata.indices.contains_key("orders"),+ "restart loads the log and does not apply it until replay"+ );+ restarted.replay_committed().await;+ assert!(+ has_orders(&restarted),+ "the caught-up log must restore metadata with the peers stopped from helping"+ );+}diff --git a/engine/crates/xerj-cluster/tests/persistent_raft_test.rs b/engine/crates/xerj-cluster/tests/persistent_raft_test.rsindex 790443a8d..e28e17892 100644--- a/engine/crates/xerj-cluster/tests/persistent_raft_test.rs+++ b/engine/crates/xerj-cluster/tests/persistent_raft_test.rs@@ -8,7 +8,7 @@ use std::fs;use tempfile::tempdir;-use xerj_cluster::raft::{ClusterCommand, RaftNode};+use xerj_cluster::raft::{ClusterCommand, LogEntry, RaftMessage, RaftNode};#[test]fn single_node_persistent_roundtrip() {@@ -74,3 +74,121 @@ fn single_node_persistent_roundtrip() {"reopening the log must not re-serialize existing entries");}++fn entry(term: u64, index: u64, name: &str) -> LogEntry {+ LogEntry {+ term,+ index,+ command: ClusterCommand::CreateIndex {+ name: name.to_string(),+ schema_json: "{}".into(),+ },+ }+}++fn append_ok(replies: &[xerj_cluster::raft::OutboundMessage]) -> bool {+ replies.iter().any(|out| {+ matches!(+ out.msg,+ RaftMessage::AppendEntriesResponse { success: true, .. }+ )+ })+}++fn vote_granted(replies: &[xerj_cluster::raft::OutboundMessage]) -> bool {+ replies.iter().any(|out| {+ matches!(+ out.msg,+ RaftMessage::RequestVoteResponse {+ vote_granted: true,+ ..+ }+ )+ })+}++/// A follower has to store entries before it says yes. Reopening must not+/// bring back a tail the leader already replaced.+#[test]+fn follower_persists_appended_entries_and_drops_a_replaced_tail() {+ let dir = tempdir().unwrap();+ let path = dir.path();+ let mut node = RaftNode::with_storage("n2".into(), vec!["n1".into()], path).unwrap();++ let replies = node.handle_message(RaftMessage::AppendEntries {+ term: 1,+ leader_id: "n1".into(),+ prev_log_index: 0,+ prev_log_term: 0,+ entries: vec![entry(1, 1, "a"), entry(1, 2, "b"), entry(1, 3, "c")],+ leader_commit: 1,+ });+ assert!(append_ok(&replies), "first append must be stored");+ assert_eq!(node.commit_index(), 1);+ assert_eq!(node.log_len(), 3);++ let replies = node.handle_message(RaftMessage::AppendEntries {+ term: 2,+ leader_id: "n1".into(),+ prev_log_index: 1,+ prev_log_term: 1,+ entries: vec![entry(2, 2, "b2")],+ leader_commit: 2,+ });+ assert!(append_ok(&replies), "conflicting tail must be replaced");+ assert_eq!(node.entry_term(2), Some(2));+ assert_eq!(node.entry_term(3), None);++ drop(node);+ let node = RaftNode::with_storage("n2".into(), vec!["n1".into()], path).unwrap();+ assert_eq!(+ node.log_len(),+ 2,+ "index 3 must not come back from the file"+ );+ assert_eq!(node.entry_term(1), Some(1));+ assert_eq!(node.entry_term(2), Some(2));+ assert_eq!(node.entry_term(3), None);+ assert_eq!(node.commit_index(), 2);+}++/// A vote that was granted has to stick across a restart in the same term.+#[test]+fn a_restart_does_not_grant_a_second_vote_in_the_same_term() {+ let dir = tempdir().unwrap();+ let path = dir.path();+ let mut node =+ RaftNode::with_storage("n1".into(), vec!["n2".into(), "n3".into()], path).unwrap();+ let replies = node.handle_message(RaftMessage::RequestVote {+ term: 1,+ candidate_id: "n2".into(),+ last_log_index: 0,+ last_log_term: 0,+ });+ assert!(vote_granted(&replies), "first vote in the term is granted");++ drop(node);+ let mut node =+ RaftNode::with_storage("n1".into(), vec!["n2".into(), "n3".into()], path).unwrap();+ assert_eq!(node.current_term(), 1);+ let replies = node.handle_message(RaftMessage::RequestVote {+ term: 1,+ candidate_id: "n3".into(),+ last_log_index: 0,+ last_log_term: 0,+ });+ assert!(+ !vote_granted(&replies),+ "a second candidate must not get a vote in this term"+ );+ let replies = node.handle_message(RaftMessage::RequestVote {+ term: 1,+ candidate_id: "n2".into(),+ last_log_index: 0,+ last_log_term: 0,+ });+ assert!(+ vote_granted(&replies),+ "the original candidate is still the vote"+ );+}diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex f58db54d3..4aa83a954 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -4248,7 +4248,32 @@ async fn async_main() -> Result<()> {.filter(|id| id != &node_id).collect();- let node = ClusterNode::new(node_id.clone(), peer_ids, Box::new(transport));+ // The ring's metadata lives in this directory. An in-memory log+ // would boot a restarted node with no shard plan. Opening it+ // is not optional: a failure here refuses to start, rather+ // than serving single-node writes the peers still consider+ // replicated.+ let raft_dir = xerj_cluster::raft_log::raft_storage_dir(&cfg.server.data_dir);+ let node = ClusterNode::with_storage(+ node_id.clone(),+ peer_ids,+ Box::new(transport),+ &raft_dir,+ )+ .with_context(|| {+ format!(+ "open the on-disk raft log at {}; refusing to start cluster mode \+ with an in-memory log",+ raft_dir.display()+ )+ })?;+ info!(+ node = %node_id,+ dir = %raft_dir.display(),+ commit = node.raft.commit_index(),+ entries = node.raft.log_len(),+ "opened on-disk raft log"+ );cluster_transport_outcome = xerj_api::state::ClusterTransportStatus::Running;pending_cluster = Some((node, tick, shutdown_rx));}@@ -4465,12 +4490,16 @@ async fn async_main() -> Result<()> {}));let node_id = node.raft.id.clone();let handle = node.handle();+ // Attach before replay. Replica-store recovery asks which node this+ // is; before attach, that id is the single-node "local" and no plan+ // names it.state.engine.attach_cluster(handle.clone());// Document and search forwarding. Installed before the runner and// before HTTP accept, so a committed shard plan can be served.handle.set_shard_handler(std::sync::Arc::new(xerj_api::cluster_shard::ApiShardHandler::new(state.clone()),));+ node.replay_committed().await;let mut runner = ClusterRunner::new(node, tick, shutdown_rx);info!(node = %node_id, "Starting cluster runner (Raft mode)");tokio::spawn(async move { runner.run().await });diff --git a/landing/docs/clustering.html b/landing/docs/clustering.htmlindex b458382a7..5e606888d 100644--- a/landing/docs/clustering.html+++ b/landing/docs/clustering.html@@ -234,17 +234,17 @@<h1>Clustering</h1>- <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A node that missed the log does not catch up (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>+ <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log, install committed metadata from the on-disk Raft log. Documents that member never stored are not copied (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p><h2>What cluster mode does not give you yet</h2><div class="enum-list"><div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no <code class="inline">wait_for_active_shards</code>. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is <code class="inline">copies / 2 + 1</code>. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it. Nothing replays the writes it missed, so it stays out. If fewer replicas confirm than that quorum, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, and a retry of the index copies it to the replicas still in sync. <code class="inline">number_of_replicas</code> 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.</span></div><div class="item">Single-document routing only<span class="dsc"><code class="inline">_bulk</code>, <code class="inline">_update</code>, update-by-query, and the native <code class="inline">/v1</code> ingest paths still write on the node that received them. They do not consult the shard plan.</span></div><div class="item">Search merges hits, not every clause<span class="dsc">A query, including <code class="inline">match_all</code>, is run on each owner and the hits are merged by the per-shard score (not a global IDF), then <code class="inline">from</code>/<code class="inline">size</code> is applied. Aggregations, suggest, collapse, rescore, profile, <code class="inline">search_after</code>, and scroll are refused with <code class="inline">400</code> when the index has more than one owner — the response is not a partial merge. <code class="inline">/_search</code> with no index, and a comma-separated or wildcard index list, still search only the node that received them. The per-shard window is capped at 10,000 hits.</span></div>- <div class="item">No snapshot catch-up<span class="dsc">A node that missed a metadata entry does not install it later. The server's Raft log is in memory. Restart keeps indices and documents already stored in that node's data directory, including a replica copy, and does not replay the log, so the shard plan is gone and later writes are not routed. The surviving bytes are not offered as a replica until the plan exists again.</span></div>+ <div class="item">No document backfill<span class="dsc">A restarted member reopens <code class="inline">{data_dir}/_raft</code> and applies the committed log before it serves, so the shard plan and the in-sync set come back and later writes are routed. A member that joins with an empty log copies that log from the leader and applies it, which creates the index metadata and an empty replica store when this node is a replica. It does not copy documents written before it joined. A replica removed from the in-sync set is not filled in. The log is not compacted; there is no InstallSnapshot.</span></div><div class="item">No node-lifecycle APIs<span class="dsc">There is no drain / activate / remove endpoint and no region split-merge. Earlier revisions of this page documented them; the binary has never shipped them.</span></div></div>- <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log, and only while a leader can commit.</p>+ <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log. A restart replays the prefix that member had already stored. A new metadata change still needs a leader who can commit.</p><h2>What you get</h2><ul>@@ -254,7 +254,7 @@<li><strong>Index metadata replication</strong> — <code class="inline">PUT /{index}</code>, <code class="inline">DELETE /{index}</code>, <code class="inline">PUT /{index}/_mapping</code>, and <code class="inline">PUT /{index}/_settings</code> are proposed on the member that received them. A follower forwards to the leader. The HTTP call succeeds only after the entry commits on a majority and this node has applied it. No elected leader, or a commit that does not land in time, is <code class="inline">503</code> and the change is not applied on the caller. With <code class="inline">[cluster] enabled = false</code> these calls stay local, as before.</li><li><strong>Primary shard plan</strong> — the create entry names one primary per member (member ids sorted, shard <code class="inline">i</code> on the <code class="inline">i</code>th) and, when <code class="inline">index.number_of_replicas</code> is set, the next members in that ring as replicas. <code class="inline">Engine::route_write</code> returns the primary, or nothing when this node is the primary or the index has no plan. Document id is the routing key (jump consistent hash).</li><li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. The primary copies an index or delete to each in-sync replica and answers when a quorum of the copies has confirmed. A search of one concrete index fans out to every primary and merges hits, and reads an in-sync replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li>- <li><strong>In-memory Raft log in the server</strong> — the on-disk log type exists and is tested, and the server does not open it. Applied indices and the documents a member owned live in that node's data directory, so a restart still has those bytes. The shard plan does not. It lives in the log entry and in each process's routing table, and neither is reloaded. After a restart, <code class="inline">route_write</code> treats the index as unassigned and a new write stays on the node that accepted it. Creating the index again does not rebuild the plan: the index already exists. Deleting it rebuilds a plan and deletes the documents.</li>+ <li><strong>On-disk Raft log</strong> — cluster mode opens <code class="inline">{data_dir}/_raft</code> (<code class="inline">raft.log</code> plus the commit index, term, and vote). The directory name is not a valid index name, so deleting an index cannot delete the log. On startup the node replays the committed prefix into the shard plan and the in-sync set before it serves. Indices and documents already in that data directory stay; the plan is what tells a later write where they live. A member whose log is empty copies the leader's log and applies it. The log is not compacted.</li></ul><h2>Bootstrapping a 3-node cluster</h2>@@ -332,14 +332,14 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre><h2>Failure modes</h2><div class="enum-list">- <div class="item">Leader crash<span class="dsc">Followers time out and elect a new leader — measured ~250 ms on a 3-node loopback ring, including one split-vote retry. The ring stays stable with the dead member absent. Metadata already applied on the survivors stays; the dead member's in-memory log does not.</span></div>+ <div class="item">Leader crash<span class="dsc">Followers time out and elect a new leader — measured ~250 ms on a 3-node loopback ring, including one split-vote retry. The ring stays stable with the dead member absent. Metadata already applied on the survivors stays. The dead member's log is under its data directory; a restart replays it.</span></div><div class="item">Hung or dead peer<span class="dsc">One WARN on the down transition, then quiet: retries back off 100 ms → 6.4 s, and messages to live peers are never delayed behind a dead one. A peer that answers again logs "peer reachable again" at INFO.</span></div><div class="item">Minority partition<span class="dsc">The minority side cannot reach quorum and its elections never conclude; the majority side keeps its leader. Raft safety holds — at most one leader per term.</span></div>- <div class="item">Node rejoin<span class="dsc">Restart with the same config: the node re-enters the election with an empty in-memory Raft log. Indices it already applied are still in its data directory. A metadata change committed while it was down is not installed. Snapshot catch-up is still open (<a href="https://github.com/xerj-org/xerj/issues/1170">#1170</a>).</span></div>+ <div class="item">Node rejoin<span class="dsc">Restart with the same data directory reloads the log and the shard plan. A metadata entry committed while this node was down is installed from a peer that still has it, once this node is back. Documents this node never stored are not copied. A wiped <code class="inline">_raft</code> directory catches metadata up the same way and starts with empty indices. There is no InstallSnapshot (<a href="https://github.com/xerj-org/xerj/issues/1170">#1170</a>).</span></div></div><h2>When not to cluster</h2>- <p>When you need a replica to accept writes after the primary is gone, a replica that missed a write to rejoin the in-sync set, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, and acknowledges a single-document write once a quorum of its copies has it. It does not promote a replica, and it does not catch a rejoining node or a dropped replica up. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>+ <p>When you need a replica to accept writes after the primary is gone, a replica that missed a write to rejoin the in-sync set, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a single-document write once a quorum of its copies has it, and reloads that metadata when a member restarts. It does not promote a replica, and it does not copy documents onto a node that never stored them. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p><p class="source-note">Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs</p>