Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(cluster): replay the on-disk Raft log when a node restarts (#2313)

xerj-org57b8a5058c5f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b9e8eaeb3..24ea8ad2d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -21,11 +21,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
confirm. Fewer confirmations than the quorum is still HTTP 503
`replica_not_acknowledged` — the primary has the write, and retrying the
index copies it to the replicas still in sync. There is still no
- `wait_for_active_shards`, no replica promotion, and no snapshot catch-up
- for a node that missed the log (part of
+ `wait_for_active_shards` and no replica promotion. A restarted member
+ reloads metadata from the on-disk Raft log; document backfill is still
+ open (part of
[#1170](https://github.com/xerj-org/xerj/issues/1170)). Single-node mode
does not take this path.
+- With `[cluster] enabled = true`, the server opens an on-disk Raft log
+ at `{data_dir}/_raft` and replays its committed prefix before it
+ serves. A restart restores the shard plan and the in-sync set, so a
+ later document write is routed to the primary again. A member that
+ starts with an empty log copies the leader's log and applies it, which
+ installs index metadata (and an empty replica store when this node is
+ a replica). It does not copy documents written before it joined. A
+ replica removed from the in-sync set is not filled in. The log is not
+ compacted; there is no InstallSnapshot. Term and vote are stored with
+ the log, so a restarted node does not grant a second vote in the same
+ term. `[cluster] enabled = false` does not open the log (part of
+ [#1170](https://github.com/xerj-org/xerj/issues/1170)).
+
- A DeepSeek Harness profile bundle in `dsh/` (`xerj-dsh`, Apache-2.0)
inserts the README's `mcp-xerj` stdio row via
`@deepseek-ai/dsh-mcp-client`. The row is still untested on DSH.
diff --git a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
index bf7033549..70e8dde3c 100644
--- a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
+++ b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
@@ -25,6 +25,7 @@ struct Member {
handle: ClusterHandle,
/// Held when the runner is not started, so the proposal channel stays open.
_node: Option<ClusterNode>,
+ runner: Option<tokio::task::JoinHandle<()>>,
_dir: tempfile::TempDir,
}
@@ -55,16 +56,16 @@ async fn ring(start_runners: bool) -> Vec<Member> {
handle.set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(state.clone())));
let engine = std::sync::Arc::clone(&state.engine);
let (shutdown, shutdown_rx) = watch::channel(false);
- let kept = if start_runners {
+ let (kept, runner) = if start_runners {
let mut runner = ClusterRunner::new(node, Duration::from_millis(5), shutdown_rx);
- tokio::spawn(async move {
+ let task = tokio::spawn(async move {
runner.run().await;
});
- None
+ (None, Some(task))
} else {
handle.set_timeout(Duration::from_millis(400));
let _ = shutdown_rx;
- Some(node)
+ (Some(node), None)
};
let app = xerj_api::router::build_es_compat_router(state.clone());
members.push(Member {
@@ -74,6 +75,7 @@ async fn ring(start_runners: bool) -> Vec<Member> {
shutdown,
handle,
_node: kept,
+ runner,
_dir: dir,
});
}
@@ -860,3 +862,156 @@ async fn a_document_is_stored_on_its_shard_owner_and_search_sees_it_from_any_mem
let _ = member.shutdown.send(true);
}
}
+
+async fn shutdown_and_keep(members: Vec<Member>) -> Vec<tempfile::TempDir> {
+ let mut dirs = Vec::with_capacity(members.len());
+ for mut member in members {
+ let _ = member.shutdown.send(true);
+ if let Some(task) = member.runner.take() {
+ match tokio::time::timeout(Duration::from_secs(5), task).await {
+ Ok(Ok(())) => {}
+ other => panic!("cluster runner did not stop: {other:?}"),
+ }
+ }
+ // ApiShardHandler holds AppState, which holds Engine, which holds
+ // this handle. The engine's data-dir node.lock stays taken until
+ // that cycle breaks, so a later Engine::new on the same directory
+ // sees this process as another running node.
+ member.handle.clear_shard_handler();
+ drop(member.app);
+ drop(member.state);
+ drop(member.engine);
+ drop(member.handle);
+ dirs.push(member._dir);
+ }
+ dirs
+}
+
+/// Same ring as [`ring`], but each member opens `{data_dir}/_raft` and
+/// replays the committed prefix before the runner starts.
+async fn durable_ring(dirs: Vec<tempfile::TempDir>) -> Vec<Member> {
+ let bus = InMemoryBus::new();
+ let ids = ["n1", "n2", "n3"];
+ let mut members = Vec::new();
+ for (id, dir) in ids.into_iter().zip(dirs) {
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+
+ let peers: Vec<String> = ids
+ .iter()
+ .filter(|peer| **peer != id)
+ .map(|peer| (*peer).to_string())
+ .collect();
+ let transport = InMemoryTransport::new(id.to_string(), bus.clone()).await;
+ let raft_dir = xerj_cluster::raft_log::raft_storage_dir(dir.path());
+ let mut node =
+ ClusterNode::with_storage(id.to_string(), peers, Box::new(transport), &raft_dir)
+ .expect("open raft log");
+ node.set_applier(std::sync::Arc::new(EngineMetadataApplier {
+ engine: std::sync::Arc::clone(&state.engine),
+ }));
+ let handle = node.handle();
+ state.engine.attach_cluster(handle.clone());
+ handle.set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(state.clone())));
+ node.replay_committed().await;
+ let engine = std::sync::Arc::clone(&state.engine);
+ let (shutdown, shutdown_rx) = watch::channel(false);
+ let mut runner = ClusterRunner::new(node, Duration::from_millis(5), shutdown_rx);
+ let task = tokio::spawn(async move {
+ runner.run().await;
+ });
+ let app = xerj_api::router::build_es_compat_router(state.clone());
+ members.push(Member {
+ app,
+ state,
+ engine,
+ shutdown,
+ handle,
+ _node: None,
+ runner: Some(task),
+ _dir: dir,
+ });
+ }
+ members
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn restart_reloads_the_shard_plan_and_still_routes_the_document() {
+ let dirs = (0..3)
+ .map(|_| tempfile::tempdir().expect("tempdir"))
+ .collect();
+ let members = durable_ring(dirs).await;
+ let dead = [false, false, false];
+ let _leader = wait_leader(&members, &dead).await;
+
+ let (status, body) = put_json(&members[0].app, "/orders", json!({})).await;
+ assert!(status.is_success(), "create: {status} {body}");
+ for member in &members {
+ wait_status(&member.app, "/orders", StatusCode::OK).await;
+ }
+ let (status, body) = put_json(
+ &members[0].app,
+ "/orders/_doc/1",
+ json!({ "title": "kept" }),
+ )
+ .await;
+ assert!(status.is_success(), "write: {status} {body}");
+
+ let owner = members
+ .iter()
+ .position(|member| member.engine.owns_shard("orders", "1"))
+ .expect("one primary");
+ let owner_id = members[owner].handle.node_id().to_string();
+ let other = (0..3).find(|i| *i != owner).expect("a non-owner");
+ assert_eq!(
+ members[other].engine.route_write("orders", "1").as_deref(),
+ Some(owner_id.as_str()),
+ "before restart the non-owner forwards"
+ );
+ let (status, body) = post_json(&members[owner].app, "/orders/_refresh", json!({})).await;
+ assert!(
+ status.is_success(),
+ "refresh before restart: {status} {body}"
+ );
+
+ let dirs = shutdown_and_keep(members).await;
+ let members = durable_ring(dirs).await;
+
+ assert!(
+ members[owner].engine.owns_shard("orders", "1"),
+ "replay must put the primary back on {owner_id}"
+ );
+ assert_eq!(
+ members[other].engine.route_write("orders", "1").as_deref(),
+ Some(owner_id.as_str()),
+ "restart must not treat the index as unassigned"
+ );
+ let (status, body) = get(&members[other].app, "/orders/_doc/1").await;
+ assert!(
+ status.is_success(),
+ "GET from a non-owner after restart: {status} {body}"
+ );
+
+ let _leader = wait_leader(&members, &dead).await;
+ let (status, body) = put_json(
+ &members[other].app,
+ "/orders/_doc/2",
+ json!({ "title": "after" }),
+ )
+ .await;
+ assert!(status.is_success(), "write after restart: {status} {body}");
+ for member in &members {
+ let (status, body) = get(&member.app, "/orders/_doc/2").await;
+ assert!(
+ status.is_success(),
+ "doc 2 readable after restart: {status} {body}"
+ );
+ }
+
+ for member in &members {
+ let _ = member.shutdown.send(true);
+ }
+}
diff --git a/engine/crates/xerj-cluster/src/node.rs b/engine/crates/xerj-cluster/src/node.rs
index 257b77aab..3c8aae835 100644
--- a/engine/crates/xerj-cluster/src/node.rs
+++ b/engine/crates/xerj-cluster/src/node.rs
@@ -114,6 +114,40 @@ impl ClusterNode {
}
}
+ /// Like [`ClusterNode::new`], opening the Raft log at `storage_dir`.
+ ///
+ /// The committed entries are loaded into the Raft log and are not applied
+ /// yet. Call [`ClusterNode::replay_committed`] after [`ClusterNode::set_applier`]
+ /// so the engine sees the same entries the metadata store does. Applying
+ /// here would run before the applier exists and a restarted index plan
+ /// would stay in metadata only.
+ pub fn with_storage(
+ id: String,
+ peers: Vec<String>,
+ transport: Box<dyn ClusterTransport>,
+ storage_dir: impl AsRef<std::path::Path>,
+ ) -> Result<Self> {
+ let mut node = Self::new(id.clone(), peers, transport);
+ let peer_ids = node.raft.peers().to_vec();
+ node.raft = RaftNode::with_storage(id, peer_ids, storage_dir)?;
+ Ok(node)
+ }
+
+ /// Apply every committed log entry that this process has not applied yet.
+ ///
+ /// On a fresh open that is the whole committed prefix. The applier, when
+ /// one is installed, runs for each entry before this returns. Safe to call
+ /// again: a second call finds nothing pending.
+ pub async fn replay_committed(&mut self) {
+ let commit = self.raft.commit_index();
+ if commit == 0 {
+ info!(node = %self.raft.id, "raft log has no committed entries to replay");
+ return;
+ }
+ info!(node = %self.raft.id, commit, "replaying committed raft log");
+ self.apply_ready().await;
+ }
+
/// Handle the engine layer uses to propose metadata and read leadership.
pub fn handle(&self) -> ClusterHandle {
self.handle.clone()
diff --git a/engine/crates/xerj-cluster/src/raft.rs b/engine/crates/xerj-cluster/src/raft.rs
index 4b24c9ced..88a34e01e 100644
--- a/engine/crates/xerj-cluster/src/raft.rs
+++ b/engine/crates/xerj-cluster/src/raft.rs
@@ -99,8 +99,8 @@ pub enum ClusterCommand {
/// `request_id` is unique on the node that accepted the caller. That node
/// completes the waiting request when this entry is applied. Other nodes
/// apply `inner` and ignore the id. The wrapper is persisted with the
- /// log entry so a restarted node (once the server opens the file log)
- /// can still tell client entries from internal ones.
+ /// log entry so a restarted node, which opens the file log, can still
+ /// tell client entries from internal ones.
ClientMetadata {
/// Node that accepted the caller. Only that node completes a waiter.
origin: String,
@@ -356,6 +356,15 @@ impl RaftNode {
}
}
node.commit_index = flog.commit_index();
+ // A vote is only valid in the term it was cast. A hard-state term
+ // ahead of the log (we campaigned or stepped down without appending)
+ // wins; a missing hard state leaves the term reconstructed from the log.
+ if flog.hard_term() > node.current_term {
+ node.current_term = flog.hard_term();
+ }
+ if flog.hard_term() == node.current_term && flog.hard_term() > 0 {
+ node.voted_for = flog.hard_vote().map(str::to_string);
+ }
node.persistent = Some(flog);
Ok(node)
}
@@ -399,6 +408,14 @@ impl RaftNode {
new_term = msg_term,
"Higher term observed — stepping down to follower"
);
+ // The vote from the previous term must not survive into this one.
+ if !self.write_hard_state(msg_term, None) {
+ warn!(
+ node = %self.id,
+ term = msg_term,
+ "failed to persist the higher term; stepping down anyway"
+ );
+ }
self.current_term = msg_term;
self.voted_for = None;
self.become_follower(None);
@@ -521,6 +538,11 @@ impl RaftNode {
.map_err(|e| anyhow::anyhow!("append raft log: {e}"))?;
flog.fsync()
.map_err(|e| anyhow::anyhow!("fsync raft log: {e}"))?;
+ if let Err(e) = flog.set_log_end(index) {
+ flog.truncate_from(index);
+ let _ = flog.set_log_end(index.saturating_sub(1));
+ return Err(anyhow::anyhow!("persist raft log end: {e}"));
+ }
}
self.log.push(entry);
@@ -626,6 +648,97 @@ impl RaftNode {
}
}
+ /// Persist `term` and `voted_for` before they take effect.
+ ///
+ /// Returns `true` when there is no file log, or the write replaced
+ /// `hard.meta`. A `false` means the caller must not grant a vote or
+ /// start an election: a crash would otherwise forget the vote and
+ /// grant a second one in the same term.
+ fn write_hard_state(&mut self, term: u64, voted_for: Option<String>) -> bool {
+ let Some(flog) = self.persistent.as_mut() else {
+ return true;
+ };
+ match flog.set_hard_state(term, voted_for.as_deref()) {
+ Ok(()) => true,
+ Err(e) => {
+ warn!(node = %self.id, error = %e, term, "failed to persist raft term and vote");
+ false
+ }
+ }
+ }
+
+ /// Copy the in-memory log onto the file log and fsync it.
+ ///
+ /// Heartbeats that do not change the log do not fsync. Returns `false`
+ /// when the file log could not be updated; the caller must not tell the
+ /// leader the append succeeded.
+ fn mirror_log(&mut self) -> bool {
+ if self.persistent.is_none() {
+ return true;
+ }
+ let node_id = self.id.clone();
+ let mem_last = self.last_log_index();
+ let encoded: Vec<(u64, u64, Vec<u8>)> = match self
+ .log
+ .iter()
+ .map(|entry| {
+ serde_json::to_vec(&entry.command).map(|payload| (entry.index, entry.term, payload))
+ })
+ .collect()
+ {
+ Ok(entries) => entries,
+ Err(e) => {
+ warn!(node = %node_id, error = %e, "encode raft entry");
+ return false;
+ }
+ };
+ let flog = self
+ .persistent
+ .as_mut()
+ .expect("persistent log checked above");
+ let mut dirty = false;
+ if flog.last_index() > mem_last {
+ flog.truncate_from(mem_last + 1);
+ dirty = true;
+ }
+ for (index, term, payload) in &encoded {
+ match flog.term_of(*index) {
+ Some(have) if have == *term => {}
+ Some(_) => {
+ flog.truncate_from(*index);
+ if let Err(e) = flog.append(*term, *index, payload) {
+ warn!(node = %node_id, error = %e, index, "append raft log");
+ return false;
+ }
+ dirty = true;
+ }
+ None => {
+ if let Err(e) = flog.append(*term, *index, payload) {
+ warn!(node = %node_id, error = %e, index, "append raft log");
+ return false;
+ }
+ dirty = true;
+ }
+ }
+ }
+ if !dirty {
+ match flog.persisted_log_end() {
+ Some(end) if end == mem_last => return true,
+ None if flog.last_index() == mem_last => return true,
+ _ => {}
+ }
+ }
+ if let Err(e) = flog.fsync() {
+ warn!(node = %node_id, error = %e, "fsync raft log");
+ return false;
+ }
+ if let Err(e) = flog.set_log_end(mem_last) {
+ warn!(node = %node_id, error = %e, "persist raft log end");
+ return false;
+ }
+ true
+ }
+
fn cluster_size(&self) -> usize {
self.peers.len() + 1 // peers + self
}
@@ -691,9 +804,17 @@ impl RaftNode {
}
fn start_election(&mut self) -> Vec<OutboundMessage> {
- self.current_term += 1;
+ let new_term = self.current_term + 1;
+ let vote = self.id.clone();
+ // The self-vote has to be durable before the RequestVote goes out.
+ // A failed disk stays a follower and tries again after the next timeout.
+ if !self.write_hard_state(new_term, Some(vote.clone())) {
+ self.last_heartbeat = Instant::now();
+ return vec![];
+ }
+ self.current_term = new_term;
self.state = RaftState::Candidate;
- self.voted_for = Some(self.id.clone()); // vote for self
+ self.voted_for = Some(vote);
self.votes_received = 1; // count self-vote
self.election_timeout = Self::random_election_timeout();
self.last_heartbeat = Instant::now();
@@ -819,8 +940,11 @@ impl RaftNode {
Some(e) if e.term != prev_log_term => {
// 3. If an existing entry conflicts with a new one (same index,
// different terms), delete the existing entry and all that
- // follow it (§5.3)
+ // follow it (§5.3). The shortened log is durable before
+ // the rejection, so a crash cannot bring the tail back
+ // and win an election with it.
self.log.retain(|e| e.index < prev_log_index);
+ let _ = self.mirror_log();
return vec![OutboundMessage {
to: leader_id,
msg: RaftMessage::AppendEntriesResponse {
@@ -852,6 +976,21 @@ impl RaftNode {
// Ensure log is sorted by index (should already be, but be safe)
self.log.sort_by_key(|e| e.index);
+ // Durable before the success reply. A heartbeat that changes nothing
+ // does not fsync. A failed mirror is a rejection: the leader retries,
+ // and this process still has the entries in memory.
+ if !self.mirror_log() {
+ return vec![OutboundMessage {
+ to: leader_id,
+ msg: RaftMessage::AppendEntriesResponse {
+ from: self.id.clone(),
+ term: self.current_term,
+ success: false,
+ match_index: 0,
+ },
+ }];
+ }
+
// 5. If leaderCommit > commitIndex, set commitIndex =
// min(leaderCommit, index of last new entry) (§5.3)
if leader_commit > self.commit_index {
@@ -965,10 +1104,14 @@ impl RaftNode {
&& last_log_index >= self.last_log_index());
if can_vote && candidate_log_ok {
- self.voted_for = Some(candidate_id.clone());
- self.last_heartbeat = Instant::now(); // reset election timer
- vote_granted = true;
- debug!(node = %self.id, candidate = %candidate_id, "Granting vote");
+ // Grant only after the vote is durable. Otherwise a restart
+ // in this term would vote for someone else.
+ if self.write_hard_state(self.current_term, Some(candidate_id.clone())) {
+ self.voted_for = Some(candidate_id.clone());
+ self.last_heartbeat = Instant::now(); // reset election timer
+ vote_granted = true;
+ debug!(node = %self.id, candidate = %candidate_id, "Granting vote");
+ }
}
}
diff --git a/engine/crates/xerj-cluster/src/raft_log.rs b/engine/crates/xerj-cluster/src/raft_log.rs
index 199d8962e..b55b9ef44 100644
--- a/engine/crates/xerj-cluster/src/raft_log.rs
+++ b/engine/crates/xerj-cluster/src/raft_log.rs
@@ -25,9 +25,14 @@
//! `BTreeMap<index, (term, file_offset, payload_length)>`, and re-derive
//! `commit_index` + `last_applied` from a sidecar `commit.meta` file.
//!
-//! `fsync` is batched: by default every 100 ms or whenever 256 entries
-//! accumulate in the OS page cache, whichever comes first. Both values
-//! are tunable via `[cluster] raft_fsync_ms = 100` / `raft_fsync_batch = 256`.
+//! The server opens this log at `{data_dir}/_raft` (`RAFT_DIR_NAME`).
+//! `_raft` is not a valid index name, and the directory has no `wal`
+//! child, so the engine's index scan does not open it.
+//!
+//! `hard.meta` stores the current term, the vote, and the logical end of
+//! the log. Truncation does not rewrite `raft.log`; entries past that end
+//! stay in the file and are ignored on the next open. A missing `hard.meta`
+//! (a log written before this file existed) keeps every CRC-valid record.
//!
//! Exit criterion for M5.1 (verified by `raft_log_persist_test`):
//!
@@ -55,8 +60,19 @@ use crc32fast::Hasher as Crc32;
/// Magic for the raft log file format. Distinct from the WAL magic so
/// we never confuse one for the other on recovery.
const MAGIC: u32 = 0x5A52_4C32; // "ZRL2" (ASCII Z R L 2)
+const HARD_MAGIC: u32 = 0x5A52_4853; // "ZRHS"
const LOG_FILE: &str = "raft.log";
const COMMIT_FILE: &str = "commit.meta";
+const HARD_FILE: &str = "hard.meta";
+
+/// Directory name under the node's data dir. Not a valid index name, so a
+/// user index cannot occupy the same path and take the log with it on delete.
+pub const RAFT_DIR_NAME: &str = "_raft";
+
+/// `{data_dir}/_raft`, where the server opens [`FileRaftLog`].
+pub fn raft_storage_dir(data_dir: impl AsRef<Path>) -> PathBuf {
+ data_dir.as_ref().join(RAFT_DIR_NAME)
+}
/// Raft payload kinds. We only use `ClusterCommand` today but leave room
/// for future out-of-band messages (config changes, snapshots).
@@ -96,6 +112,13 @@ pub struct FileRaftLog {
last_index: u64,
/// Commit index persisted to `commit.meta`.
commit_index: u64,
+ /// Current term persisted in `hard.meta`. 0 when that file is absent.
+ hard_term: u64,
+ /// Vote in `hard_term`, when one was persisted.
+ hard_vote: Option<String>,
+ /// Logical end of the log. `None` means `hard.meta` did not record one
+ /// and the scan is authoritative. `Some(0)` is an empty log.
+ log_end: Option<u64>,
}
impl FileRaftLog {
@@ -197,6 +220,11 @@ impl FileRaftLog {
// Load committed index (best-effort).
let commit_index = Self::load_commit_index(&dir).unwrap_or(0);
+ let (hard_term, hard_vote, log_end) = Self::load_hard_state(&dir)?;
+ if let Some(end) = log_end {
+ seek_table.retain(|&idx, _| idx <= end);
+ last_index = seek_table.keys().next_back().copied().unwrap_or(0);
+ }
Ok(Self {
dir,
@@ -204,6 +232,9 @@ impl FileRaftLog {
seek_table,
last_index,
commit_index,
+ hard_term,
+ hard_vote,
+ log_end,
})
}
@@ -295,14 +326,13 @@ impl FileRaftLog {
Ok(Some((term, payload)))
}
- /// Truncate the log by removing every entry with `index >= from`.
+ /// Drop every seek-table entry with `index >= from`.
///
- /// This is called by Raft when a leader steps down and the new leader
- /// instructs the follower to overwrite conflicting tail entries.
- /// We don't physically truncate the file — the seek table evicts the
- /// removed entries, the CRC of later writes will differ, and on
- /// recovery we stop reading at the first CRC mismatch. That keeps
- /// this method O(log n) instead of O(file_size).
+ /// The file is not rewritten. The drop becomes durable when
+ /// [`FileRaftLog::set_log_end`] records the new end; until that file
+ /// is replaced, a crash reopens the previous tail. Callers that have
+ /// already decided the tail is gone must set the end before they
+ /// acknowledge the append that conflicted.
pub fn truncate_from(&mut self, from: u64) {
self.seek_table.retain(|&idx, _| idx < from);
self.last_index = self
@@ -324,6 +354,140 @@ impl FileRaftLog {
Ok(())
}
+ /// Term and vote from `hard.meta`. Term 0 and a missing vote mean the
+ /// file was absent.
+ pub fn hard_term(&self) -> u64 {
+ self.hard_term
+ }
+
+ pub fn hard_vote(&self) -> Option<&str> {
+ self.hard_vote.as_deref()
+ }
+
+ /// Logical end recorded in `hard.meta`, if that file had one.
+ pub fn persisted_log_end(&self) -> Option<u64> {
+ self.log_end
+ }
+
+ /// Record the current term and vote. The logical log end is left as it is.
+ ///
+ /// Memory is updated only after `hard.meta` is replaced. A failed write
+ /// leaves the previous term and vote in place, so a later log-end save
+ /// cannot publish an election this process aborted.
+ pub fn set_hard_state(&mut self, term: u64, voted_for: Option<&str>) -> Result<()> {
+ self.persist_hard(term, voted_for, self.log_end)?;
+ self.hard_term = term;
+ self.hard_vote = voted_for.map(str::to_string);
+ Ok(())
+ }
+
+ /// Record the highest index that is still part of the log.
+ ///
+ /// The file is replaced before the seek table drops indexes above `end`.
+ /// A failed write leaves those entries readable. The next open ignores
+ /// them once this call has succeeded.
+ pub fn set_log_end(&mut self, end: u64) -> Result<()> {
+ self.persist_hard(self.hard_term, self.hard_vote.as_deref(), Some(end))?;
+ if end < self.last_index {
+ self.truncate_from(end + 1);
+ }
+ self.log_end = Some(end);
+ Ok(())
+ }
+
+ /// Replace `hard.meta`. The caller updates its fields only after this
+ /// returns `Ok`.
+ fn persist_hard(&self, term: u64, vote: Option<&str>, log_end: Option<u64>) -> Result<()> {
+ let path = self.dir.join(HARD_FILE);
+ let tmp = self.dir.join("hard.meta.tmp");
+ let mut buf = Vec::with_capacity(64);
+ buf.extend_from_slice(&HARD_MAGIC.to_le_bytes());
+ buf.extend_from_slice(&1u32.to_le_bytes());
+ buf.extend_from_slice(&term.to_le_bytes());
+ match log_end {
+ Some(end) => {
+ buf.push(1);
+ buf.extend_from_slice(&end.to_le_bytes());
+ }
+ None => {
+ buf.push(0);
+ buf.extend_from_slice(&0u64.to_le_bytes());
+ }
+ }
+ match vote {
+ Some(vote) => {
+ let bytes = vote.as_bytes();
+ let len = u32::try_from(bytes.len()).context("raft vote longer than u32")?;
+ buf.extend_from_slice(&len.to_le_bytes());
+ buf.extend_from_slice(bytes);
+ }
+ None => buf.extend_from_slice(&u32::MAX.to_le_bytes()),
+ }
+ {
+ let mut file = OpenOptions::new()
+ .write(true)
+ .create(true)
+ .truncate(true)
+ .open(&tmp)
+ .with_context(|| format!("writing {tmp:?}"))?;
+ file.write_all(&buf)
+ .with_context(|| format!("writing {tmp:?}"))?;
+ file.sync_all()
+ .with_context(|| format!("syncing {tmp:?}"))?;
+ }
+ std::fs::rename(&tmp, &path).with_context(|| format!("replacing {path:?}"))?;
+ // The rename is the publication. Sync the directory so a crash cannot
+ // lose the directory entry that names the new file.
+ let dir =
+ File::open(&self.dir).with_context(|| format!("opening {:?} to sync", self.dir))?;
+ dir.sync_all()
+ .with_context(|| format!("syncing raft dir {:?}", self.dir))?;
+ Ok(())
+ }
+
+ fn load_hard_state(dir: &Path) -> Result<(u64, Option<String>, Option<u64>)> {
+ let path = dir.join(HARD_FILE);
+ if !path.exists() {
+ return Ok((0, None, None));
+ }
+ let bytes = std::fs::read(&path).with_context(|| format!("reading {path:?}"))?;
+ if bytes.len() < 4 + 4 + 8 + 1 + 8 + 4 {
+ return Err(anyhow!("raft hard state {path:?} is truncated"));
+ }
+ let magic = u32::from_le_bytes(bytes[0..4].try_into().expect("4 bytes"));
+ if magic != HARD_MAGIC {
+ return Err(anyhow!("raft hard state {path:?} has a bad magic"));
+ }
+ let version = u32::from_le_bytes(bytes[4..8].try_into().expect("4 bytes"));
+ if version != 1 {
+ return Err(anyhow!(
+ "raft hard state {path:?} has version {version}, want 1"
+ ));
+ }
+ let term = u64::from_le_bytes(bytes[8..16].try_into().expect("8 bytes"));
+ let has_end = bytes[16];
+ let end = u64::from_le_bytes(bytes[17..25].try_into().expect("8 bytes"));
+ let log_end = match has_end {
+ 0 => None,
+ 1 => Some(end),
+ other => return Err(anyhow!("raft hard state {path:?} has log-end flag {other}")),
+ };
+ let vote_len = u32::from_le_bytes(bytes[25..29].try_into().expect("4 bytes"));
+ let vote = if vote_len == u32::MAX {
+ None
+ } else {
+ let start = 29;
+ let end_at = start + vote_len as usize;
+ if bytes.len() < end_at {
+ return Err(anyhow!("raft hard state {path:?} vote is truncated"));
+ }
+ let text = std::str::from_utf8(&bytes[start..end_at])
+ .with_context(|| format!("raft vote in {path:?} is not utf-8"))?;
+ Some(text.to_string())
+ };
+ Ok((term, vote, log_end))
+ }
+
/// Highest log index appended so far (0 if the log is empty).
pub fn last_index(&self) -> u64 {
self.last_index
@@ -424,4 +588,56 @@ mod tests {
assert!(log2.len() >= 4);
assert!(log2.len() <= 5);
}
+
+ #[test]
+ fn log_end_hides_a_truncated_tail_on_reopen() {
+ let dir = tempdir().unwrap();
+ let mut log = FileRaftLog::open(dir.path()).unwrap();
+ for i in 1..=5 {
+ log.append(1, i, format!("{i}").as_bytes()).unwrap();
+ }
+ log.fsync().unwrap();
+ log.truncate_from(4);
+ log.set_log_end(3).unwrap();
+ drop(log);
+
+ let mut log2 = FileRaftLog::open(dir.path()).unwrap();
+ assert_eq!(log2.last_index(), 3);
+ assert!(log2.read(4).unwrap().is_none());
+ assert!(log2.read(5).unwrap().is_none());
+ assert_eq!(log2.read(3).unwrap().unwrap().1, b"3");
+
+ log2.append(2, 4, b"new").unwrap();
+ log2.fsync().unwrap();
+ log2.set_log_end(4).unwrap();
+ drop(log2);
+
+ let mut log3 = FileRaftLog::open(dir.path()).unwrap();
+ assert_eq!(log3.last_index(), 4);
+ assert_eq!(log3.read(4).unwrap().unwrap().1, b"new");
+ assert!(log3.read(5).unwrap().is_none());
+ }
+
+ #[test]
+ fn hard_state_roundtrip_and_a_truncated_file_refuses_to_open() {
+ let dir = tempdir().unwrap();
+ let mut log = FileRaftLog::open(dir.path()).unwrap();
+ log.set_hard_state(4, Some("n2")).unwrap();
+ log.set_log_end(0).unwrap();
+ drop(log);
+
+ let log2 = FileRaftLog::open(dir.path()).unwrap();
+ assert_eq!(log2.hard_term(), 4);
+ assert_eq!(log2.hard_vote(), Some("n2"));
+ assert_eq!(log2.persisted_log_end(), Some(0));
+
+ std::fs::write(dir.path().join(HARD_FILE), [1u8, 2, 3]).unwrap();
+ match FileRaftLog::open(dir.path()) {
+ Err(err) => assert!(
+ err.to_string().contains("truncated"),
+ "corrupt hard state must not look like no vote: {err}"
+ ),
+ Ok(_) => panic!("corrupt hard state opened"),
+ }
+ }
}
diff --git a/engine/crates/xerj-cluster/tests/log_catchup.rs b/engine/crates/xerj-cluster/tests/log_catchup.rs
new file mode 100644
index 000000000..b7bc59d72
--- /dev/null
+++ b/engine/crates/xerj-cluster/tests/log_catchup.rs
@@ -0,0 +1,236 @@
+//! The file Raft log is what a restarted member and a member that starts
+//! with an empty directory use to install committed metadata.
+//!
+//! Document bytes are not in this log. These tests stop at the metadata
+//! state machine: the index and the in-sync set.
+
+use std::path::PathBuf;
+use std::sync::{Arc, Mutex};
+use std::time::{Duration, Instant};
+
+use async_trait::async_trait;
+use tempfile::TempDir;
+use xerj_cluster::node::in_memory::{InMemoryBus, InMemoryTransport};
+use xerj_cluster::{ApplyEffect, ClusterCommand, ClusterNode, MetadataApplier};
+
+struct RecordingApplier {
+ events: Arc<Mutex<Vec<String>>>,
+}
+
+#[async_trait]
+impl MetadataApplier for RecordingApplier {
+ async fn apply_metadata(&self, command: &ClusterCommand) -> ApplyEffect {
+ let label = match command.as_metadata() {
+ ClusterCommand::CreateIndex { name, .. } => format!("create:{name}"),
+ ClusterCommand::SetShardInSync { index, shard, .. } => {
+ format!("insync:{index}:{shard}")
+ }
+ other => format!("other:{other:?}"),
+ };
+ self.events.lock().expect("events").push(label);
+ ApplyEffect::Applied
+ }
+}
+
+async fn open_node(id: &str, ids: &[&str], bus: &InMemoryBus, dir: &PathBuf) -> ClusterNode {
+ let peers: Vec<String> = ids
+ .iter()
+ .filter(|peer| **peer != id)
+ .map(|peer| (*peer).to_string())
+ .collect();
+ let transport = InMemoryTransport::new(id.to_string(), bus.clone()).await;
+ ClusterNode::with_storage(id.to_string(), peers, Box::new(transport), dir)
+ .expect("open raft log")
+}
+
+async fn boot(ids: &[&str], dirs: &[PathBuf]) -> Vec<ClusterNode> {
+ let bus = InMemoryBus::new();
+ let mut nodes = Vec::new();
+ for (id, dir) in ids.iter().zip(dirs.iter()) {
+ nodes.push(open_node(id, ids, &bus, dir).await);
+ }
+ nodes
+}
+
+fn fresh_dirs(n: usize) -> Vec<(TempDir, PathBuf)> {
+ (0..n)
+ .map(|_| {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let path = dir.path().to_path_buf();
+ (dir, path)
+ })
+ .collect()
+}
+
+async fn pump_until(nodes: &mut [ClusterNode], mut done: impl FnMut(&[ClusterNode]) -> bool) {
+ let deadline = Instant::now() + Duration::from_secs(10);
+ while Instant::now() < deadline {
+ for node in nodes.iter_mut() {
+ node.step(Duration::from_millis(5))
+ .await
+ .expect("raft step");
+ }
+ if done(nodes) {
+ return;
+ }
+ }
+ let summary: Vec<String> = nodes
+ .iter()
+ .map(|node| {
+ format!(
+ "{} state={} leader={:?} commit={} log={} indices={:?}",
+ node.raft.id,
+ node.raft.state(),
+ node.leader_id(),
+ node.raft.commit_index(),
+ node.raft.log_len(),
+ node.metadata.indices.keys().collect::<Vec<_>>()
+ )
+ })
+ .collect();
+ panic!("timed out: {summary:?}");
+}
+
+fn has_orders(node: &ClusterNode) -> bool {
+ node.metadata.indices.contains_key("orders")
+ && node
+ .metadata
+ .config
+ .get("shard_in_sync:orders:0")
+ .map(String::as_str)
+ == Some("n2")
+}
+
+#[tokio::test]
+async fn restart_replays_the_committed_index_and_in_sync_set() {
+ let ids = ["n1", "n2", "n3"];
+ let held = fresh_dirs(3);
+ let paths: Vec<PathBuf> = held.iter().map(|(_, path)| path.clone()).collect();
+ let mut nodes = boot(&ids, &paths).await;
+
+ pump_until(&mut nodes, |nodes| {
+ nodes.iter().filter(|n| n.is_leader()).count() == 1
+ })
+ .await;
+ let leader = nodes.iter().position(|n| n.is_leader()).expect("leader");
+ nodes[leader]
+ .propose(ClusterCommand::CreateIndex {
+ name: "orders".into(),
+ schema_json: "{}".into(),
+ })
+ .expect("propose create");
+ nodes[leader]
+ .propose(ClusterCommand::SetShardInSync {
+ index: "orders".into(),
+ shard: 0,
+ nodes: vec!["n2".into()],
+ })
+ .expect("propose in-sync");
+ pump_until(&mut nodes, |nodes| nodes.iter().all(has_orders)).await;
+ drop(nodes);
+
+ let mut restored = boot(&ids, &paths).await;
+ for node in &restored {
+ assert!(
+ !node.metadata.indices.contains_key("orders"),
+ "opening the log must not apply it before replay"
+ );
+ assert!(
+ node.raft.commit_index() >= 2,
+ "commit index must be on disk"
+ );
+ }
+ let events = Arc::new(Mutex::new(Vec::new()));
+ for node in &mut restored {
+ node.set_applier(Arc::new(RecordingApplier {
+ events: Arc::clone(&events),
+ }));
+ node.replay_committed().await;
+ assert!(
+ has_orders(node),
+ "replay must restore metadata on {}",
+ node.raft.id
+ );
+ }
+ let events = events.lock().expect("events").clone();
+ assert_eq!(
+ events
+ .iter()
+ .filter(|e| e.as_str() == "create:orders")
+ .count(),
+ 3
+ );
+ assert_eq!(
+ events
+ .iter()
+ .filter(|e| e.as_str() == "insync:orders:0")
+ .count(),
+ 3
+ );
+}
+
+#[tokio::test]
+async fn an_empty_member_installs_the_leader_log_and_keeps_it_across_restart() {
+ let ids = ["n1", "n2", "n3"];
+ let held = fresh_dirs(3);
+ let paths: Vec<PathBuf> = held.iter().map(|(_, path)| path.clone()).collect();
+ let bus = InMemoryBus::new();
+
+ // n3 is in the membership but not started. Two of three is a majority.
+ let mut nodes = vec![
+ open_node("n1", &ids, &bus, &paths[0]).await,
+ open_node("n2", &ids, &bus, &paths[1]).await,
+ ];
+ pump_until(&mut nodes, |nodes| {
+ nodes.iter().filter(|n| n.is_leader()).count() == 1
+ })
+ .await;
+ let leader = nodes.iter().position(|n| n.is_leader()).expect("leader");
+ nodes[leader]
+ .propose(ClusterCommand::CreateIndex {
+ name: "orders".into(),
+ schema_json: "{}".into(),
+ })
+ .expect("propose create");
+ nodes[leader]
+ .propose(ClusterCommand::SetShardInSync {
+ index: "orders".into(),
+ shard: 0,
+ nodes: vec!["n2".into()],
+ })
+ .expect("propose in-sync");
+ pump_until(&mut nodes, |nodes| nodes.iter().all(has_orders)).await;
+
+ let events = Arc::new(Mutex::new(Vec::new()));
+ let mut late = open_node("n3", &ids, &bus, &paths[2]).await;
+ late.set_applier(Arc::new(RecordingApplier {
+ events: Arc::clone(&events),
+ }));
+ nodes.push(late);
+ pump_until(&mut nodes, |nodes| nodes.iter().all(has_orders)).await;
+ assert!(
+ nodes[2].raft.log_len() >= 2,
+ "the empty member must have stored the leader's entries"
+ );
+ let caught_up = events.lock().expect("events").clone();
+ assert!(
+ caught_up.iter().any(|e| e == "create:orders"),
+ "the applier runs as the empty member catches up: {caught_up:?}"
+ );
+
+ // Drop the late member and reopen only its directory. The other two
+ // stay up so a missing disk log cannot be refilled before the assert.
+ let late = nodes.pop().expect("n3");
+ assert_eq!(late.raft.id, "n3");
+ drop(late);
+ let mut restarted = open_node("n3", &ids, &bus, &paths[2]).await;
+ assert!(
+ !restarted.metadata.indices.contains_key("orders"),
+ "restart loads the log and does not apply it until replay"
+ );
+ restarted.replay_committed().await;
+ assert!(
+ has_orders(&restarted),
+ "the caught-up log must restore metadata with the peers stopped from helping"
+ );
+}
diff --git a/engine/crates/xerj-cluster/tests/persistent_raft_test.rs b/engine/crates/xerj-cluster/tests/persistent_raft_test.rs
index 790443a8d..e28e17892 100644
--- a/engine/crates/xerj-cluster/tests/persistent_raft_test.rs
+++ b/engine/crates/xerj-cluster/tests/persistent_raft_test.rs
@@ -8,7 +8,7 @@ use std::fs;
use tempfile::tempdir;
-use xerj_cluster::raft::{ClusterCommand, RaftNode};
+use xerj_cluster::raft::{ClusterCommand, LogEntry, RaftMessage, RaftNode};
#[test]
fn single_node_persistent_roundtrip() {
@@ -74,3 +74,121 @@ fn single_node_persistent_roundtrip() {
"reopening the log must not re-serialize existing entries"
);
}
+
+fn entry(term: u64, index: u64, name: &str) -> LogEntry {
+ LogEntry {
+ term,
+ index,
+ command: ClusterCommand::CreateIndex {
+ name: name.to_string(),
+ schema_json: "{}".into(),
+ },
+ }
+}
+
+fn append_ok(replies: &[xerj_cluster::raft::OutboundMessage]) -> bool {
+ replies.iter().any(|out| {
+ matches!(
+ out.msg,
+ RaftMessage::AppendEntriesResponse { success: true, .. }
+ )
+ })
+}
+
+fn vote_granted(replies: &[xerj_cluster::raft::OutboundMessage]) -> bool {
+ replies.iter().any(|out| {
+ matches!(
+ out.msg,
+ RaftMessage::RequestVoteResponse {
+ vote_granted: true,
+ ..
+ }
+ )
+ })
+}
+
+/// A follower has to store entries before it says yes. Reopening must not
+/// bring back a tail the leader already replaced.
+#[test]
+fn follower_persists_appended_entries_and_drops_a_replaced_tail() {
+ let dir = tempdir().unwrap();
+ let path = dir.path();
+ let mut node = RaftNode::with_storage("n2".into(), vec!["n1".into()], path).unwrap();
+
+ let replies = node.handle_message(RaftMessage::AppendEntries {
+ term: 1,
+ leader_id: "n1".into(),
+ prev_log_index: 0,
+ prev_log_term: 0,
+ entries: vec![entry(1, 1, "a"), entry(1, 2, "b"), entry(1, 3, "c")],
+ leader_commit: 1,
+ });
+ assert!(append_ok(&replies), "first append must be stored");
+ assert_eq!(node.commit_index(), 1);
+ assert_eq!(node.log_len(), 3);
+
+ let replies = node.handle_message(RaftMessage::AppendEntries {
+ term: 2,
+ leader_id: "n1".into(),
+ prev_log_index: 1,
+ prev_log_term: 1,
+ entries: vec![entry(2, 2, "b2")],
+ leader_commit: 2,
+ });
+ assert!(append_ok(&replies), "conflicting tail must be replaced");
+ assert_eq!(node.entry_term(2), Some(2));
+ assert_eq!(node.entry_term(3), None);
+
+ drop(node);
+ let node = RaftNode::with_storage("n2".into(), vec!["n1".into()], path).unwrap();
+ assert_eq!(
+ node.log_len(),
+ 2,
+ "index 3 must not come back from the file"
+ );
+ assert_eq!(node.entry_term(1), Some(1));
+ assert_eq!(node.entry_term(2), Some(2));
+ assert_eq!(node.entry_term(3), None);
+ assert_eq!(node.commit_index(), 2);
+}
+
+/// A vote that was granted has to stick across a restart in the same term.
+#[test]
+fn a_restart_does_not_grant_a_second_vote_in_the_same_term() {
+ let dir = tempdir().unwrap();
+ let path = dir.path();
+ let mut node =
+ RaftNode::with_storage("n1".into(), vec!["n2".into(), "n3".into()], path).unwrap();
+ let replies = node.handle_message(RaftMessage::RequestVote {
+ term: 1,
+ candidate_id: "n2".into(),
+ last_log_index: 0,
+ last_log_term: 0,
+ });
+ assert!(vote_granted(&replies), "first vote in the term is granted");
+
+ drop(node);
+ let mut node =
+ RaftNode::with_storage("n1".into(), vec!["n2".into(), "n3".into()], path).unwrap();
+ assert_eq!(node.current_term(), 1);
+ let replies = node.handle_message(RaftMessage::RequestVote {
+ term: 1,
+ candidate_id: "n3".into(),
+ last_log_index: 0,
+ last_log_term: 0,
+ });
+ assert!(
+ !vote_granted(&replies),
+ "a second candidate must not get a vote in this term"
+ );
+ let replies = node.handle_message(RaftMessage::RequestVote {
+ term: 1,
+ candidate_id: "n2".into(),
+ last_log_index: 0,
+ last_log_term: 0,
+ });
+ assert!(
+ vote_granted(&replies),
+ "the original candidate is still the vote"
+ );
+}
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index f58db54d3..4aa83a954 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -4248,7 +4248,32 @@ async fn async_main() -> Result<()> {
.filter(|id| id != &node_id)
.collect();
- let node = ClusterNode::new(node_id.clone(), peer_ids, Box::new(transport));
+ // The ring's metadata lives in this directory. An in-memory log
+ // would boot a restarted node with no shard plan. Opening it
+ // is not optional: a failure here refuses to start, rather
+ // than serving single-node writes the peers still consider
+ // replicated.
+ let raft_dir = xerj_cluster::raft_log::raft_storage_dir(&cfg.server.data_dir);
+ let node = ClusterNode::with_storage(
+ node_id.clone(),
+ peer_ids,
+ Box::new(transport),
+ &raft_dir,
+ )
+ .with_context(|| {
+ format!(
+ "open the on-disk raft log at {}; refusing to start cluster mode \
+ with an in-memory log",
+ raft_dir.display()
+ )
+ })?;
+ info!(
+ node = %node_id,
+ dir = %raft_dir.display(),
+ commit = node.raft.commit_index(),
+ entries = node.raft.log_len(),
+ "opened on-disk raft log"
+ );
cluster_transport_outcome = xerj_api::state::ClusterTransportStatus::Running;
pending_cluster = Some((node, tick, shutdown_rx));
}
@@ -4465,12 +4490,16 @@ async fn async_main() -> Result<()> {
}));
let node_id = node.raft.id.clone();
let handle = node.handle();
+ // Attach before replay. Replica-store recovery asks which node this
+ // is; before attach, that id is the single-node "local" and no plan
+ // names it.
state.engine.attach_cluster(handle.clone());
// Document and search forwarding. Installed before the runner and
// before HTTP accept, so a committed shard plan can be served.
handle.set_shard_handler(std::sync::Arc::new(
xerj_api::cluster_shard::ApiShardHandler::new(state.clone()),
));
+ node.replay_committed().await;
let mut runner = ClusterRunner::new(node, tick, shutdown_rx);
info!(node = %node_id, "Starting cluster runner (Raft mode)");
tokio::spawn(async move { runner.run().await });
diff --git a/landing/docs/clustering.html b/landing/docs/clustering.html
index b458382a7..5e606888d 100644
--- a/landing/docs/clustering.html
+++ b/landing/docs/clustering.html
@@ -234,17 +234,17 @@
<h1>Clustering</h1>
- <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A node that missed the log does not catch up (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>
+ <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log, install committed metadata from the on-disk Raft log. Documents that member never stored are not copied (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>
<h2>What cluster mode does not give you yet</h2>
<div class="enum-list">
<div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no <code class="inline">wait_for_active_shards</code>. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is <code class="inline">copies / 2 + 1</code>. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it. Nothing replays the writes it missed, so it stays out. If fewer replicas confirm than that quorum, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, and a retry of the index copies it to the replicas still in sync. <code class="inline">number_of_replicas</code> 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.</span></div>
<div class="item">Single-document routing only<span class="dsc"><code class="inline">_bulk</code>, <code class="inline">_update</code>, update-by-query, and the native <code class="inline">/v1</code> ingest paths still write on the node that received them. They do not consult the shard plan.</span></div>
<div class="item">Search merges hits, not every clause<span class="dsc">A query, including <code class="inline">match_all</code>, is run on each owner and the hits are merged by the per-shard score (not a global IDF), then <code class="inline">from</code>/<code class="inline">size</code> is applied. Aggregations, suggest, collapse, rescore, profile, <code class="inline">search_after</code>, and scroll are refused with <code class="inline">400</code> when the index has more than one owner — the response is not a partial merge. <code class="inline">/_search</code> with no index, and a comma-separated or wildcard index list, still search only the node that received them. The per-shard window is capped at 10,000 hits.</span></div>
- <div class="item">No snapshot catch-up<span class="dsc">A node that missed a metadata entry does not install it later. The server's Raft log is in memory. Restart keeps indices and documents already stored in that node's data directory, including a replica copy, and does not replay the log, so the shard plan is gone and later writes are not routed. The surviving bytes are not offered as a replica until the plan exists again.</span></div>
+ <div class="item">No document backfill<span class="dsc">A restarted member reopens <code class="inline">{data_dir}/_raft</code> and applies the committed log before it serves, so the shard plan and the in-sync set come back and later writes are routed. A member that joins with an empty log copies that log from the leader and applies it, which creates the index metadata and an empty replica store when this node is a replica. It does not copy documents written before it joined. A replica removed from the in-sync set is not filled in. The log is not compacted; there is no InstallSnapshot.</span></div>
<div class="item">No node-lifecycle APIs<span class="dsc">There is no drain / activate / remove endpoint and no region split-merge. Earlier revisions of this page documented them; the binary has never shipped them.</span></div>
</div>
- <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log, and only while a leader can commit.</p>
+ <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log. A restart replays the prefix that member had already stored. A new metadata change still needs a leader who can commit.</p>
<h2>What you get</h2>
<ul>
@@ -254,7 +254,7 @@
<li><strong>Index metadata replication</strong> — <code class="inline">PUT /{index}</code>, <code class="inline">DELETE /{index}</code>, <code class="inline">PUT /{index}/_mapping</code>, and <code class="inline">PUT /{index}/_settings</code> are proposed on the member that received them. A follower forwards to the leader. The HTTP call succeeds only after the entry commits on a majority and this node has applied it. No elected leader, or a commit that does not land in time, is <code class="inline">503</code> and the change is not applied on the caller. With <code class="inline">[cluster] enabled = false</code> these calls stay local, as before.</li>
<li><strong>Primary shard plan</strong> — the create entry names one primary per member (member ids sorted, shard <code class="inline">i</code> on the <code class="inline">i</code>th) and, when <code class="inline">index.number_of_replicas</code> is set, the next members in that ring as replicas. <code class="inline">Engine::route_write</code> returns the primary, or nothing when this node is the primary or the index has no plan. Document id is the routing key (jump consistent hash).</li>
<li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. The primary copies an index or delete to each in-sync replica and answers when a quorum of the copies has confirmed. A search of one concrete index fans out to every primary and merges hits, and reads an in-sync replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li>
- <li><strong>In-memory Raft log in the server</strong> — the on-disk log type exists and is tested, and the server does not open it. Applied indices and the documents a member owned live in that node's data directory, so a restart still has those bytes. The shard plan does not. It lives in the log entry and in each process's routing table, and neither is reloaded. After a restart, <code class="inline">route_write</code> treats the index as unassigned and a new write stays on the node that accepted it. Creating the index again does not rebuild the plan: the index already exists. Deleting it rebuilds a plan and deletes the documents.</li>
+ <li><strong>On-disk Raft log</strong> — cluster mode opens <code class="inline">{data_dir}/_raft</code> (<code class="inline">raft.log</code> plus the commit index, term, and vote). The directory name is not a valid index name, so deleting an index cannot delete the log. On startup the node replays the committed prefix into the shard plan and the in-sync set before it serves. Indices and documents already in that data directory stay; the plan is what tells a later write where they live. A member whose log is empty copies the leader's log and applies it. The log is not compacted.</li>
</ul>
<h2>Bootstrapping a 3-node cluster</h2>
@@ -332,14 +332,14 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre>
<h2>Failure modes</h2>
<div class="enum-list">
- <div class="item">Leader crash<span class="dsc">Followers time out and elect a new leader — measured ~250 ms on a 3-node loopback ring, including one split-vote retry. The ring stays stable with the dead member absent. Metadata already applied on the survivors stays; the dead member's in-memory log does not.</span></div>
+ <div class="item">Leader crash<span class="dsc">Followers time out and elect a new leader — measured ~250 ms on a 3-node loopback ring, including one split-vote retry. The ring stays stable with the dead member absent. Metadata already applied on the survivors stays. The dead member's log is under its data directory; a restart replays it.</span></div>
<div class="item">Hung or dead peer<span class="dsc">One WARN on the down transition, then quiet: retries back off 100 ms → 6.4 s, and messages to live peers are never delayed behind a dead one. A peer that answers again logs "peer reachable again" at INFO.</span></div>
<div class="item">Minority partition<span class="dsc">The minority side cannot reach quorum and its elections never conclude; the majority side keeps its leader. Raft safety holds — at most one leader per term.</span></div>
- <div class="item">Node rejoin<span class="dsc">Restart with the same config: the node re-enters the election with an empty in-memory Raft log. Indices it already applied are still in its data directory. A metadata change committed while it was down is not installed. Snapshot catch-up is still open (<a href="https://github.com/xerj-org/xerj/issues/1170">#1170</a>).</span></div>
+ <div class="item">Node rejoin<span class="dsc">Restart with the same data directory reloads the log and the shard plan. A metadata entry committed while this node was down is installed from a peer that still has it, once this node is back. Documents this node never stored are not copied. A wiped <code class="inline">_raft</code> directory catches metadata up the same way and starts with empty indices. There is no InstallSnapshot (<a href="https://github.com/xerj-org/xerj/issues/1170">#1170</a>).</span></div>
</div>
<h2>When not to cluster</h2>
- <p>When you need a replica to accept writes after the primary is gone, a replica that missed a write to rejoin the in-sync set, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, and acknowledges a single-document write once a quorum of its copies has it. It does not promote a replica, and it does not catch a rejoining node or a dropped replica up. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>
+ <p>When you need a replica to accept writes after the primary is gone, a replica that missed a write to rejoin the in-sync set, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a single-document write once a quorum of its copies has it, and reloads that metadata when a member restarts. It does not promote a replica, and it does not copy documents onto a node that never stored them. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>
<p class="source-note">Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs</p>