Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(cluster): copy primary documents onto a replica that is behind (#2328)

xerj-org4f4b613f8758
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3c163e753..82fbb460b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- With `[cluster] enabled = true`, a replica that does not have the
+ primary's live documents copies them and then rejoins the in-sync set.
+ That covers a member that starts after the writes (the create entry
+ opens an empty replica store) and a replica removed from the in-sync
+ set because it missed an acknowledgement. The primary sends the live
+ documents, drops ids the replica still has that the primary no longer
+ does, and publishes the enlarged in-sync set through the Raft log. The
+ primary holds that shard's replica lock across the copy, so a write
+ that arrives during the copy is replicated to the new set when the
+ lock is released. A later single-document index or delete is copied to
+ that replica with the others. A write still needs the primary. Changing
+ membership does not move copies. `[cluster] enabled = false` does not
+ do this (part of
+ [#1170](https://github.com/xerj-org/xerj/issues/1170)).
+
- With `[cluster] enabled = true`, a member compacts its Raft log once
64 metadata entries have been applied since the previous snapshot.
The compacted prefix is stored in `{data_dir}/_raft/snapshot.meta` and
@@ -17,9 +32,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
same index metadata an AppendEntries catch-up would have. The snapshot
is that command prefix, not a smaller folded image and not the
documents. A history that does not fit in 8 MiB stays in the log.
- Restart replays the snapshot and then the entries after it. A node
- that never stored a document still does not receive it, and a replica
- removed from the in-sync set stays out (part of
+ Restart replays the snapshot and then the entries after it. Copying
+ the documents themselves is the replica backfill entry above (part of
[#1170](https://github.com/xerj-org/xerj/issues/1170)).
- With `[cluster] enabled = true`, a single-document index or delete is
@@ -35,8 +49,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`replica_not_acknowledged` — the primary has the write, and retrying the
index copies it to the replicas still in sync. There is still no
`wait_for_active_shards` and no replica promotion. A restarted member
- reloads metadata from the on-disk Raft log; document backfill is still
- open (part of
+ reloads metadata from the on-disk Raft log. A replica that missed the
+ write is filled by the backfill entry above (part of
[#1170](https://github.com/xerj-org/xerj/issues/1170)). Single-node mode
does not take this path.
@@ -46,8 +60,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
later document write is routed to the primary again. A member that
starts with an empty log copies the leader's log and applies it, which
installs index metadata (and an empty replica store when this node is
- a replica). It does not copy documents written before it joined. A
- replica removed from the in-sync set is not filled in. Term and vote are
+ a replica). Documents written before it joined are copied by the
+ backfill entry above. Term and vote are
stored with the log, so a restarted node does not grant a second vote in the same
term. `[cluster] enabled = false` does not open the log (part of
[#1170](https://github.com/xerj-org/xerj/issues/1170)).
diff --git a/engine/crates/xerj-api/src/cluster_shard.rs b/engine/crates/xerj-api/src/cluster_shard.rs
index a892b832d..15b6cd958 100644
--- a/engine/crates/xerj-api/src/cluster_shard.rs
+++ b/engine/crates/xerj-api/src/cluster_shard.rs
@@ -6,7 +6,9 @@
//! copies an index or delete onto the in-sync replicas and acknowledges the
//! caller when a quorum of the copies has confirmed. A replica that does not
//! confirm is dropped from the in-sync set when the rest already make a
-//! quorum. [`fanout_search`] asks every
+//! quorum. That replica copies the primary's live documents when it can
+//! answer again, and the primary then publishes it back into the in-sync
+//! set. [`fanout_search`] asks every
//! primary and merges hits. If a primary does not answer, a GET or a search
//! of that shard reads a replica copy. Single-node mode (`has_cluster`
//! false) returns before any of this.
@@ -15,6 +17,7 @@
//! a primary search does not count them twice. The owner's handler does not
//! route a `replica` apply again.
+use std::collections::HashSet;
use std::sync::Arc;
use std::time::{Duration, Instant};
@@ -581,6 +584,14 @@ impl xerj_cluster::ShardHandler for ApiShardHandler {
body_json,
params_json,
} => local_search(&self.state, &index, &body_json, &params_json).await,
+ ShardOp::BackfillReplica {
+ index,
+ shard,
+ replica_id,
+ } => backfill_replica(&self.state, &index, shard, &replica_id).await,
+ ShardOp::RetainReplicaDocs { index, shard, ids } => {
+ retain_replica_docs(&self.state, &index, shard, &ids).await
+ }
}
}
}
@@ -913,10 +924,11 @@ fn replica_nack(index: &str, replica: &str, detail: &str) -> ShardResult {
///
/// Every in-sync replica is asked. When enough confirm and some do not, the
/// ones that confirmed are published through the Raft log before the caller
-/// is acknowledged, and later reads skip the rest. A replica that fell out
-/// is not added back: nothing in this process replays the writes it missed.
-/// Two writes of the same shard do not publish different sets; the primary
-/// holds `replica_sync_lock` across the copy and the publish.
+/// is acknowledged, and later reads skip the rest until
+/// [`backfill_replica`] copies the primary's live documents and publishes
+/// the replica again. Two writes of the same shard do not publish different
+/// sets; the primary holds `replica_sync_lock` across the copy and the
+/// publish. Backfill takes that same lock.
async fn replicate_write(
state: &AppState,
index: &str,
@@ -1061,6 +1073,256 @@ fn replica_confirmed(op: &ShardOp, result: &ShardResult) -> bool {
}
}
+fn backfill_body(status: u16, type_name: &str, reason: impl Into<String>) -> ShardResult {
+ let reason = reason.into();
+ ShardResult::http(
+ status,
+ json!({
+ "error": { "type": type_name, "reason": reason },
+ "status": status
+ })
+ .to_string(),
+ )
+}
+
+fn already_in_sync() -> ShardResult {
+ ShardResult::http(
+ 200,
+ json!({ "already_in_sync": true, "copied": 0 }).to_string(),
+ )
+}
+
+/// Live documents on this primary for `shard`.
+///
+/// The user index on the primary holds the documents it owns. Ids that
+/// route elsewhere are skipped. A tombstone is not in the list. `Err` is
+/// the reason the copy must not publish an in-sync set.
+async fn primary_live_docs(
+ state: &AppState,
+ index: &str,
+ shard: u32,
+) -> Result<Vec<(String, Value)>, String> {
+ let idx = state
+ .engine
+ .get_index(index)
+ .map_err(|err| format!("primary index [{index}] is not open: {err}"))?;
+ let mut docs = Vec::new();
+ for id in idx.live_document_ids() {
+ let Some((doc_shard, _)) = state.engine.primary_of(index, &id) else {
+ continue;
+ };
+ if doc_shard != shard {
+ continue;
+ }
+ match idx.get_document(&id).await {
+ Ok(Some(source)) => docs.push((id, source)),
+ Ok(None) => {}
+ Err(err) => return Err(format!("reading [{index}] document [{id}]: {err}")),
+ }
+ }
+ Ok(docs)
+}
+
+/// Copy this primary's live documents onto `replica_id` and publish it.
+///
+/// The replica lock is held from the snapshot through the in-sync publish.
+/// A document write applies on the primary first and then waits for this
+/// lock before it copies, so a write that missed the snapshot is replicated
+/// to the set that includes `replica_id`. A failed copy leaves the in-sync
+/// set unchanged. The replica retries.
+async fn backfill_replica(
+ state: &AppState,
+ index: &str,
+ shard: u32,
+ replica_id: &str,
+) -> ShardResult {
+ if replica_id.is_empty() {
+ return backfill_body(400, "illegal_argument_exception", "replica id is empty");
+ }
+ let me = state.engine.routing_node_id();
+ if replica_id == me {
+ return backfill_body(
+ 400,
+ "illegal_argument_exception",
+ "a primary does not backfill itself",
+ );
+ }
+ let Some(owner) = state.engine.shard_primary(index, shard) else {
+ return backfill_body(
+ 409,
+ "shard_not_ready",
+ format!("[{index}] shard {shard} has no primary on this node"),
+ );
+ };
+ if owner != me {
+ return backfill_body(
+ 409,
+ "shard_not_primary",
+ format!("[{index}] shard {shard} is owned by [{owner}], not [{me}]"),
+ );
+ }
+ let placed = state.engine.placed_replica_nodes(index, shard);
+ if !placed.iter().any(|id| id == replica_id) {
+ return backfill_body(
+ 400,
+ "illegal_argument_exception",
+ format!("[{replica_id}] is not a replica of [{index}] shard {shard}"),
+ );
+ }
+ if state
+ .engine
+ .replica_nodes(index, shard)
+ .iter()
+ .any(|id| id == replica_id)
+ {
+ return already_in_sync();
+ }
+
+ let sync = state.engine.replica_sync_lock(index, shard);
+ let _guard = sync.lock().await;
+ if state
+ .engine
+ .replica_nodes(index, shard)
+ .iter()
+ .any(|id| id == replica_id)
+ {
+ return already_in_sync();
+ }
+
+ let docs = match primary_live_docs(state, index, shard).await {
+ Ok(docs) => docs,
+ Err(reason) => return backfill_body(503, "replica_backfill_failed", reason),
+ };
+ for (id, source) in &docs {
+ let source_json = match serde_json::to_string(source) {
+ Ok(json) => json,
+ Err(err) => {
+ return backfill_body(
+ 503,
+ "replica_backfill_failed",
+ format!("document [{id}] is not JSON: {err}"),
+ );
+ }
+ };
+ let op = ShardOp::IndexDoc {
+ index: index.to_string(),
+ id: id.clone(),
+ source_json,
+ op: "index".to_string(),
+ if_seq_no: None,
+ if_primary_term: None,
+ version: None,
+ version_type: None,
+ replica: true,
+ shard,
+ };
+ if let Err(detail) = confirm_replica(state, replica_id, &op).await {
+ return backfill_body(
+ 503,
+ "replica_backfill_failed",
+ format!("replica [{replica_id}] did not take document [{id}]: {detail}"),
+ );
+ }
+ }
+ let ids: Vec<String> = docs.iter().map(|(id, _)| id.clone()).collect();
+ let retain = ShardOp::RetainReplicaDocs {
+ index: index.to_string(),
+ shard,
+ ids,
+ };
+ if let Err(detail) = confirm_replica(state, replica_id, &retain).await {
+ return backfill_body(
+ 503,
+ "replica_backfill_failed",
+ format!(
+ "replica [{replica_id}] did not drop documents the primary no longer has: {detail}"
+ ),
+ );
+ }
+
+ let mut nodes = state.engine.replica_nodes(index, shard);
+ if !nodes.iter().any(|id| id == replica_id) {
+ nodes.push(replica_id.to_string());
+ }
+ nodes.sort();
+ nodes.dedup();
+ let Some(handle) = state.engine.cluster_handle() else {
+ return backfill_body(
+ 503,
+ "replica_backfill_failed",
+ "no cluster handle to publish the in-sync set",
+ );
+ };
+ let copied = docs.len();
+ match handle
+ .propose(ClusterCommand::SetShardInSync {
+ index: index.to_string(),
+ shard,
+ nodes,
+ })
+ .await
+ {
+ Ok(ApplyEffect::Applied) => {
+ let published = state.engine.replica_nodes(index, shard);
+ if !published.iter().any(|id| id == replica_id) {
+ return backfill_body(
+ 503,
+ "replica_backfill_failed",
+ "the in-sync set committed without the replica that was filled",
+ );
+ }
+ tracing::info!(
+ index = %index,
+ shard,
+ replica = %replica_id,
+ copied,
+ "copied primary documents onto a replica and published the in-sync set"
+ );
+ ShardResult::http(
+ 200,
+ json!({ "already_in_sync": false, "copied": copied }).to_string(),
+ )
+ }
+ Ok(other) => backfill_body(
+ 503,
+ "replica_backfill_failed",
+ format!("in-sync update applied as {other:?}"),
+ ),
+ Err(err) => backfill_body(503, "replica_backfill_failed", err.to_string()),
+ }
+}
+
+/// Drop replica-store documents that are not in `ids`.
+async fn retain_replica_docs(
+ state: &AppState,
+ index: &str,
+ shard: u32,
+ ids: &[String],
+) -> ShardResult {
+ let store = match doc_store(index, true, shard) {
+ Ok(store) => store,
+ Err(result) => return result,
+ };
+ let idx = match open_store(state, &store).await {
+ Ok(idx) => idx,
+ Err(_) => return ShardResult::http(503, shard_not_ready_body(index)),
+ };
+ let keep: HashSet<&str> = ids.iter().map(String::as_str).collect();
+ for id in idx.live_document_ids() {
+ if keep.contains(id.as_str()) {
+ continue;
+ }
+ if let Err(err) = idx.delete_document(&id).await {
+ return backfill_body(
+ 503,
+ "replica_backfill_failed",
+ format!("dropping [{id}] from [{store}]: {err}"),
+ );
+ }
+ }
+ ShardResult::http(200, json!({ "retained": ids.len() }).to_string())
+}
+
/// Search the shards `primary` owns by reading one replica copy of each.
///
/// `None` when this index has no replica for that primary, or none of them
diff --git a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
index 60a732736..7f3ca13c1 100644
--- a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
+++ b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
@@ -519,6 +519,9 @@ async fn an_acknowledged_write_is_on_the_replica_and_readable_after_the_primary_
}
async fn wait_in_sync(members: &[Member], shard: u32, want: &[String]) {
+ let mut want = want.to_vec();
+ want.sort();
+ want.dedup();
let deadline = Instant::now() + Duration::from_secs(5);
loop {
let ready = members.iter().all(|member| {
@@ -663,6 +666,10 @@ async fn a_quorum_of_copies_acks_when_one_of_two_replicas_does_not() {
Some("copied"),
"the replica that confirmed has the write: {on_live:?}"
);
+ // The handler on this replica stays cleared. Restoring it would let
+ // document backfill copy the write and put the replica back in the
+ // in-sync set. That path is covered by
+ // `a_replica_dropped_from_the_in_sync_set_copies_the_missed_write_and_rejoins`.
let on_stale = members[replica_a]
.engine
.get_index(&store)
@@ -1169,3 +1176,320 @@ async fn a_late_member_installs_a_compacted_create_and_routes_a_later_write() {
let _ = member.shutdown.send(true);
}
}
+
+async fn replica_title(member: &Member, shard: u32, id: &str) -> Option<String> {
+ let store = xerj_cluster::replica_index_name("orders", shard)?;
+ let Ok(idx) = member.engine.get_index(&store) else {
+ return None;
+ };
+ idx.get_document(id)
+ .await
+ .expect("replica get")
+ .and_then(|doc| {
+ doc.get("title")
+ .and_then(|value| value.as_str())
+ .map(str::to_string)
+ })
+}
+
+async fn wait_replica_title(member: &Member, shard: u32, id: &str, want: Option<&str>) {
+ let deadline = Instant::now() + Duration::from_secs(10);
+ loop {
+ let got = replica_title(member, shard, id).await;
+ if got.as_deref() == want {
+ return;
+ }
+ if Instant::now() > deadline {
+ panic!("replica title of {id} on shard {shard} stayed {got:?}, wanted {want:?}");
+ }
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ }
+}
+
+/// n3 is in the configured membership and is not started until after the
+/// documents exist. Its replica store is created empty by the create entry.
+/// Backfill then copies the primary's live set, including a delete that
+/// landed before n3 started, and publishes n3 into the in-sync set.
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn a_late_member_copies_documents_written_before_it_joined() {
+ let ids = ["n1", "n2", "n3"];
+ let bus = InMemoryBus::new();
+ let mut members = Vec::new();
+ for id in ["n1", "n2"] {
+ let dir = tempfile::tempdir().expect("tempdir");
+ members.push(open_durable_member(id, &ids, &bus, dir, 64).await);
+ }
+ let _leader = wait_leader(&members, &[false, false]).await;
+
+ let (status, body) = put_json(
+ &members[0].app,
+ "/orders",
+ json!({
+ "settings": { "index": { "number_of_replicas": 2 } },
+ "mappings": { "properties": { "title": { "type": "text" } } }
+ }),
+ )
+ .await;
+ assert!(status.is_success(), "create: {status} {body}");
+ for member in &members {
+ wait_status(&member.app, "/orders", StatusCode::OK).await;
+ }
+
+ let mut keep: Option<(String, u32, String, Vec<String>)> = None;
+ let mut gone_id = None;
+ for n in 0..256 {
+ let id = format!("b{n}");
+ let Some((shard, owner)) = members[0].engine.primary_of("orders", &id) else {
+ continue;
+ };
+ if owner == "n3" {
+ continue;
+ }
+ let replicas = members[0].engine.placed_replica_nodes("orders", shard);
+ if !replicas.iter().any(|replica| replica == "n3") {
+ continue;
+ }
+ if keep.is_none() {
+ keep = Some((id, shard, owner, replicas));
+ continue;
+ }
+ if keep
+ .as_ref()
+ .is_some_and(|(_, shard0, _, _)| *shard0 == shard)
+ {
+ gone_id = Some(id);
+ break;
+ }
+ }
+ let (keep_id, shard, owner, replicas) = keep.expect("a shard whose replica list includes n3");
+ let gone_id = gone_id.expect("a second id on that shard");
+ let live: Vec<String> = replicas.into_iter().filter(|id| id != "n3").collect();
+ assert_eq!(live.len(), 1, "one replica is up while n3 is stopped");
+
+ let (status, body) = put_json(
+ &members[0].app,
+ &format!("/orders/_doc/{keep_id}"),
+ json!({ "title": "kept" }),
+ )
+ .await;
+ assert!(status.is_success(), "write kept: {status} {body}");
+ let (status, body) = put_json(
+ &members[0].app,
+ &format!("/orders/_doc/{gone_id}"),
+ json!({ "title": "gone" }),
+ )
+ .await;
+ assert!(status.is_success(), "write gone: {status} {body}");
+ let (status, body) = delete(&members[0].app, &format!("/orders/_doc/{gone_id}")).await;
+ assert!(status.is_success(), "delete gone: {status} {body}");
+ wait_in_sync(&members, shard, &live).await;
+
+ let live_member = members
+ .iter()
+ .find(|member| member.handle.node_id() == live[0])
+ .expect("live replica");
+ assert_eq!(
+ replica_title(live_member, shard, &keep_id).await.as_deref(),
+ Some("kept"),
+ "the replica that acked the write has the document before n3 starts"
+ );
+
+ let dir = tempfile::tempdir().expect("tempdir");
+ members.push(open_durable_member("n3", &ids, &bus, dir, 64).await);
+ let late = members.len() - 1;
+ wait_replica_title(&members[late], shard, &keep_id, Some("kept")).await;
+ let gone = replica_title(&members[late], shard, &gone_id).await;
+ assert!(
+ gone.is_none(),
+ "a document deleted before n3 joined is not copied: {gone:?}"
+ );
+ let on_user = members[late]
+ .engine
+ .get_index("orders")
+ .expect("user index")
+ .get_document(&keep_id)
+ .await
+ .expect("user get");
+ assert!(
+ on_user.is_none(),
+ "the backfill writes the replica store, not the user index: {on_user:?}"
+ );
+ let mut want = live.clone();
+ want.push("n3".to_string());
+ wait_in_sync(&members, shard, &want).await;
+
+ let primary_at = members
+ .iter()
+ .position(|member| member.handle.node_id() == owner)
+ .expect("primary");
+ let live_at = members
+ .iter()
+ .position(|member| member.handle.node_id() == live[0])
+ .expect("live replica");
+ members[primary_at].handle.clear_shard_handler();
+ members[live_at].handle.clear_shard_handler();
+ let (status, body) = get(&members[late].app, &format!("/orders/_doc/{keep_id}")).await;
+ assert!(
+ status.is_success(),
+ "GET uses the copy backfill put on n3: {status} {body}"
+ );
+ assert_eq!(
+ body.pointer("/_source/title")
+ .and_then(|value| value.as_str()),
+ Some("kept"),
+ "{body}"
+ );
+
+ members[primary_at]
+ .handle
+ .set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(
+ members[primary_at].state.clone(),
+ )));
+ members[live_at]
+ .handle
+ .set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(
+ members[live_at].state.clone(),
+ )));
+ let _leader = wait_leader(&members, &[false, false, false]).await;
+ let (status, body) = put_json(
+ &members[late].app,
+ &format!("/orders/_doc/{keep_id}"),
+ json!({ "title": "after" }),
+ )
+ .await;
+ assert!(
+ status.is_success(),
+ "write after n3 is in sync: {status} {body}"
+ );
+ assert_eq!(
+ replica_title(&members[late], shard, &keep_id)
+ .await
+ .as_deref(),
+ Some("after"),
+ "the in-sync replica receives the next write"
+ );
+ assert!(
+ members[primary_at]
+ .engine
+ .replica_nodes("orders", shard)
+ .iter()
+ .any(|id| id == "n3"),
+ "the write did not drop n3 from the in-sync set"
+ );
+
+ for member in &members {
+ let _ = member.shutdown.send(true);
+ }
+}
+
+/// A replica that missed an acknowledged write is out of the in-sync set.
+/// When it answers again it copies the primary's live document and rejoins.
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn a_replica_dropped_from_the_in_sync_set_copies_the_missed_write_and_rejoins() {
+ let members = ring(true).await;
+ let _leader = wait_leader(&members, &[false, false, false]).await;
+
+ let (status, body) = put_json(
+ &members[0].app,
+ "/orders",
+ json!({ "settings": { "index": { "number_of_replicas": 2 } } }),
+ )
+ .await;
+ assert!(status.is_success(), "create: {status} {body}");
+ for member in &members {
+ wait_status(&member.app, "/orders", StatusCode::OK).await;
+ }
+
+ let mut placed = None;
+ for n in 0..64 {
+ let id = format!("f{n}");
+ let Some(primary) = members
+ .iter()
+ .position(|member| member.engine.owns_shard("orders", &id))
+ else {
+ continue;
+ };
+ let (shard, _) = members[primary]
+ .engine
+ .primary_of("orders", &id)
+ .expect("placement");
+ let replicas = members[primary]
+ .engine
+ .placed_replica_nodes("orders", shard);
+ if replicas.len() != 2 {
+ continue;
+ }
+ let at = |node: &str| {
+ members
+ .iter()
+ .position(|member| member.handle.node_id() == node)
+ .expect("replica is a member")
+ };
+ placed = Some((id, primary, at(&replicas[0]), at(&replicas[1]), shard));
+ break;
+ }
+ let (id, primary, replica_a, replica_b, shard) = placed.expect("a shard with two replicas");
+
+ let (status, body) = put_json(
+ &members[primary].app,
+ &format!("/orders/_doc/{id}"),
+ json!({ "title": "first" }),
+ )
+ .await;
+ assert!(status.is_success(), "first write: {status} {body}");
+ assert_eq!(
+ replica_title(&members[replica_a], shard, &id)
+ .await
+ .as_deref(),
+ Some("first")
+ );
+ assert_eq!(
+ replica_title(&members[replica_b], shard, &id)
+ .await
+ .as_deref(),
+ Some("first")
+ );
+
+ members[replica_a].handle.clear_shard_handler();
+ let (status, body) = put_json(
+ &members[primary].app,
+ &format!("/orders/_doc/{id}"),
+ json!({ "title": "second" }),
+ )
+ .await;
+ assert!(
+ status.is_success(),
+ "primary plus the live replica is a quorum: {status} {body}"
+ );
+ assert_eq!(
+ replica_title(&members[replica_a], shard, &id)
+ .await
+ .as_deref(),
+ Some("first"),
+ "the replica that missed the write still has the previous copy"
+ );
+ let live_id = members[replica_b].handle.node_id().to_string();
+ wait_in_sync(&members, shard, std::slice::from_ref(&live_id)).await;
+
+ members[replica_a]
+ .handle
+ .set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(
+ members[replica_a].state.clone(),
+ )));
+ wait_replica_title(&members[replica_a], shard, &id, Some("second")).await;
+ let mut want = vec![live_id, members[replica_a].handle.node_id().to_string()];
+ wait_in_sync(&members, shard, &want).await;
+ want.sort();
+ assert_eq!(
+ {
+ let mut got = members[primary].engine.replica_nodes("orders", shard);
+ got.sort();
+ got
+ },
+ want
+ );
+
+ for member in &members {
+ let _ = member.shutdown.send(true);
+ }
+}
diff --git a/engine/crates/xerj-cluster/src/node.rs b/engine/crates/xerj-cluster/src/node.rs
index 9689e856d..0f6c9624a 100644
--- a/engine/crates/xerj-cluster/src/node.rs
+++ b/engine/crates/xerj-cluster/src/node.rs
@@ -164,6 +164,13 @@ impl ClusterNode {
self.applier = Some(applier);
}
+ /// Applier installed by [`ClusterNode::set_applier`], if any.
+ ///
+ /// The runner uses this to copy replica documents off the Raft step.
+ pub(crate) fn applier(&self) -> Option<Arc<dyn MetadataApplier>> {
+ self.applier.clone()
+ }
+
/// See [`crate::raft::RaftNode::set_snapshot_every`].
pub fn set_snapshot_every(&mut self, every: u64) {
self.raft.set_snapshot_every(every);
diff --git a/engine/crates/xerj-cluster/src/propose.rs b/engine/crates/xerj-cluster/src/propose.rs
index baa93e7db..0c36c2d3e 100644
--- a/engine/crates/xerj-cluster/src/propose.rs
+++ b/engine/crates/xerj-cluster/src/propose.rs
@@ -213,10 +213,18 @@ impl ClusterHandle {
/// Applies a committed metadata command to the local engine.
///
/// Called on the cluster run loop for every committed entry, on every member.
-/// Implementations must be deterministic: the same command against the same
-/// prior state does the same thing on every node. They must not propose a
-/// new command.
+/// Implementations of [`MetadataApplier::apply_metadata`] must be
+/// deterministic: the same command against the same prior state does the
+/// same thing on every node, and that method must not propose a new command.
#[async_trait::async_trait]
pub trait MetadataApplier: Send + Sync {
async fn apply_metadata(&self, command: &ClusterCommand) -> ApplyEffect;
+
+ /// Copy documents this node is missing as a replica.
+ ///
+ /// The runner calls this off the Raft step. The default does nothing.
+ /// The engine implementation asks each primary to copy that shard's live
+ /// documents and to publish this node into the in-sync set. That call
+ /// may propose. It is not part of applying a log entry.
+ async fn backfill_local_replicas(&self) {}
}
diff --git a/engine/crates/xerj-cluster/src/router.rs b/engine/crates/xerj-cluster/src/router.rs
index 292081590..47cb5908b 100644
--- a/engine/crates/xerj-cluster/src/router.rs
+++ b/engine/crates/xerj-cluster/src/router.rs
@@ -190,6 +190,11 @@ impl ShardRouter {
shards
}
+ /// Primary node id for one shard, when the plan names one.
+ pub fn primary_node(&self, index: &str, shard: u32) -> Option<String> {
+ self.routing_table.get(&(index.to_string(), shard)).cloned()
+ }
+
/// Replica node ids a read or a copy uses, excluding the primary.
///
/// This is the in-sync set when one has been published, and otherwise
diff --git a/engine/crates/xerj-cluster/src/runner.rs b/engine/crates/xerj-cluster/src/runner.rs
index b30a93c78..007887bc0 100644
--- a/engine/crates/xerj-cluster/src/runner.rs
+++ b/engine/crates/xerj-cluster/src/runner.rs
@@ -18,6 +18,13 @@ use tracing::{info, warn};
use crate::node::ClusterNode;
+/// How often a member asks primaries to fill replica shards it is missing.
+///
+/// The first pass runs as soon as the runner starts. Later passes wait this
+/// long, or less when shutdown is signalled. A pass that finds nothing out
+/// of sync does not call a peer.
+const REPLICA_BACKFILL_INTERVAL: Duration = Duration::from_millis(200);
+
// ── ClusterRunner ─────────────────────────────────────────────────────────────
/// Drives the Raft state machine in a background async task.
@@ -96,6 +103,7 @@ impl ClusterRunner {
);
let tick = self.tick_interval;
+ let backfill = self.spawn_replica_backfill();
loop {
// Fast path: already asked to shut down before we start a step.
if *self.shutdown.borrow() {
@@ -124,9 +132,38 @@ impl ClusterRunner {
}
}
+ backfill.abort();
info!(node = %self.node.raft.id, "ClusterRunner stopped");
}
+ /// Copy documents onto replica shards this node does not have in sync.
+ ///
+ /// Runs beside the Raft step so a copy does not delay an election. The
+ /// task stops when this runner stops.
+ fn spawn_replica_backfill(&self) -> tokio::task::JoinHandle<()> {
+ let applier = self.node.applier();
+ let mut shutdown = self.shutdown.clone();
+ tokio::spawn(async move {
+ let Some(applier) = applier else {
+ return;
+ };
+ loop {
+ if *shutdown.borrow() {
+ break;
+ }
+ applier.backfill_local_replicas().await;
+ tokio::select! {
+ changed = shutdown.changed() => {
+ if changed.is_err() || *shutdown.borrow() {
+ break;
+ }
+ }
+ _ = tokio::time::sleep(REPLICA_BACKFILL_INTERVAL) => {}
+ }
+ }
+ })
+ }
+
/// Whether the node currently believes itself to be the Raft leader.
pub fn is_leader(&self) -> bool {
self.node.is_leader()
diff --git a/engine/crates/xerj-cluster/src/shard.rs b/engine/crates/xerj-cluster/src/shard.rs
index 9eff4d724..312c94972 100644
--- a/engine/crates/xerj-cluster/src/shard.rs
+++ b/engine/crates/xerj-cluster/src/shard.rs
@@ -234,6 +234,28 @@ pub enum ShardOp {
/// JSON [`crate`] search query params. `from` and `size` match the body.
params_json: String,
},
+ /// Copy this primary's live documents onto `replica_id` and, once that
+ /// copy matches, publish `replica_id` in the in-sync set.
+ ///
+ /// The caller is a replica that is placed for `shard` and is not in the
+ /// in-sync set. The primary holds the shard's replica lock across the
+ /// copy and the publish, so a document write that lands during the copy
+ /// is replicated to the new set when the lock is released. Documents are
+ /// not part of the Raft snapshot; this is the copy that fills them in.
+ BackfillReplica {
+ index: String,
+ shard: u32,
+ replica_id: String,
+ },
+ /// Delete replica-store documents whose ids are not in `ids`.
+ ///
+ /// Sent by the primary at the end of [`ShardOp::BackfillReplica`], after
+ /// the live documents have been upserted. Does not replicate further.
+ RetainReplicaDocs {
+ index: String,
+ shard: u32,
+ ids: Vec<String>,
+ },
}
/// What the owner sends back. `status` is an HTTP status; `body` is JSON.
diff --git a/engine/crates/xerj-engine/src/cluster_apply.rs b/engine/crates/xerj-engine/src/cluster_apply.rs
index ab0c95542..393402fba 100644
--- a/engine/crates/xerj-engine/src/cluster_apply.rs
+++ b/engine/crates/xerj-engine/src/cluster_apply.rs
@@ -38,6 +38,10 @@ impl MetadataApplier for EngineMetadataApplier {
async fn apply_metadata(&self, command: &ClusterCommand) -> ApplyEffect {
apply_committed(&self.engine, command.as_metadata()).await
}
+
+ async fn backfill_local_replicas(&self) {
+ self.engine.backfill_out_of_sync_replicas().await;
+ }
}
pub(crate) fn propose_error(err: ProposeError) -> EngineError {
@@ -151,6 +155,68 @@ impl Engine {
}
}
+ /// Ask each primary to copy live documents onto a replica shard this
+ /// node holds that is not in the in-sync set.
+ ///
+ /// One pass. The cluster runner calls it again. A shard that is already
+ /// in sync, or whose replica store is not open yet, is skipped. The
+ /// primary copies the documents and publishes this node into the in-sync
+ /// set. A pass that finds nothing out of sync does not call a peer.
+ pub async fn backfill_out_of_sync_replicas(&self) {
+ let me = self.routing_node_id();
+ for index in self.index_name_list() {
+ // Replica stores are hidden indices. The plan lives on the user
+ // index, and that is the name the primary copies from.
+ if index.starts_with('.') {
+ continue;
+ }
+ for shard in self.local_replica_shards(&index) {
+ if self.replica_nodes(&index, shard).iter().any(|id| id == &me) {
+ continue;
+ }
+ let Some(store) = xerj_cluster::replica_index_name(&index, shard) else {
+ continue;
+ };
+ if self.get_index(&store).is_err() {
+ continue;
+ }
+ let Some(primary) = self.shard_primary(&index, shard) else {
+ continue;
+ };
+ if primary == me {
+ continue;
+ }
+ let op = xerj_cluster::ShardOp::BackfillReplica {
+ index: index.clone(),
+ shard,
+ replica_id: me.clone(),
+ };
+ match self.forward_shard(&primary, op).await {
+ Ok(result) if (200..300).contains(&result.status) => {}
+ Ok(result) => {
+ tracing::debug!(
+ index = %index,
+ shard,
+ primary = %primary,
+ status = result.status,
+ body = %result.body,
+ "replica backfill was not applied"
+ );
+ }
+ Err(err) => {
+ tracing::debug!(
+ index = %index,
+ shard,
+ primary = %primary,
+ error = %err,
+ "replica backfill call failed"
+ );
+ }
+ }
+ }
+ }
+ }
+
/// Forward one shard operation to `owner`. A failure is not a local write.
pub async fn forward_shard(
&self,
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index e1fd8f53f..a729ce7b1 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -3770,6 +3770,11 @@ impl Engine {
owner.map(|owner| (shard, owner.to_string()))
}
+ /// Primary node id for one shard of `index`, when the plan names one.
+ pub fn shard_primary(&self, index: &str, shard: u32) -> Option<String> {
+ self.shard_router.read().primary_node(index, shard)
+ }
+
/// Replica node ids a read or a copy uses, excluding the primary.
///
/// The create-time list until an in-sync update commits, then the
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 0248ee6d1..f6202b498 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -19758,6 +19758,14 @@ impl Index {
}
}
+ /// Live document ids, sorted by id.
+ ///
+ /// A tombstone is not included. The id is present once the index call
+ /// has returned, including before refresh.
+ pub fn live_document_ids(&self) -> Vec<String> {
+ self.store.version_map.live_ids()
+ }
+
/// Retrieve a document by its string ID.
///
/// Checks the memtable first, then searches on-disk segments.
diff --git a/engine/crates/xerj-storage/src/version_map.rs b/engine/crates/xerj-storage/src/version_map.rs
index 9a166b129..33355d615 100644
--- a/engine/crates/xerj-storage/src/version_map.rs
+++ b/engine/crates/xerj-storage/src/version_map.rs
@@ -664,6 +664,21 @@ impl VersionMap {
self.inner.is_empty()
}
+ /// Live document ids, sorted by id.
+ ///
+ /// Tombstones are left out. The order is the id string, not sequence
+ /// number, so two copies of the same primary agree on the list.
+ pub fn live_ids(&self) -> Vec<String> {
+ let mut ids: Vec<String> = self
+ .inner
+ .iter()
+ .filter(|entry| !entry.value().deleted)
+ .map(|entry| entry.key().clone())
+ .collect();
+ ids.sort();
+ ids
+ }
+
/// Return the count of live (non-deleted) documents.
///
/// O(1) — reads the maintained counter instead of iterating the map
@@ -702,6 +717,16 @@ impl Default for VersionMap {
mod tests {
use super::*;
+ #[test]
+ fn live_ids_skips_tombstones_and_is_sorted() {
+ let vm = VersionMap::new();
+ vm.set("b", 1, "s", false);
+ vm.set("a", 2, "s", false);
+ vm.set("c", 3, "s", false);
+ assert!(vm.delete("b", 4, "s").unwrap());
+ assert_eq!(vm.live_ids(), vec!["a".to_string(), "c".to_string()]);
+ }
+
#[test]
fn basic_set_and_get() {
let vm = VersionMap::new();
diff --git a/landing/docs/clustering.html b/landing/docs/clustering.html
index 2acc29a30..195d3f673 100644
--- a/landing/docs/clustering.html
+++ b/landing/docs/clustering.html
@@ -234,17 +234,17 @@
<h1>Clustering</h1>
- <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log or behind a compacted prefix, installs committed metadata from the on-disk Raft log or from InstallSnapshot. Documents that member never stored are not copied (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>
+ <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log or behind a compacted prefix, installs committed metadata from the on-disk Raft log or from InstallSnapshot. A replica store that is missing the primary's live documents then copies them and rejoins the in-sync set (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>
<h2>What cluster mode does not give you yet</h2>
<div class="enum-list">
- <div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no <code class="inline">wait_for_active_shards</code>. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is <code class="inline">copies / 2 + 1</code>. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it. Nothing replays the writes it missed, so it stays out. If fewer replicas confirm than that quorum, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, and a retry of the index copies it to the replicas still in sync. <code class="inline">number_of_replicas</code> 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.</span></div>
+ <div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no <code class="inline">wait_for_active_shards</code>. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is <code class="inline">copies / 2 + 1</code>. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it until it has copied the primary's live documents and that enlarged set has committed. If fewer replicas confirm than that quorum, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, and a retry of the index copies it to the replicas still in sync. <code class="inline">number_of_replicas</code> 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.</span></div>
<div class="item">Single-document routing only<span class="dsc"><code class="inline">_bulk</code>, <code class="inline">_update</code>, update-by-query, and the native <code class="inline">/v1</code> ingest paths still write on the node that received them. They do not consult the shard plan.</span></div>
<div class="item">Search merges hits, not every clause<span class="dsc">A query, including <code class="inline">match_all</code>, is run on each owner and the hits are merged by the per-shard score (not a global IDF), then <code class="inline">from</code>/<code class="inline">size</code> is applied. Aggregations, suggest, collapse, rescore, profile, <code class="inline">search_after</code>, and scroll are refused with <code class="inline">400</code> when the index has more than one owner — the response is not a partial merge. <code class="inline">/_search</code> with no index, and a comma-separated or wildcard index list, still search only the node that received them. The per-shard window is capped at 10,000 hits.</span></div>
- <div class="item">No document backfill<span class="dsc">A restarted member reopens <code class="inline">{data_dir}/_raft</code> and applies the snapshot plus the log suffix before it serves, so the shard plan and the in-sync set come back and later writes are routed. After 64 applied metadata entries the prefix is snapshotted: <code class="inline">raft.log</code> drops it, and a member whose log does not overlap receives InstallSnapshot and applies those commands. That creates the index metadata and an empty replica store when this node is a replica. The snapshot is the commands, not the documents. A member does not copy documents written before it joined. A replica removed from the in-sync set is not filled in. A metadata history bigger than 8 MiB stays in the log instead of being snapshotted.</span></div>
+ <div class="item">No replica promotion<span class="dsc">A restarted member reopens <code class="inline">{data_dir}/_raft</code> and applies the snapshot plus the log suffix before it serves, so the shard plan and the in-sync set come back. After 64 applied metadata entries the prefix is snapshotted and a member whose log does not overlap receives InstallSnapshot. The snapshot is the metadata commands, not the documents. A placed replica that is missing those documents copies them from the primary and is published back into the in-sync set; a write still needs the primary, and there is no promotion. A metadata history bigger than 8 MiB stays in the log instead of being snapshotted. Changing membership does not move copies.</span></div>
<div class="item">No node-lifecycle APIs<span class="dsc">There is no drain / activate / remove endpoint and no region split-merge. Earlier revisions of this page documented them; the binary has never shipped them.</span></div>
</div>
- <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log. A restart replays the prefix that member had already stored. A new metadata change still needs a leader who can commit.</p>
+ <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set until it has copied the primary's live documents. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log. A restart replays the prefix that member had already stored. A new metadata change still needs a leader who can commit.</p>
<h2>What you get</h2>
<ul>
@@ -321,7 +321,7 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre>
<h2>What replicates, and what stays on one node</h2>
<p><code class="inline">PUT /orders</code> on any member — leader or follower — creates <code class="inline">orders</code> on every member, including its mapping, the settings and aliases in that request, and a primary on each member. <code class="inline">GET /orders</code> and <code class="inline">GET /orders/_mapping</code> on another member succeed after the call returns. <code class="inline">DELETE /orders</code> removes it on every member. A settings or mapping change is one log entry per concrete index. A follower that cannot reach a leader returns <code class="inline">503</code>; it does not create the index locally and report success.</p>
- <p><code class="inline">PUT /orders/_doc/1</code> on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. With <code class="inline">number_of_replicas</code> 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set and a later read does not use it. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p>
+ <p><code class="inline">PUT /orders/_doc/1</code> on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. With <code class="inline">number_of_replicas</code> 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set until it copies the primary's live documents and the enlarged set commits, and a later read does not use it before that. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p>
<p>Several things around that path stay local, on purpose for this step:</p>
<ul>
<li><strong><code class="inline">_bulk</code>, <code class="inline">_update</code>, and the native ingest API</strong> do not route or replicate. A bulk body accepted by one member is indexed there, even when the shard plan says another member owns the id. For a copy of those writes, point the <a href="/docs/config#wal_tap">WAL tap</a> at another XERJ or any ES-compatible target.</li>
@@ -335,11 +335,11 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre>
<div class="item">Leader crash<span class="dsc">Followers time out and elect a new leader — measured ~250 ms on a 3-node loopback ring, including one split-vote retry. The ring stays stable with the dead member absent. Metadata already applied on the survivors stays. The dead member's log is under its data directory; a restart replays it.</span></div>
<div class="item">Hung or dead peer<span class="dsc">One WARN on the down transition, then quiet: retries back off 100 ms → 6.4 s, and messages to live peers are never delayed behind a dead one. A peer that answers again logs "peer reachable again" at INFO.</span></div>
<div class="item">Minority partition<span class="dsc">The minority side cannot reach quorum and its elections never conclude; the majority side keeps its leader. Raft safety holds — at most one leader per term.</span></div>
- <div class="item">Node rejoin<span class="dsc">Restart with the same data directory reloads the snapshot, the log suffix, and the shard plan. A metadata entry committed while this node was down is installed from a peer that still has it, or from that peer's snapshot once the log prefix has been compacted. Documents this node never stored are not copied. A wiped <code class="inline">_raft</code> directory catches metadata up the same way and starts with empty indices (<a href="https://github.com/xerj-org/xerj/issues/1170">#1170</a>).</span></div>
+ <div class="item">Node rejoin<span class="dsc">Restart with the same data directory reloads the snapshot, the log suffix, and the shard plan. A metadata entry committed while this node was down is installed from a peer that still has it, or from that peer's snapshot once the log prefix has been compacted. A replica store that does not have the primary's live documents copies them after that metadata is applied. A wiped data directory catches metadata up the same way and then runs that copy (<a href="https://github.com/xerj-org/xerj/issues/1170">#1170</a>).</span></div>
</div>
<h2>When not to cluster</h2>
- <p>When you need a replica to accept writes after the primary is gone, a replica that missed a write to rejoin the in-sync set, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a single-document write once a quorum of its copies has it, and reloads that metadata when a member restarts. It does not promote a replica, and it does not copy documents onto a node that never stored them. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>
+ <p>When you need a replica to accept writes after the primary is gone, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a single-document write once a quorum of its copies has it, copies the primary's live documents onto a replica that joined late or missed a write, and reloads metadata when a member restarts. It does not promote a replica, and adding or removing a member does not move copies. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>
<p class="source-note">Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs</p>