AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(cluster): acknowledge a document write at a quorum of copies (#2303)
xerj-orga28166e744b4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 61d6f4892..6b6b97b52 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- With `[cluster] enabled = true`, a single-document index or delete is+ acknowledged when a quorum of the shard's copies has confirmed, not only+ when every placed replica has. Copies are the primary plus the replicas+ named by `index.number_of_replicas` at create. A quorum is+ `copies / 2 + 1`, and the primary already counts, so one replica out of+ two is enough. The replica that did not confirm is removed from the+ in-sync set through the Raft log and is not used for a later read. Nothing+ replays the writes it missed, so it stays out. `number_of_replicas` 0 or 1+ is unchanged: there is no replica to wait for, or that one replica must+ confirm. Fewer confirmations than the quorum is still HTTP 503+ `replica_not_acknowledged` — the primary has the write, and retrying the+ index copies it to the replicas still in sync. There is still no+ `wait_for_active_shards`, no replica promotion, and no snapshot catch-up+ for a node that missed the log (part of+ [#1170](https://github.com/xerj-org/xerj/issues/1170)). Single-node mode+ does not take this path.+- A DeepSeek Harness profile bundle in `dsh/` (`xerj-dsh`, Apache-2.0)inserts the README's `mcp-xerj` stdio row via`@deepseek-ai/dsh-mcp-client`. The row is still untested on DSH.diff --git a/engine/crates/xerj-api/src/cluster_shard.rs b/engine/crates/xerj-api/src/cluster_shard.rsindex a6587965e..a892b832d 100644--- a/engine/crates/xerj-api/src/cluster_shard.rs+++ b/engine/crates/xerj-api/src/cluster_shard.rs@@ -3,9 +3,10 @@//! With a cluster handle attached, a replicated index carries a primary plan//! in the same Raft entry as the create. [`dispatch_index`] forwards a//! document to the owner [`Engine::route_write`] names. The primary then-//! copies an acknowledged index or delete onto the replicas named by-//! `index.number_of_replicas` at create time, and the caller is not-//! acknowledged until those copies confirm. [`fanout_search`] asks every+//! copies an index or delete onto the in-sync replicas and acknowledges the+//! caller when a quorum of the copies has confirmed. A replica that does not+//! confirm is dropped from the in-sync set when the rest already make a+//! quorum. [`fanout_search`] asks every//! primary and merges hits. If a primary does not answer, a GET or a search//! of that shard reads a replica copy. Single-node mode (`has_cluster`//! false) returns before any of this.@@ -21,7 +22,7 @@ use axum::http::StatusCode;use axum::response::{IntoResponse, Response};use axum::Json;use serde_json::{json, Value};-use xerj_cluster::{ShardOp, ShardResult};+use xerj_cluster::{ApplyEffect, ClusterCommand, ShardOp, ShardResult};use crate::es_compat::{apply_get_doc_source_filter, search_shard_local, EsSearchBody, EsSearchJson,@@ -870,6 +871,22 @@ fn doc_store(index: &str, replica: bool, shard: u32) -> Result<String, ShardResu})}+fn quorum_nack(index: &str, needed: usize, got: usize, detail: &str) -> ShardResult {+ ShardResult::http(+ 503,+ json!({+ "error": {+ "type": "replica_not_acknowledged",+ "reason": format!(+ "primary applied the write on [{index}] but only {got} of {needed} replica acknowledgements are in sync: {detail}. The primary has the write; retry the index so an in-sync replica can take the copy"+ )+ },+ "status": 503+ })+ .to_string(),+ )+}+fn replica_nack(index: &str, replica: &str, detail: &str) -> ShardResult {ShardResult::http(503,@@ -886,10 +903,20 @@ fn replica_nack(index: &str, replica: &str, detail: &str) -> ShardResult {)}-/// Copy `op` onto every replica. An empty replica list returns immediately.+/// Copy `op` onto the in-sync replicas and wait for a quorum.///-/// The copy is an upsert or a delete without the caller's sequence check:-/// the replica's own sequence numbers are not the primary's.+/// The quorum is a majority of the copies placed at create (the primary,+/// which already has the write, plus [`xerj_cluster::replica_quorum_acks`]+/// replica confirmations). An empty placement returns immediately. The copy+/// is an upsert or a delete without the caller's sequence check: the+/// replica's own sequence numbers are not the primary's.+///+/// Every in-sync replica is asked. When enough confirm and some do not, the+/// ones that confirmed are published through the Raft log before the caller+/// is acknowledged, and later reads skip the rest. A replica that fell out+/// is not added back: nothing in this process replays the writes it missed.+/// Two writes of the same shard do not publish different sets; the primary+/// holds `replica_sync_lock` across the copy and the publish.async fn replicate_write(state: &AppState,index: &str,@@ -899,25 +926,100 @@ async fn replicate_write(let Some((shard, _)) = state.engine.primary_of(index, id) else {return Ok(());};- let replicas = state.engine.replica_nodes(index, shard);- if replicas.is_empty() {+ let sync = state.engine.replica_sync_lock(index, shard);+ let _sync = sync.lock().await;+ let placed = state.engine.placed_replica_nodes(index, shard);+ if placed.is_empty() {return Ok(());}+ let targets = state.engine.replica_nodes(index, shard);+ let needed = xerj_cluster::replica_quorum_acks(placed.len());+ if targets.len() < needed {+ return Err(quorum_nack(+ index,+ needed,+ targets.len(),+ "fewer replicas are in sync than the quorum requires",+ ));+ }let op = replica_copy(op, shard);- for replica in &replicas {- let deadline = Instant::now() + OWNER_READY_WAIT;- loop {- match state.engine.forward_shard(replica, op.clone()).await {- Ok(result) if is_shard_not_ready(&result) && Instant::now() < deadline => {- tokio::time::sleep(Duration::from_millis(20)).await;+ let mut acked: Vec<String> = Vec::new();+ let mut failed: Option<(String, String)> = None;+ for replica in &targets {+ match confirm_replica(state, replica, &op).await {+ Ok(()) => acked.push(replica.clone()),+ Err(detail) => {+ if failed.is_none() {+ failed = Some((replica.clone(), detail));}- Ok(result) if replica_confirmed(&op, &result) => break,- Ok(result) => return Err(replica_nack(index, replica, &result.body)),- Err(err) => return Err(replica_nack(index, replica, &err.to_string())),}}}- Ok(())+ if acked.len() < needed {+ let (replica, detail) = failed.unwrap_or_else(|| {+ (+ "none".to_string(),+ "no in-sync replica acknowledged the write".to_string(),+ )+ });+ return Err(replica_nack(index, &replica, &detail));+ }+ if acked.len() == targets.len() {+ return Ok(());+ }+ acked.sort();+ acked.dedup();+ let Some(handle) = state.engine.cluster_handle() else {+ return Err(quorum_nack(+ index,+ needed,+ acked.len(),+ "no cluster handle to publish the in-sync set",+ ));+ };+ match handle+ .propose(ClusterCommand::SetShardInSync {+ index: index.to_string(),+ shard,+ nodes: acked.clone(),+ })+ .await+ {+ Ok(ApplyEffect::Applied) => {+ let published = state.engine.replica_nodes(index, shard);+ if acked.iter().all(|id| published.iter().any(|got| got == id)) {+ Ok(())+ } else {+ Err(quorum_nack(+ index,+ needed,+ acked.len(),+ "the in-sync set committed without the replicas that confirmed this write",+ ))+ }+ }+ Ok(other) => Err(quorum_nack(+ index,+ needed,+ acked.len(),+ &format!("in-sync update applied as {other:?}"),+ )),+ Err(err) => Err(quorum_nack(index, needed, acked.len(), &err.to_string())),+ }+}++async fn confirm_replica(state: &AppState, replica: &str, op: &ShardOp) -> Result<(), String> {+ let deadline = Instant::now() + OWNER_READY_WAIT;+ loop {+ match state.engine.forward_shard(replica, op.clone()).await {+ Ok(result) if is_shard_not_ready(&result) && Instant::now() < deadline => {+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ Ok(result) if replica_confirmed(op, &result) => return Ok(()),+ Ok(result) => return Err(result.body),+ Err(err) => return Err(err.to_string()),+ }+ }}fn replica_copy(op: ShardOp, shard: u32) -> ShardOp {diff --git a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rsindex d97e814de..bf7033549 100644--- a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs+++ b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs@@ -19,6 +19,7 @@ use xerj_engine::EngineMetadataApplier;struct Member {app: Router,+ state: xerj_api::state::AppState,engine: std::sync::Arc<xerj_engine::Engine>,shutdown: watch::Sender<bool>,handle: ClusterHandle,@@ -65,9 +66,10 @@ async fn ring(start_runners: bool) -> Vec<Member> {let _ = shutdown_rx;Some(node)};- let app = xerj_api::router::build_es_compat_router(state);+ let app = xerj_api::router::build_es_compat_router(state.clone());members.push(Member {app,+ state,engine,shutdown,handle,@@ -514,6 +516,215 @@ async fn an_acknowledged_write_is_on_the_replica_and_readable_after_the_primary_}}+async fn wait_in_sync(members: &[Member], shard: u32, want: &[String]) {+ let deadline = Instant::now() + Duration::from_secs(5);+ loop {+ let ready = members.iter().all(|member| {+ let mut got = member.engine.replica_nodes("orders", shard);+ got.sort();+ got == want+ });+ if ready {+ return;+ }+ if Instant::now() > deadline {+ let observed: Vec<Vec<String>> = members+ .iter()+ .map(|member| member.engine.replica_nodes("orders", shard))+ .collect();+ panic!("in-sync for shard {shard} stayed {observed:?}, wanted {want:?}");+ }+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+}++#[tokio::test(flavor = "multi_thread", worker_threads = 4)]+async fn a_quorum_of_copies_acks_when_one_of_two_replicas_does_not() {+ let members = ring(true).await;+ let dead = [false, false, false];+ let _leader = wait_leader(&members, &dead).await;++ let (status, body) = put_json(+ &members[0].app,+ "/orders",+ json!({ "settings": { "index": { "number_of_replicas": 2 } } }),+ )+ .await;+ assert!(status.is_success(), "create: {status} {body}");+ for member in &members {+ wait_status(&member.app, "/orders", StatusCode::OK).await;+ }++ let mut placed = None;+ for n in 0..64 {+ let id = format!("q{n}");+ let primary = members+ .iter()+ .position(|member| member.engine.owns_shard("orders", &id));+ let Some(primary) = primary else {+ continue;+ };+ let (shard, _) = members[primary]+ .engine+ .primary_of("orders", &id)+ .expect("primary placement");+ let replicas = members[primary]+ .engine+ .placed_replica_nodes("orders", shard);+ if replicas.len() != 2 {+ continue;+ }+ let replica_at = |id: &str| {+ members+ .iter()+ .position(|member| member.handle.node_id() == id)+ .expect("replica is a member")+ };+ let replica_a = replica_at(&replicas[0]);+ let replica_b = replica_at(&replicas[1]);+ assert_ne!(replica_a, primary);+ assert_ne!(replica_b, primary);+ assert_ne!(replica_a, replica_b);+ placed = Some((id, primary, replica_a, replica_b, shard));+ break;+ }+ let (id, primary, replica_a, replica_b, shard) =+ placed.expect("an id whose shard has two replicas");+ let writer = (0..3)+ .find(|n| *n != primary)+ .expect("a non-primary writer");++ members[replica_a].handle.clear_shard_handler();+ members[replica_b].handle.clear_shard_handler();+ let (status, body) = put_json(+ &members[writer].app,+ &format!("/orders/_doc/{id}"),+ json!({ "title": "refused" }),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::SERVICE_UNAVAILABLE,+ "the primary alone is not a quorum of three copies: {status} {body}"+ );+ assert_eq!(+ body.pointer("/error/type").and_then(|v| v.as_str()),+ Some("replica_not_acknowledged"),+ "{body}"+ );++ members[replica_b]+ .handle+ .set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(+ members[replica_b].state.clone(),+ )));++ let (status, body) = put_json(+ &members[writer].app,+ &format!("/orders/_doc/{id}"),+ json!({ "title": "copied" }),+ )+ .await;+ assert!(+ status.is_success(),+ "primary plus one replica is a quorum of three copies: {status} {body}"+ );++ let on_primary = members[primary]+ .engine+ .get_index("orders")+ .expect("primary index")+ .get_document(&id)+ .await+ .expect("primary get");+ assert_eq!(+ on_primary+ .as_ref()+ .and_then(|doc| doc.get("title"))+ .and_then(|v| v.as_str()),+ Some("copied")+ );++ let store = xerj_cluster::replica_index_name("orders", shard).expect("store name");+ let on_live = members[replica_b]+ .engine+ .get_index(&store)+ .expect("live replica store")+ .get_document(&id)+ .await+ .expect("live replica get");+ assert_eq!(+ on_live+ .as_ref()+ .and_then(|doc| doc.get("title"))+ .and_then(|v| v.as_str()),+ Some("copied"),+ "the replica that confirmed has the write: {on_live:?}"+ );+ let on_stale = members[replica_a]+ .engine+ .get_index(&store)+ .expect("stale replica store")+ .get_document(&id)+ .await+ .expect("stale replica get");+ assert!(+ on_stale.is_none(),+ "the replica that did not confirm does not have the document: {on_stale:?}"+ );++ let live_id = members[replica_b].handle.node_id().to_string();+ wait_in_sync(&members, shard, &[live_id]).await;+ assert_eq!(+ members[primary]+ .engine+ .placed_replica_nodes("orders", shard)+ .len(),+ 2,+ "placement stays at the create-time replica count"+ );++ members[primary].handle.clear_shard_handler();+ let (status, body) = get(&members[replica_a].app, &format!("/orders/_doc/{id}")).await;+ assert!(+ status.is_success(),+ "GET uses the in-sync replica, not the stale one: {status} {body}"+ );+ assert_eq!(+ body.pointer("/_source/title").and_then(|v| v.as_str()),+ Some("copied"),+ "a read after the primary stops uses the replica that confirmed: {body}"+ );++ let (status, body) = post_json(+ &members[replica_a].app,+ "/orders/_search",+ json!({ "query": { "match_all": {} }, "size": 10 }),+ )+ .await;+ assert!(status.is_success(), "search: {status} {body}");+ assert_eq!(+ body.pointer("/hits/total/value").and_then(|v| v.as_u64()),+ Some(1),+ "{body}"+ );+ assert_eq!(+ body.pointer("/hits/hits/0/_source/title")+ .and_then(|v| v.as_str()),+ Some("copied"),+ "{body}"+ );+ assert_eq!(+ body.pointer("/hits/hits/0/_index").and_then(|v| v.as_str()),+ Some("orders"),+ "{body}"+ );++ for member in &members {+ let _ = member.shutdown.send(true);+ }+}+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]async fn a_document_is_stored_on_its_shard_owner_and_search_sees_it_from_any_member() {let members = ring(true).await;diff --git a/engine/crates/xerj-cluster/src/lib.rs b/engine/crates/xerj-cluster/src/lib.rsindex 0077d8f1e..b6b241a73 100644--- a/engine/crates/xerj-cluster/src/lib.rs+++ b/engine/crates/xerj-cluster/src/lib.rs@@ -80,8 +80,8 @@ pub use router::ShardRouter;pub use runner::{ClusterRunner, ClusterRunnerBuilder};pub use search::{merge_search_responses, SearchHit, SearchMessage};pub use shard::{- number_of_replicas_from_settings, primary_shard_plan, replica_index_name, shard_plan,- shard_plan_from_schema_json, MemoryShardDir, MemoryShardRpc, NullShardRpc, ShardAssignment,- ShardError, ShardHandler, ShardOp, ShardResult, ShardRpc,+ number_of_replicas_from_settings, primary_shard_plan, replica_index_name, replica_quorum_acks,+ shard_plan, shard_plan_from_schema_json, MemoryShardDir, MemoryShardRpc, NullShardRpc,+ ShardAssignment, ShardError, ShardHandler, ShardOp, ShardResult, ShardRpc,};pub use transport::TcpTransport;diff --git a/engine/crates/xerj-cluster/src/metadata.rs b/engine/crates/xerj-cluster/src/metadata.rsindex a58820dfc..fc10d4722 100644--- a/engine/crates/xerj-cluster/src/metadata.rs+++ b/engine/crates/xerj-cluster/src/metadata.rs@@ -152,6 +152,18 @@ impl ClusterMetadata {self.config.insert(format!("index_settings:{index}"), settings_json.clone());}+ ClusterCommand::SetShardInSync {+ index,+ shard,+ nodes,+ } => {+ let mut nodes = nodes.clone();+ nodes.sort();+ nodes.dedup();+ info!(index = %index, shard, replicas = ?nodes, "Applying SetShardInSync");+ self.config+ .insert(format!("shard_in_sync:{index}:{shard}"), nodes.join(","));+ }ClusterCommand::ClientMetadata { .. } => {warn!("nested ClientMetadata wrapper; not applied");}@@ -197,6 +209,9 @@ impl ClusterMetadata {}// Clean up shard assignments for this indexself.shard_assignments.retain(|(idx, _), _| idx != name);+ let in_sync_prefix = format!("shard_in_sync:{name}:");+ self.config+ .retain(|key, _| !key.starts_with(&in_sync_prefix));info!(index = %name, "Applying DeleteIndex");}diff --git a/engine/crates/xerj-cluster/src/raft.rs b/engine/crates/xerj-cluster/src/raft.rsindex afe5b65d1..4b24c9ced 100644--- a/engine/crates/xerj-cluster/src/raft.rs+++ b/engine/crates/xerj-cluster/src/raft.rs@@ -83,6 +83,17 @@ pub enum ClusterCommand {index: String,settings_json: String,},+ /// Replicas of one shard that a later read or copy may use.+ ///+ /// `nodes` is the subset of the create-time replica list that confirmed+ /// the last acknowledged write. The engine applier drops any id that is+ /// not a placed replica. An empty `nodes` list means no replica is in+ /// sync; the primary is not a member of this list.+ SetShardInSync {+ index: String,+ shard: u32,+ nodes: Vec<String>,+ },/// A client-originated metadata command.////// `request_id` is unique on the node that accepted the caller. That nodediff --git a/engine/crates/xerj-cluster/src/replication.rs b/engine/crates/xerj-cluster/src/replication.rsindex 2438e7c14..5a440a25d 100644--- a/engine/crates/xerj-cluster/src/replication.rs+++ b/engine/crates/xerj-cluster/src/replication.rs@@ -50,6 +50,11 @@ pub enum ReplicationMode {/// Ack after primary + at least `min_replicas` replicas write.Sync { min_replicas: usize },/// Ack after primary + a strict majority of replicas write.+ ///+ /// Not the document path. `PUT /{index}/_doc` uses+ /// [`crate::shard::replica_quorum_acks`]: a majority of the primary plus+ /// the replicas placed at create. This variant's majority is of the+ /// replica list alone, which on two replicas still requires both.Quorum,}diff --git a/engine/crates/xerj-cluster/src/router.rs b/engine/crates/xerj-cluster/src/router.rsindex d36ac48b3..292081590 100644--- a/engine/crates/xerj-cluster/src/router.rs+++ b/engine/crates/xerj-cluster/src/router.rs@@ -55,6 +55,9 @@ pub struct ShardRouter {routing_table: HashMap<(String, u32), String>,/// `(index_name, shard_id)` → list of replica node IDs (excluding primary)replica_table: HashMap<(String, u32), Vec<String>>,+ /// Replicas a read or a later copy may use. Absent means every placed+ /// replica is in sync. Present, including empty, replaces that list.+ in_sync: HashMap<(String, u32), Vec<String>>,}impl ShardRouter {@@ -65,6 +68,7 @@ impl ShardRouter {num_shards,routing_table: HashMap::new(),replica_table: HashMap::new(),+ in_sync: HashMap::new(),}}@@ -85,6 +89,7 @@ impl ShardRouter {pub fn clear_index(&mut self, index: &str) {self.routing_table.retain(|(idx, _), _| idx != index);self.replica_table.retain(|(idx, _), _| idx != index);+ self.in_sync.retain(|(idx, _), _| idx != index);}/// Route a document to its shard and owning node.@@ -126,6 +131,7 @@ impl ShardRouter {pub fn update_from_metadata(&mut self, metadata: &ClusterMetadata) {self.routing_table.clear();self.replica_table.clear();+ self.in_sync.clear();let mut max_shards: u32 = 0;for ((index, shard), node_id) in &metadata.shard_assignments {self.routing_table@@ -184,13 +190,24 @@ impl ShardRouter {shards}- /// Return all replica node IDs for a shard, **excluding the primary owner**.+ /// Replica node ids a read or a copy uses, excluding the primary.///- /// Used by [`WalReplicator`] to determine which nodes need to receive a- /// copy of each WAL entry.- ///- /// [`WalReplicator`]: crate::replication::WalReplicator+ /// This is the in-sync set when one has been published, and otherwise+ /// every replica placed at create.pub fn get_replicas(&self, index: &str, shard: u32) -> Vec<String> {+ let key = (index.to_string(), shard);+ if let Some(nodes) = self.in_sync.get(&key) {+ return nodes.clone();+ }+ self.placed_replicas(index, shard)+ }++ /// Replicas named when the index was created, excluding the primary.+ ///+ /// The quorum is computed from this count. A later in-sync shrink does+ /// not change it: losing replicas does not lower the bar to the primary+ /// alone.+ pub fn placed_replicas(&self, index: &str, shard: u32) -> Vec<String> {let key = (index.to_string(), shard);let primary = self.routing_table.get(&key);@@ -205,6 +222,26 @@ impl ShardRouter {}).unwrap_or_default()}++ /// Publish the replicas that confirmed the last acknowledged write.+ ///+ /// Ids that are not placed replicas of this shard are dropped. A shard+ /// with no placed replicas is left unchanged, so a command that arrives+ /// before the plan does not install an empty overlay.+ pub fn set_in_sync(&mut self, index: &str, shard: u32, nodes: &[String]) {+ let placed = self.placed_replicas(index, shard);+ if placed.is_empty() {+ return;+ }+ let mut kept: Vec<String> = nodes+ .iter()+ .filter(|id| placed.iter().any(|placed_id| placed_id == *id))+ .cloned()+ .collect();+ kept.sort();+ kept.dedup();+ self.in_sync.insert((index.to_string(), shard), kept);+ }}// ── Tests ─────────────────────────────────────────────────────────────────────@@ -399,4 +436,26 @@ mod tests {"both shards should be reachable after refresh (a={saw_node_a}, b={saw_node_b})");}++ #[test]+ fn in_sync_replaces_the_read_list_and_leaves_placement_alone() {+ let mut router = ShardRouter::new(1);+ router.assign("orders", 0, "n1");+ router.add_replica("orders", 0, "n2");+ router.add_replica("orders", 0, "n3");+ assert_eq!(+ router.get_replicas("orders", 0),+ vec!["n2".to_string(), "n3".to_string()]+ );++ router.set_in_sync("orders", 0, &["n3".into(), "n1".into(), "n3".into()]);+ assert_eq!(router.get_replicas("orders", 0), vec!["n3".to_string()]);+ assert_eq!(+ router.placed_replicas("orders", 0),+ vec!["n2".to_string(), "n3".to_string()]+ );++ router.set_in_sync("missing", 0, &["n2".into()]);+ assert!(router.get_replicas("missing", 0).is_empty());+ }}diff --git a/engine/crates/xerj-cluster/src/shard.rs b/engine/crates/xerj-cluster/src/shard.rsindex 743bc1939..7606749cd 100644--- a/engine/crates/xerj-cluster/src/shard.rs+++ b/engine/crates/xerj-cluster/src/shard.rs@@ -112,6 +112,30 @@ pub fn number_of_replicas_from_settings(settings: &serde_json::Value) -> u32 {.unwrap_or(0)}+/// Replica acknowledgements a document write needs before the caller is told it landed.+///+/// `placed` is how many replicas the create entry named, not how many are+/// still in sync. Copies are that count plus the primary. A quorum is+/// `copies / 2 + 1` (a majority of the group). The primary has already+/// applied the write, so it counts as one and the return value is the+/// number of replica confirmations still required.+///+/// | placed replicas | copies | quorum | replica acks |+/// | --- | --- | --- | --- |+/// | 0 | 1 | 1 | 0 |+/// | 1 | 2 | 2 | 1 |+/// | 2 | 3 | 2 | 1 |+/// | 3 | 4 | 3 | 2 |+///+/// This is the majority of the replication group. [`crate::replication::ReplicationMode::Quorum`]+/// is a different helper (a majority of replicas only, not counting the+/// primary) and is not the document path.+pub fn replica_quorum_acks(placed: usize) -> usize {+ let copies = placed.saturating_add(1);+ let quorum = copies / 2 + 1;+ quorum.saturating_sub(1)+}+/// Assignments embedded in a `CreateIndex` schema envelope, if any.////// A schema that is not the replicated envelope — including the `"{}"`@@ -407,6 +431,11 @@ mod tests {let flat = serde_json::json!({"number_of_replicas": 1});assert_eq!(number_of_replicas_from_settings(&flat), 1);assert_eq!(number_of_replicas_from_settings(&serde_json::json!({})), 0);+ assert_eq!(replica_quorum_acks(0), 0);+ assert_eq!(replica_quorum_acks(1), 1);+ assert_eq!(replica_quorum_acks(2), 1);+ assert_eq!(replica_quorum_acks(3), 2);+ assert_eq!(replica_quorum_acks(4), 2);assert_eq!(replica_index_name("orders", 1).as_deref(),Some(".xerj_replica_orders_1")diff --git a/engine/crates/xerj-engine/src/cluster_apply.rs b/engine/crates/xerj-engine/src/cluster_apply.rsindex 3e1eadd55..ab0c95542 100644--- a/engine/crates/xerj-engine/src/cluster_apply.rs+++ b/engine/crates/xerj-engine/src/cluster_apply.rs@@ -366,6 +366,14 @@ async fn apply_committed(engine: &Engine, command: &ClusterCommand) -> ApplyEffeengine.assign_one_shard(index, *shard, node_id);ApplyEffect::Applied}+ ClusterCommand::SetShardInSync {+ index,+ shard,+ nodes,+ } => {+ engine.set_shard_in_sync(index, *shard, nodes);+ ApplyEffect::Applied+ }// Membership and generic config are stored by ClusterMetadata.// They do not change this node's indices._ => ApplyEffect::Applied,diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex 472eedca3..aeff700d8 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -840,6 +840,11 @@ pub struct Engine {/// `ShardRouter::update_from_metadata` whenever the Raft log/// commits a new shard assignment.pub shard_router: Arc<parking_lot::RwLock<xerj_cluster::router::ShardRouter>>,+ /// One lock per `(index, shard)` held across a replica copy and the+ /// in-sync publish that follows it. Document writes for a shard all run+ /// on its primary, so this is what keeps two of those writes from+ /// publishing different in-sync sets.+ replica_sync: Arc<DashMap<(String, u32), Arc<tokio::sync::Mutex<()>>>>,/// Exclusive lock on `<data_dir>/node.lock`, held for the engine's/// whole lifetime (RC4 blocker 13). Acquired in [`Engine::new`] BEFORE@@ -1051,6 +1056,7 @@ impl Engine {shard_router: Arc::new(parking_lot::RwLock::new(xerj_cluster::router::ShardRouter::new(1),)),+ replica_sync: Arc::new(DashMap::new()),_node_lock: node_lock,cluster_state_write: Arc::new(parking_lot::Mutex::new(())),cluster_state_boot: Arc::new(cluster_state_status),@@ -3663,11 +3669,36 @@ impl Engine {owner.map(|owner| (shard, owner.to_string()))}- /// Replica node ids for one shard, excluding the primary.+ /// Replica node ids a read or a copy uses, excluding the primary.+ ///+ /// The create-time list until an in-sync update commits, then the+ /// replicas that confirmed the last acknowledged write.pub fn replica_nodes(&self, index: &str, shard: u32) -> Vec<String> {self.shard_router.read().get_replicas(index, shard)}+ /// Replicas named at create, excluding the primary. The quorum uses this+ /// count, which does not shrink when a replica falls out of sync.+ pub fn placed_replica_nodes(&self, index: &str, shard: u32) -> Vec<String> {+ self.shard_router.read().placed_replicas(index, shard)+ }++ /// Record the replicas that may serve a later read of this shard.+ ///+ /// Ids outside the create-time list are ignored. A shard with no placed+ /// replicas is left as it is.+ pub fn set_shard_in_sync(&self, index: &str, shard: u32, nodes: &[String]) {+ self.shard_router.write().set_in_sync(index, shard, nodes);+ }++ /// Lock that serializes replica copies of one shard on this process.+ pub fn replica_sync_lock(&self, index: &str, shard: u32) -> Arc<tokio::sync::Mutex<()>> {+ self.replica_sync+ .entry((index.to_string(), shard))+ .or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))+ .clone()+ }+/// Primary shards of `index` owned by `node_id`, sorted.pub fn shards_owned_by(&self, index: &str, node_id: &str) -> Vec<u32> {self.shard_router.read().shards_owned_by(index, node_id)diff --git a/landing/docs/clustering.html b/landing/docs/clustering.htmlindex f124059c1..449377d8c 100644--- a/landing/docs/clustering.html+++ b/landing/docs/clustering.html@@ -234,17 +234,17 @@<h1>Clustering</h1>- <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies and the write is acknowledged only after they confirm; the default is 0, which keeps a single copy. A node that missed the log does not catch up (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>+ <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A node that missed the log does not catch up (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p><h2>What cluster mode does not give you yet</h2><div class="enum-list">- <div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no quorum short of every placed replica, and no <code class="inline">wait_for_active_shards</code>. The caller is acknowledged only after each replica has the write. If one does not confirm, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, the replica may not, and a retry of the index is what finishes the copy.</span></div>+ <div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no <code class="inline">wait_for_active_shards</code>. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is <code class="inline">copies / 2 + 1</code>. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it. Nothing replays the writes it missed, so it stays out. If fewer replicas confirm than that quorum, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, and a retry of the index copies it to the replicas still in sync. <code class="inline">number_of_replicas</code> 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.</span></div><div class="item">Single-document routing only<span class="dsc"><code class="inline">_bulk</code>, <code class="inline">_update</code>, update-by-query, and the native <code class="inline">/v1</code> ingest paths still write on the node that received them. They do not consult the shard plan.</span></div><div class="item">Search merges hits, not every clause<span class="dsc">A query, including <code class="inline">match_all</code>, is run on each owner and the hits are merged by the per-shard score (not a global IDF), then <code class="inline">from</code>/<code class="inline">size</code> is applied. Aggregations, suggest, collapse, rescore, profile, <code class="inline">search_after</code>, and scroll are refused with <code class="inline">400</code> when the index has more than one owner — the response is not a partial merge. <code class="inline">/_search</code> with no index, and a comma-separated or wildcard index list, still search only the node that received them. The per-shard window is capped at 10,000 hits.</span></div><div class="item">No snapshot catch-up<span class="dsc">A node that missed a metadata entry does not install it later. The server's Raft log is in memory. Restart keeps indices and documents already stored in that node's data directory, including a replica copy, and does not replay the log, so the shard plan is gone and later writes are not routed. The surviving bytes are not offered as a replica until the plan exists again.</span></div><div class="item">No node-lifecycle APIs<span class="dsc">There is no drain / activate / remove endpoint and no region split-merge. Earlier revisions of this page documented them; the binary has never shipped them.</span></div></div>- <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to the copy when the primary does not answer. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log, and only while a leader can commit.</p>+ <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log, and only while a leader can commit.</p><h2>What you get</h2><ul>@@ -253,7 +253,7 @@<li><strong>Dead-peer isolation</strong> — a hung or dead peer costs one <code class="inline">WARN</code> on the transition and a retry ladder backing off 100 ms → 6.4 s; it never delays messages to live peers.</li><li><strong>Index metadata replication</strong> — <code class="inline">PUT /{index}</code>, <code class="inline">DELETE /{index}</code>, <code class="inline">PUT /{index}/_mapping</code>, and <code class="inline">PUT /{index}/_settings</code> are proposed on the member that received them. A follower forwards to the leader. The HTTP call succeeds only after the entry commits on a majority and this node has applied it. No elected leader, or a commit that does not land in time, is <code class="inline">503</code> and the change is not applied on the caller. With <code class="inline">[cluster] enabled = false</code> these calls stay local, as before.</li><li><strong>Primary shard plan</strong> — the create entry names one primary per member (member ids sorted, shard <code class="inline">i</code> on the <code class="inline">i</code>th) and, when <code class="inline">index.number_of_replicas</code> is set, the next members in that ring as replicas. <code class="inline">Engine::route_write</code> returns the primary, or nothing when this node is the primary or the index has no plan. Document id is the routing key (jump consistent hash).</li>- <li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. The primary copies an acknowledged index or delete to each replica before the caller is answered. A search of one concrete index fans out to every primary and merges hits, and reads a replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li>+ <li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. The primary copies an index or delete to each in-sync replica and answers when a quorum of the copies has confirmed. A search of one concrete index fans out to every primary and merges hits, and reads an in-sync replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li><li><strong>In-memory Raft log in the server</strong> — the on-disk log type exists and is tested, and the server does not open it. Applied indices and the documents a member owned live in that node's data directory, so a restart still has those bytes. The shard plan does not. It lives in the log entry and in each process's routing table, and neither is reloaded. After a restart, <code class="inline">route_write</code> treats the index as unassigned and a new write stays on the node that accepted it. Creating the index again does not rebuild the plan: the index already exists. Deleting it rebuilds a plan and deletes the documents.</li></ul>@@ -321,7 +321,7 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre><h2>What replicates, and what stays on one node</h2><p><code class="inline">PUT /orders</code> on any member — leader or follower — creates <code class="inline">orders</code> on every member, including its mapping, the settings and aliases in that request, and a primary on each member. <code class="inline">GET /orders</code> and <code class="inline">GET /orders/_mapping</code> on another member succeed after the call returns. <code class="inline">DELETE /orders</code> removes it on every member. A settings or mapping change is one log entry per concrete index. A follower that cannot reach a leader returns <code class="inline">503</code>; it does not create the index locally and report success.</p>- <p><code class="inline">PUT /orders/_doc/1</code> on any member stores the document on the primary for that id, and on each replica named when the index was created. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from a replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using a replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p>+ <p><code class="inline">PUT /orders/_doc/1</code> on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. With <code class="inline">number_of_replicas</code> 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set and a later read does not use it. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p><p>Several things around that path stay local, on purpose for this step:</p><ul><li><strong><code class="inline">_bulk</code>, <code class="inline">_update</code>, and the native ingest API</strong> do not route or replicate. A bulk body accepted by one member is indexed there, even when the shard plan says another member owns the id. For a copy of those writes, point the <a href="/docs/config#wal_tap">WAL tap</a> at another XERJ or any ES-compatible target.</li>@@ -339,7 +339,7 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre></div><h2>When not to cluster</h2>- <p>When you need a replica to accept writes after the primary is gone, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, and copies acknowledged single-document writes to the replicas named at create. It does not promote a replica, and it does not catch a rejoining node up. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>+ <p>When you need a replica to accept writes after the primary is gone, a replica that missed a write to rejoin the in-sync set, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, and acknowledges a single-document write once a quorum of its copies has it. It does not promote a replica, and it does not catch a rejoining node or a dropped replica up. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p><p class="source-note">Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs</p>