Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(cluster): acknowledge a document write at a quorum of copies (#2303)

xerj-orga28166e744b4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 61d6f4892..6b6b97b52 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- With `[cluster] enabled = true`, a single-document index or delete is
+ acknowledged when a quorum of the shard's copies has confirmed, not only
+ when every placed replica has. Copies are the primary plus the replicas
+ named by `index.number_of_replicas` at create. A quorum is
+ `copies / 2 + 1`, and the primary already counts, so one replica out of
+ two is enough. The replica that did not confirm is removed from the
+ in-sync set through the Raft log and is not used for a later read. Nothing
+ replays the writes it missed, so it stays out. `number_of_replicas` 0 or 1
+ is unchanged: there is no replica to wait for, or that one replica must
+ confirm. Fewer confirmations than the quorum is still HTTP 503
+ `replica_not_acknowledged` — the primary has the write, and retrying the
+ index copies it to the replicas still in sync. There is still no
+ `wait_for_active_shards`, no replica promotion, and no snapshot catch-up
+ for a node that missed the log (part of
+ [#1170](https://github.com/xerj-org/xerj/issues/1170)). Single-node mode
+ does not take this path.
+
- A DeepSeek Harness profile bundle in `dsh/` (`xerj-dsh`, Apache-2.0)
inserts the README's `mcp-xerj` stdio row via
`@deepseek-ai/dsh-mcp-client`. The row is still untested on DSH.
diff --git a/engine/crates/xerj-api/src/cluster_shard.rs b/engine/crates/xerj-api/src/cluster_shard.rs
index a6587965e..a892b832d 100644
--- a/engine/crates/xerj-api/src/cluster_shard.rs
+++ b/engine/crates/xerj-api/src/cluster_shard.rs
@@ -3,9 +3,10 @@
//! With a cluster handle attached, a replicated index carries a primary plan
//! in the same Raft entry as the create. [`dispatch_index`] forwards a
//! document to the owner [`Engine::route_write`] names. The primary then
-//! copies an acknowledged index or delete onto the replicas named by
-//! `index.number_of_replicas` at create time, and the caller is not
-//! acknowledged until those copies confirm. [`fanout_search`] asks every
+//! copies an index or delete onto the in-sync replicas and acknowledges the
+//! caller when a quorum of the copies has confirmed. A replica that does not
+//! confirm is dropped from the in-sync set when the rest already make a
+//! quorum. [`fanout_search`] asks every
//! primary and merges hits. If a primary does not answer, a GET or a search
//! of that shard reads a replica copy. Single-node mode (`has_cluster`
//! false) returns before any of this.
@@ -21,7 +22,7 @@ use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use axum::Json;
use serde_json::{json, Value};
-use xerj_cluster::{ShardOp, ShardResult};
+use xerj_cluster::{ApplyEffect, ClusterCommand, ShardOp, ShardResult};
use crate::es_compat::{
apply_get_doc_source_filter, search_shard_local, EsSearchBody, EsSearchJson,
@@ -870,6 +871,22 @@ fn doc_store(index: &str, replica: bool, shard: u32) -> Result<String, ShardResu
})
}
+fn quorum_nack(index: &str, needed: usize, got: usize, detail: &str) -> ShardResult {
+ ShardResult::http(
+ 503,
+ json!({
+ "error": {
+ "type": "replica_not_acknowledged",
+ "reason": format!(
+ "primary applied the write on [{index}] but only {got} of {needed} replica acknowledgements are in sync: {detail}. The primary has the write; retry the index so an in-sync replica can take the copy"
+ )
+ },
+ "status": 503
+ })
+ .to_string(),
+ )
+}
+
fn replica_nack(index: &str, replica: &str, detail: &str) -> ShardResult {
ShardResult::http(
503,
@@ -886,10 +903,20 @@ fn replica_nack(index: &str, replica: &str, detail: &str) -> ShardResult {
)
}
-/// Copy `op` onto every replica. An empty replica list returns immediately.
+/// Copy `op` onto the in-sync replicas and wait for a quorum.
///
-/// The copy is an upsert or a delete without the caller's sequence check:
-/// the replica's own sequence numbers are not the primary's.
+/// The quorum is a majority of the copies placed at create (the primary,
+/// which already has the write, plus [`xerj_cluster::replica_quorum_acks`]
+/// replica confirmations). An empty placement returns immediately. The copy
+/// is an upsert or a delete without the caller's sequence check: the
+/// replica's own sequence numbers are not the primary's.
+///
+/// Every in-sync replica is asked. When enough confirm and some do not, the
+/// ones that confirmed are published through the Raft log before the caller
+/// is acknowledged, and later reads skip the rest. A replica that fell out
+/// is not added back: nothing in this process replays the writes it missed.
+/// Two writes of the same shard do not publish different sets; the primary
+/// holds `replica_sync_lock` across the copy and the publish.
async fn replicate_write(
state: &AppState,
index: &str,
@@ -899,25 +926,100 @@ async fn replicate_write(
let Some((shard, _)) = state.engine.primary_of(index, id) else {
return Ok(());
};
- let replicas = state.engine.replica_nodes(index, shard);
- if replicas.is_empty() {
+ let sync = state.engine.replica_sync_lock(index, shard);
+ let _sync = sync.lock().await;
+ let placed = state.engine.placed_replica_nodes(index, shard);
+ if placed.is_empty() {
return Ok(());
}
+ let targets = state.engine.replica_nodes(index, shard);
+ let needed = xerj_cluster::replica_quorum_acks(placed.len());
+ if targets.len() < needed {
+ return Err(quorum_nack(
+ index,
+ needed,
+ targets.len(),
+ "fewer replicas are in sync than the quorum requires",
+ ));
+ }
let op = replica_copy(op, shard);
- for replica in &replicas {
- let deadline = Instant::now() + OWNER_READY_WAIT;
- loop {
- match state.engine.forward_shard(replica, op.clone()).await {
- Ok(result) if is_shard_not_ready(&result) && Instant::now() < deadline => {
- tokio::time::sleep(Duration::from_millis(20)).await;
+ let mut acked: Vec<String> = Vec::new();
+ let mut failed: Option<(String, String)> = None;
+ for replica in &targets {
+ match confirm_replica(state, replica, &op).await {
+ Ok(()) => acked.push(replica.clone()),
+ Err(detail) => {
+ if failed.is_none() {
+ failed = Some((replica.clone(), detail));
}
- Ok(result) if replica_confirmed(&op, &result) => break,
- Ok(result) => return Err(replica_nack(index, replica, &result.body)),
- Err(err) => return Err(replica_nack(index, replica, &err.to_string())),
}
}
}
- Ok(())
+ if acked.len() < needed {
+ let (replica, detail) = failed.unwrap_or_else(|| {
+ (
+ "none".to_string(),
+ "no in-sync replica acknowledged the write".to_string(),
+ )
+ });
+ return Err(replica_nack(index, &replica, &detail));
+ }
+ if acked.len() == targets.len() {
+ return Ok(());
+ }
+ acked.sort();
+ acked.dedup();
+ let Some(handle) = state.engine.cluster_handle() else {
+ return Err(quorum_nack(
+ index,
+ needed,
+ acked.len(),
+ "no cluster handle to publish the in-sync set",
+ ));
+ };
+ match handle
+ .propose(ClusterCommand::SetShardInSync {
+ index: index.to_string(),
+ shard,
+ nodes: acked.clone(),
+ })
+ .await
+ {
+ Ok(ApplyEffect::Applied) => {
+ let published = state.engine.replica_nodes(index, shard);
+ if acked.iter().all(|id| published.iter().any(|got| got == id)) {
+ Ok(())
+ } else {
+ Err(quorum_nack(
+ index,
+ needed,
+ acked.len(),
+ "the in-sync set committed without the replicas that confirmed this write",
+ ))
+ }
+ }
+ Ok(other) => Err(quorum_nack(
+ index,
+ needed,
+ acked.len(),
+ &format!("in-sync update applied as {other:?}"),
+ )),
+ Err(err) => Err(quorum_nack(index, needed, acked.len(), &err.to_string())),
+ }
+}
+
+async fn confirm_replica(state: &AppState, replica: &str, op: &ShardOp) -> Result<(), String> {
+ let deadline = Instant::now() + OWNER_READY_WAIT;
+ loop {
+ match state.engine.forward_shard(replica, op.clone()).await {
+ Ok(result) if is_shard_not_ready(&result) && Instant::now() < deadline => {
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ }
+ Ok(result) if replica_confirmed(op, &result) => return Ok(()),
+ Ok(result) => return Err(result.body),
+ Err(err) => return Err(err.to_string()),
+ }
+ }
}
fn replica_copy(op: ShardOp, shard: u32) -> ShardOp {
diff --git a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
index d97e814de..bf7033549 100644
--- a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
+++ b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
@@ -19,6 +19,7 @@ use xerj_engine::EngineMetadataApplier;
struct Member {
app: Router,
+ state: xerj_api::state::AppState,
engine: std::sync::Arc<xerj_engine::Engine>,
shutdown: watch::Sender<bool>,
handle: ClusterHandle,
@@ -65,9 +66,10 @@ async fn ring(start_runners: bool) -> Vec<Member> {
let _ = shutdown_rx;
Some(node)
};
- let app = xerj_api::router::build_es_compat_router(state);
+ let app = xerj_api::router::build_es_compat_router(state.clone());
members.push(Member {
app,
+ state,
engine,
shutdown,
handle,
@@ -514,6 +516,215 @@ async fn an_acknowledged_write_is_on_the_replica_and_readable_after_the_primary_
}
}
+async fn wait_in_sync(members: &[Member], shard: u32, want: &[String]) {
+ let deadline = Instant::now() + Duration::from_secs(5);
+ loop {
+ let ready = members.iter().all(|member| {
+ let mut got = member.engine.replica_nodes("orders", shard);
+ got.sort();
+ got == want
+ });
+ if ready {
+ return;
+ }
+ if Instant::now() > deadline {
+ let observed: Vec<Vec<String>> = members
+ .iter()
+ .map(|member| member.engine.replica_nodes("orders", shard))
+ .collect();
+ panic!("in-sync for shard {shard} stayed {observed:?}, wanted {want:?}");
+ }
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ }
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn a_quorum_of_copies_acks_when_one_of_two_replicas_does_not() {
+ let members = ring(true).await;
+ let dead = [false, false, false];
+ let _leader = wait_leader(&members, &dead).await;
+
+ let (status, body) = put_json(
+ &members[0].app,
+ "/orders",
+ json!({ "settings": { "index": { "number_of_replicas": 2 } } }),
+ )
+ .await;
+ assert!(status.is_success(), "create: {status} {body}");
+ for member in &members {
+ wait_status(&member.app, "/orders", StatusCode::OK).await;
+ }
+
+ let mut placed = None;
+ for n in 0..64 {
+ let id = format!("q{n}");
+ let primary = members
+ .iter()
+ .position(|member| member.engine.owns_shard("orders", &id));
+ let Some(primary) = primary else {
+ continue;
+ };
+ let (shard, _) = members[primary]
+ .engine
+ .primary_of("orders", &id)
+ .expect("primary placement");
+ let replicas = members[primary]
+ .engine
+ .placed_replica_nodes("orders", shard);
+ if replicas.len() != 2 {
+ continue;
+ }
+ let replica_at = |id: &str| {
+ members
+ .iter()
+ .position(|member| member.handle.node_id() == id)
+ .expect("replica is a member")
+ };
+ let replica_a = replica_at(&replicas[0]);
+ let replica_b = replica_at(&replicas[1]);
+ assert_ne!(replica_a, primary);
+ assert_ne!(replica_b, primary);
+ assert_ne!(replica_a, replica_b);
+ placed = Some((id, primary, replica_a, replica_b, shard));
+ break;
+ }
+ let (id, primary, replica_a, replica_b, shard) =
+ placed.expect("an id whose shard has two replicas");
+ let writer = (0..3)
+ .find(|n| *n != primary)
+ .expect("a non-primary writer");
+
+ members[replica_a].handle.clear_shard_handler();
+ members[replica_b].handle.clear_shard_handler();
+ let (status, body) = put_json(
+ &members[writer].app,
+ &format!("/orders/_doc/{id}"),
+ json!({ "title": "refused" }),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "the primary alone is not a quorum of three copies: {status} {body}"
+ );
+ assert_eq!(
+ body.pointer("/error/type").and_then(|v| v.as_str()),
+ Some("replica_not_acknowledged"),
+ "{body}"
+ );
+
+ members[replica_b]
+ .handle
+ .set_shard_handler(std::sync::Arc::new(ApiShardHandler::new(
+ members[replica_b].state.clone(),
+ )));
+
+ let (status, body) = put_json(
+ &members[writer].app,
+ &format!("/orders/_doc/{id}"),
+ json!({ "title": "copied" }),
+ )
+ .await;
+ assert!(
+ status.is_success(),
+ "primary plus one replica is a quorum of three copies: {status} {body}"
+ );
+
+ let on_primary = members[primary]
+ .engine
+ .get_index("orders")
+ .expect("primary index")
+ .get_document(&id)
+ .await
+ .expect("primary get");
+ assert_eq!(
+ on_primary
+ .as_ref()
+ .and_then(|doc| doc.get("title"))
+ .and_then(|v| v.as_str()),
+ Some("copied")
+ );
+
+ let store = xerj_cluster::replica_index_name("orders", shard).expect("store name");
+ let on_live = members[replica_b]
+ .engine
+ .get_index(&store)
+ .expect("live replica store")
+ .get_document(&id)
+ .await
+ .expect("live replica get");
+ assert_eq!(
+ on_live
+ .as_ref()
+ .and_then(|doc| doc.get("title"))
+ .and_then(|v| v.as_str()),
+ Some("copied"),
+ "the replica that confirmed has the write: {on_live:?}"
+ );
+ let on_stale = members[replica_a]
+ .engine
+ .get_index(&store)
+ .expect("stale replica store")
+ .get_document(&id)
+ .await
+ .expect("stale replica get");
+ assert!(
+ on_stale.is_none(),
+ "the replica that did not confirm does not have the document: {on_stale:?}"
+ );
+
+ let live_id = members[replica_b].handle.node_id().to_string();
+ wait_in_sync(&members, shard, &[live_id]).await;
+ assert_eq!(
+ members[primary]
+ .engine
+ .placed_replica_nodes("orders", shard)
+ .len(),
+ 2,
+ "placement stays at the create-time replica count"
+ );
+
+ members[primary].handle.clear_shard_handler();
+ let (status, body) = get(&members[replica_a].app, &format!("/orders/_doc/{id}")).await;
+ assert!(
+ status.is_success(),
+ "GET uses the in-sync replica, not the stale one: {status} {body}"
+ );
+ assert_eq!(
+ body.pointer("/_source/title").and_then(|v| v.as_str()),
+ Some("copied"),
+ "a read after the primary stops uses the replica that confirmed: {body}"
+ );
+
+ let (status, body) = post_json(
+ &members[replica_a].app,
+ "/orders/_search",
+ json!({ "query": { "match_all": {} }, "size": 10 }),
+ )
+ .await;
+ assert!(status.is_success(), "search: {status} {body}");
+ assert_eq!(
+ body.pointer("/hits/total/value").and_then(|v| v.as_u64()),
+ Some(1),
+ "{body}"
+ );
+ assert_eq!(
+ body.pointer("/hits/hits/0/_source/title")
+ .and_then(|v| v.as_str()),
+ Some("copied"),
+ "{body}"
+ );
+ assert_eq!(
+ body.pointer("/hits/hits/0/_index").and_then(|v| v.as_str()),
+ Some("orders"),
+ "{body}"
+ );
+
+ for member in &members {
+ let _ = member.shutdown.send(true);
+ }
+}
+
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_document_is_stored_on_its_shard_owner_and_search_sees_it_from_any_member() {
let members = ring(true).await;
diff --git a/engine/crates/xerj-cluster/src/lib.rs b/engine/crates/xerj-cluster/src/lib.rs
index 0077d8f1e..b6b241a73 100644
--- a/engine/crates/xerj-cluster/src/lib.rs
+++ b/engine/crates/xerj-cluster/src/lib.rs
@@ -80,8 +80,8 @@ pub use router::ShardRouter;
pub use runner::{ClusterRunner, ClusterRunnerBuilder};
pub use search::{merge_search_responses, SearchHit, SearchMessage};
pub use shard::{
- number_of_replicas_from_settings, primary_shard_plan, replica_index_name, shard_plan,
- shard_plan_from_schema_json, MemoryShardDir, MemoryShardRpc, NullShardRpc, ShardAssignment,
- ShardError, ShardHandler, ShardOp, ShardResult, ShardRpc,
+ number_of_replicas_from_settings, primary_shard_plan, replica_index_name, replica_quorum_acks,
+ shard_plan, shard_plan_from_schema_json, MemoryShardDir, MemoryShardRpc, NullShardRpc,
+ ShardAssignment, ShardError, ShardHandler, ShardOp, ShardResult, ShardRpc,
};
pub use transport::TcpTransport;
diff --git a/engine/crates/xerj-cluster/src/metadata.rs b/engine/crates/xerj-cluster/src/metadata.rs
index a58820dfc..fc10d4722 100644
--- a/engine/crates/xerj-cluster/src/metadata.rs
+++ b/engine/crates/xerj-cluster/src/metadata.rs
@@ -152,6 +152,18 @@ impl ClusterMetadata {
self.config
.insert(format!("index_settings:{index}"), settings_json.clone());
}
+ ClusterCommand::SetShardInSync {
+ index,
+ shard,
+ nodes,
+ } => {
+ let mut nodes = nodes.clone();
+ nodes.sort();
+ nodes.dedup();
+ info!(index = %index, shard, replicas = ?nodes, "Applying SetShardInSync");
+ self.config
+ .insert(format!("shard_in_sync:{index}:{shard}"), nodes.join(","));
+ }
ClusterCommand::ClientMetadata { .. } => {
warn!("nested ClientMetadata wrapper; not applied");
}
@@ -197,6 +209,9 @@ impl ClusterMetadata {
}
// Clean up shard assignments for this index
self.shard_assignments.retain(|(idx, _), _| idx != name);
+ let in_sync_prefix = format!("shard_in_sync:{name}:");
+ self.config
+ .retain(|key, _| !key.starts_with(&in_sync_prefix));
info!(index = %name, "Applying DeleteIndex");
}
diff --git a/engine/crates/xerj-cluster/src/raft.rs b/engine/crates/xerj-cluster/src/raft.rs
index afe5b65d1..4b24c9ced 100644
--- a/engine/crates/xerj-cluster/src/raft.rs
+++ b/engine/crates/xerj-cluster/src/raft.rs
@@ -83,6 +83,17 @@ pub enum ClusterCommand {
index: String,
settings_json: String,
},
+ /// Replicas of one shard that a later read or copy may use.
+ ///
+ /// `nodes` is the subset of the create-time replica list that confirmed
+ /// the last acknowledged write. The engine applier drops any id that is
+ /// not a placed replica. An empty `nodes` list means no replica is in
+ /// sync; the primary is not a member of this list.
+ SetShardInSync {
+ index: String,
+ shard: u32,
+ nodes: Vec<String>,
+ },
/// A client-originated metadata command.
///
/// `request_id` is unique on the node that accepted the caller. That node
diff --git a/engine/crates/xerj-cluster/src/replication.rs b/engine/crates/xerj-cluster/src/replication.rs
index 2438e7c14..5a440a25d 100644
--- a/engine/crates/xerj-cluster/src/replication.rs
+++ b/engine/crates/xerj-cluster/src/replication.rs
@@ -50,6 +50,11 @@ pub enum ReplicationMode {
/// Ack after primary + at least `min_replicas` replicas write.
Sync { min_replicas: usize },
/// Ack after primary + a strict majority of replicas write.
+ ///
+ /// Not the document path. `PUT /{index}/_doc` uses
+ /// [`crate::shard::replica_quorum_acks`]: a majority of the primary plus
+ /// the replicas placed at create. This variant's majority is of the
+ /// replica list alone, which on two replicas still requires both.
Quorum,
}
diff --git a/engine/crates/xerj-cluster/src/router.rs b/engine/crates/xerj-cluster/src/router.rs
index d36ac48b3..292081590 100644
--- a/engine/crates/xerj-cluster/src/router.rs
+++ b/engine/crates/xerj-cluster/src/router.rs
@@ -55,6 +55,9 @@ pub struct ShardRouter {
routing_table: HashMap<(String, u32), String>,
/// `(index_name, shard_id)` → list of replica node IDs (excluding primary)
replica_table: HashMap<(String, u32), Vec<String>>,
+ /// Replicas a read or a later copy may use. Absent means every placed
+ /// replica is in sync. Present, including empty, replaces that list.
+ in_sync: HashMap<(String, u32), Vec<String>>,
}
impl ShardRouter {
@@ -65,6 +68,7 @@ impl ShardRouter {
num_shards,
routing_table: HashMap::new(),
replica_table: HashMap::new(),
+ in_sync: HashMap::new(),
}
}
@@ -85,6 +89,7 @@ impl ShardRouter {
pub fn clear_index(&mut self, index: &str) {
self.routing_table.retain(|(idx, _), _| idx != index);
self.replica_table.retain(|(idx, _), _| idx != index);
+ self.in_sync.retain(|(idx, _), _| idx != index);
}
/// Route a document to its shard and owning node.
@@ -126,6 +131,7 @@ impl ShardRouter {
pub fn update_from_metadata(&mut self, metadata: &ClusterMetadata) {
self.routing_table.clear();
self.replica_table.clear();
+ self.in_sync.clear();
let mut max_shards: u32 = 0;
for ((index, shard), node_id) in &metadata.shard_assignments {
self.routing_table
@@ -184,13 +190,24 @@ impl ShardRouter {
shards
}
- /// Return all replica node IDs for a shard, **excluding the primary owner**.
+ /// Replica node ids a read or a copy uses, excluding the primary.
///
- /// Used by [`WalReplicator`] to determine which nodes need to receive a
- /// copy of each WAL entry.
- ///
- /// [`WalReplicator`]: crate::replication::WalReplicator
+ /// This is the in-sync set when one has been published, and otherwise
+ /// every replica placed at create.
pub fn get_replicas(&self, index: &str, shard: u32) -> Vec<String> {
+ let key = (index.to_string(), shard);
+ if let Some(nodes) = self.in_sync.get(&key) {
+ return nodes.clone();
+ }
+ self.placed_replicas(index, shard)
+ }
+
+ /// Replicas named when the index was created, excluding the primary.
+ ///
+ /// The quorum is computed from this count. A later in-sync shrink does
+ /// not change it: losing replicas does not lower the bar to the primary
+ /// alone.
+ pub fn placed_replicas(&self, index: &str, shard: u32) -> Vec<String> {
let key = (index.to_string(), shard);
let primary = self.routing_table.get(&key);
@@ -205,6 +222,26 @@ impl ShardRouter {
})
.unwrap_or_default()
}
+
+ /// Publish the replicas that confirmed the last acknowledged write.
+ ///
+ /// Ids that are not placed replicas of this shard are dropped. A shard
+ /// with no placed replicas is left unchanged, so a command that arrives
+ /// before the plan does not install an empty overlay.
+ pub fn set_in_sync(&mut self, index: &str, shard: u32, nodes: &[String]) {
+ let placed = self.placed_replicas(index, shard);
+ if placed.is_empty() {
+ return;
+ }
+ let mut kept: Vec<String> = nodes
+ .iter()
+ .filter(|id| placed.iter().any(|placed_id| placed_id == *id))
+ .cloned()
+ .collect();
+ kept.sort();
+ kept.dedup();
+ self.in_sync.insert((index.to_string(), shard), kept);
+ }
}
// ── Tests ─────────────────────────────────────────────────────────────────────
@@ -399,4 +436,26 @@ mod tests {
"both shards should be reachable after refresh (a={saw_node_a}, b={saw_node_b})"
);
}
+
+ #[test]
+ fn in_sync_replaces_the_read_list_and_leaves_placement_alone() {
+ let mut router = ShardRouter::new(1);
+ router.assign("orders", 0, "n1");
+ router.add_replica("orders", 0, "n2");
+ router.add_replica("orders", 0, "n3");
+ assert_eq!(
+ router.get_replicas("orders", 0),
+ vec!["n2".to_string(), "n3".to_string()]
+ );
+
+ router.set_in_sync("orders", 0, &["n3".into(), "n1".into(), "n3".into()]);
+ assert_eq!(router.get_replicas("orders", 0), vec!["n3".to_string()]);
+ assert_eq!(
+ router.placed_replicas("orders", 0),
+ vec!["n2".to_string(), "n3".to_string()]
+ );
+
+ router.set_in_sync("missing", 0, &["n2".into()]);
+ assert!(router.get_replicas("missing", 0).is_empty());
+ }
}
diff --git a/engine/crates/xerj-cluster/src/shard.rs b/engine/crates/xerj-cluster/src/shard.rs
index 743bc1939..7606749cd 100644
--- a/engine/crates/xerj-cluster/src/shard.rs
+++ b/engine/crates/xerj-cluster/src/shard.rs
@@ -112,6 +112,30 @@ pub fn number_of_replicas_from_settings(settings: &serde_json::Value) -> u32 {
.unwrap_or(0)
}
+/// Replica acknowledgements a document write needs before the caller is told it landed.
+///
+/// `placed` is how many replicas the create entry named, not how many are
+/// still in sync. Copies are that count plus the primary. A quorum is
+/// `copies / 2 + 1` (a majority of the group). The primary has already
+/// applied the write, so it counts as one and the return value is the
+/// number of replica confirmations still required.
+///
+/// | placed replicas | copies | quorum | replica acks |
+/// | --- | --- | --- | --- |
+/// | 0 | 1 | 1 | 0 |
+/// | 1 | 2 | 2 | 1 |
+/// | 2 | 3 | 2 | 1 |
+/// | 3 | 4 | 3 | 2 |
+///
+/// This is the majority of the replication group. [`crate::replication::ReplicationMode::Quorum`]
+/// is a different helper (a majority of replicas only, not counting the
+/// primary) and is not the document path.
+pub fn replica_quorum_acks(placed: usize) -> usize {
+ let copies = placed.saturating_add(1);
+ let quorum = copies / 2 + 1;
+ quorum.saturating_sub(1)
+}
+
/// Assignments embedded in a `CreateIndex` schema envelope, if any.
///
/// A schema that is not the replicated envelope — including the `"{}"`
@@ -407,6 +431,11 @@ mod tests {
let flat = serde_json::json!({"number_of_replicas": 1});
assert_eq!(number_of_replicas_from_settings(&flat), 1);
assert_eq!(number_of_replicas_from_settings(&serde_json::json!({})), 0);
+ assert_eq!(replica_quorum_acks(0), 0);
+ assert_eq!(replica_quorum_acks(1), 1);
+ assert_eq!(replica_quorum_acks(2), 1);
+ assert_eq!(replica_quorum_acks(3), 2);
+ assert_eq!(replica_quorum_acks(4), 2);
assert_eq!(
replica_index_name("orders", 1).as_deref(),
Some(".xerj_replica_orders_1")
diff --git a/engine/crates/xerj-engine/src/cluster_apply.rs b/engine/crates/xerj-engine/src/cluster_apply.rs
index 3e1eadd55..ab0c95542 100644
--- a/engine/crates/xerj-engine/src/cluster_apply.rs
+++ b/engine/crates/xerj-engine/src/cluster_apply.rs
@@ -366,6 +366,14 @@ async fn apply_committed(engine: &Engine, command: &ClusterCommand) -> ApplyEffe
engine.assign_one_shard(index, *shard, node_id);
ApplyEffect::Applied
}
+ ClusterCommand::SetShardInSync {
+ index,
+ shard,
+ nodes,
+ } => {
+ engine.set_shard_in_sync(index, *shard, nodes);
+ ApplyEffect::Applied
+ }
// Membership and generic config are stored by ClusterMetadata.
// They do not change this node's indices.
_ => ApplyEffect::Applied,
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index 472eedca3..aeff700d8 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -840,6 +840,11 @@ pub struct Engine {
/// `ShardRouter::update_from_metadata` whenever the Raft log
/// commits a new shard assignment.
pub shard_router: Arc<parking_lot::RwLock<xerj_cluster::router::ShardRouter>>,
+ /// One lock per `(index, shard)` held across a replica copy and the
+ /// in-sync publish that follows it. Document writes for a shard all run
+ /// on its primary, so this is what keeps two of those writes from
+ /// publishing different in-sync sets.
+ replica_sync: Arc<DashMap<(String, u32), Arc<tokio::sync::Mutex<()>>>>,
/// Exclusive lock on `<data_dir>/node.lock`, held for the engine's
/// whole lifetime (RC4 blocker 13). Acquired in [`Engine::new`] BEFORE
@@ -1051,6 +1056,7 @@ impl Engine {
shard_router: Arc::new(parking_lot::RwLock::new(
xerj_cluster::router::ShardRouter::new(1),
)),
+ replica_sync: Arc::new(DashMap::new()),
_node_lock: node_lock,
cluster_state_write: Arc::new(parking_lot::Mutex::new(())),
cluster_state_boot: Arc::new(cluster_state_status),
@@ -3663,11 +3669,36 @@ impl Engine {
owner.map(|owner| (shard, owner.to_string()))
}
- /// Replica node ids for one shard, excluding the primary.
+ /// Replica node ids a read or a copy uses, excluding the primary.
+ ///
+ /// The create-time list until an in-sync update commits, then the
+ /// replicas that confirmed the last acknowledged write.
pub fn replica_nodes(&self, index: &str, shard: u32) -> Vec<String> {
self.shard_router.read().get_replicas(index, shard)
}
+ /// Replicas named at create, excluding the primary. The quorum uses this
+ /// count, which does not shrink when a replica falls out of sync.
+ pub fn placed_replica_nodes(&self, index: &str, shard: u32) -> Vec<String> {
+ self.shard_router.read().placed_replicas(index, shard)
+ }
+
+ /// Record the replicas that may serve a later read of this shard.
+ ///
+ /// Ids outside the create-time list are ignored. A shard with no placed
+ /// replicas is left as it is.
+ pub fn set_shard_in_sync(&self, index: &str, shard: u32, nodes: &[String]) {
+ self.shard_router.write().set_in_sync(index, shard, nodes);
+ }
+
+ /// Lock that serializes replica copies of one shard on this process.
+ pub fn replica_sync_lock(&self, index: &str, shard: u32) -> Arc<tokio::sync::Mutex<()>> {
+ self.replica_sync
+ .entry((index.to_string(), shard))
+ .or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
+ .clone()
+ }
+
/// Primary shards of `index` owned by `node_id`, sorted.
pub fn shards_owned_by(&self, index: &str, node_id: &str) -> Vec<u32> {
self.shard_router.read().shards_owned_by(index, node_id)
diff --git a/landing/docs/clustering.html b/landing/docs/clustering.html
index f124059c1..449377d8c 100644
--- a/landing/docs/clustering.html
+++ b/landing/docs/clustering.html
@@ -234,17 +234,17 @@
<h1>Clustering</h1>
- <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies and the write is acknowledged only after they confirm; the default is 0, which keeps a single copy. A node that missed the log does not catch up (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>
+ <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, or <code class="inline">PUT /{index}/_create</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A node that missed the log does not catch up (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>
<h2>What cluster mode does not give you yet</h2>
<div class="enum-list">
- <div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no quorum short of every placed replica, and no <code class="inline">wait_for_active_shards</code>. The caller is acknowledged only after each replica has the write. If one does not confirm, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, the replica may not, and a retry of the index is what finishes the copy.</span></div>
+ <div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no <code class="inline">wait_for_active_shards</code>. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is <code class="inline">copies / 2 + 1</code>. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it. Nothing replays the writes it missed, so it stays out. If fewer replicas confirm than that quorum, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, and a retry of the index copies it to the replicas still in sync. <code class="inline">number_of_replicas</code> 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.</span></div>
<div class="item">Single-document routing only<span class="dsc"><code class="inline">_bulk</code>, <code class="inline">_update</code>, update-by-query, and the native <code class="inline">/v1</code> ingest paths still write on the node that received them. They do not consult the shard plan.</span></div>
<div class="item">Search merges hits, not every clause<span class="dsc">A query, including <code class="inline">match_all</code>, is run on each owner and the hits are merged by the per-shard score (not a global IDF), then <code class="inline">from</code>/<code class="inline">size</code> is applied. Aggregations, suggest, collapse, rescore, profile, <code class="inline">search_after</code>, and scroll are refused with <code class="inline">400</code> when the index has more than one owner — the response is not a partial merge. <code class="inline">/_search</code> with no index, and a comma-separated or wildcard index list, still search only the node that received them. The per-shard window is capped at 10,000 hits.</span></div>
<div class="item">No snapshot catch-up<span class="dsc">A node that missed a metadata entry does not install it later. The server's Raft log is in memory. Restart keeps indices and documents already stored in that node's data directory, including a replica copy, and does not replay the log, so the shard plan is gone and later writes are not routed. The surviving bytes are not offered as a replica until the plan exists again.</span></div>
<div class="item">No node-lifecycle APIs<span class="dsc">There is no drain / activate / remove endpoint and no region split-merge. Earlier revisions of this page documented them; the binary has never shipped them.</span></div>
</div>
- <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to the copy when the primary does not answer. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log, and only while a leader can commit.</p>
+ <p>With <code class="inline">number_of_replicas</code> at 0, a document has one home and losing that member loses it. With a replica placed at create, a GET or a search of that shard falls back to an in-sync copy when the primary does not answer. A copy that missed an acknowledged write is not in that set. A write still needs the primary: there is no promotion. Index metadata is on every member that applied the log, and only while a leader can commit.</p>
<h2>What you get</h2>
<ul>
@@ -253,7 +253,7 @@
<li><strong>Dead-peer isolation</strong> — a hung or dead peer costs one <code class="inline">WARN</code> on the transition and a retry ladder backing off 100 ms → 6.4 s; it never delays messages to live peers.</li>
<li><strong>Index metadata replication</strong> — <code class="inline">PUT /{index}</code>, <code class="inline">DELETE /{index}</code>, <code class="inline">PUT /{index}/_mapping</code>, and <code class="inline">PUT /{index}/_settings</code> are proposed on the member that received them. A follower forwards to the leader. The HTTP call succeeds only after the entry commits on a majority and this node has applied it. No elected leader, or a commit that does not land in time, is <code class="inline">503</code> and the change is not applied on the caller. With <code class="inline">[cluster] enabled = false</code> these calls stay local, as before.</li>
<li><strong>Primary shard plan</strong> — the create entry names one primary per member (member ids sorted, shard <code class="inline">i</code> on the <code class="inline">i</code>th) and, when <code class="inline">index.number_of_replicas</code> is set, the next members in that ring as replicas. <code class="inline">Engine::route_write</code> returns the primary, or nothing when this node is the primary or the index has no plan. Document id is the routing key (jump consistent hash).</li>
- <li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. The primary copies an acknowledged index or delete to each replica before the caller is answered. A search of one concrete index fans out to every primary and merges hits, and reads a replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li>
+ <li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. The primary copies an index or delete to each in-sync replica and answers when a quorum of the copies has confirmed. A search of one concrete index fans out to every primary and merges hits, and reads an in-sync replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li>
<li><strong>In-memory Raft log in the server</strong> — the on-disk log type exists and is tested, and the server does not open it. Applied indices and the documents a member owned live in that node's data directory, so a restart still has those bytes. The shard plan does not. It lives in the log entry and in each process's routing table, and neither is reloaded. After a restart, <code class="inline">route_write</code> treats the index as unassigned and a new write stays on the node that accepted it. Creating the index again does not rebuild the plan: the index already exists. Deleting it rebuilds a plan and deletes the documents.</li>
</ul>
@@ -321,7 +321,7 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre>
<h2>What replicates, and what stays on one node</h2>
<p><code class="inline">PUT /orders</code> on any member — leader or follower — creates <code class="inline">orders</code> on every member, including its mapping, the settings and aliases in that request, and a primary on each member. <code class="inline">GET /orders</code> and <code class="inline">GET /orders/_mapping</code> on another member succeed after the call returns. <code class="inline">DELETE /orders</code> removes it on every member. A settings or mapping change is one log entry per concrete index. A follower that cannot reach a leader returns <code class="inline">503</code>; it does not create the index locally and report success.</p>
- <p><code class="inline">PUT /orders/_doc/1</code> on any member stores the document on the primary for that id, and on each replica named when the index was created. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from a replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using a replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p>
+ <p><code class="inline">PUT /orders/_doc/1</code> on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. With <code class="inline">number_of_replicas</code> 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set and a later read does not use it. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p>
<p>Several things around that path stay local, on purpose for this step:</p>
<ul>
<li><strong><code class="inline">_bulk</code>, <code class="inline">_update</code>, and the native ingest API</strong> do not route or replicate. A bulk body accepted by one member is indexed there, even when the shard plan says another member owns the id. For a copy of those writes, point the <a href="/docs/config#wal_tap">WAL tap</a> at another XERJ or any ES-compatible target.</li>
@@ -339,7 +339,7 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre>
</div>
<h2>When not to cluster</h2>
- <p>When you need a replica to accept writes after the primary is gone, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, and copies acknowledged single-document writes to the replicas named at create. It does not promote a replica, and it does not catch a rejoining node up. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>
+ <p>When you need a replica to accept writes after the primary is gone, a replica that missed a write to rejoin the in-sync set, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, and acknowledges a single-document write once a quorum of its copies has it. It does not promote a replica, and it does not catch a rejoining node or a dropped replica up. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>
<p class="source-note">Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs</p>