Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(cluster): run _update_by_query on every shard primary (#2347)

xerj-org6ca1a7b8ca0e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 87b3d1f4d..22d0d5ad6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- With `[cluster] enabled = true`, `POST /{index}/_update_by_query` runs on
+ every shard primary, whichever member accepted the request. Each primary
+ updates the documents in its own shard and copies those updates to the
+ in-sync replicas, with the same quorum as `PUT /{index}/_doc/{id}`.
+ `max_docs` is one budget for the index, spent across primaries in
+ node-id order, not once per shard. A primary that does not answer
+ refuses the request before any document is written. A copy that misses
+ quorum is HTTP 503 `replica_not_acknowledged`: the primaries that ran
+ have the update, and the response still reports `updated` and
+ `failures`. `?pipeline=` runs on the primary; a missing or unrunnable
+ pipeline there is HTTP 400 and writes nothing. Pipelines are not
+ replicated. `wait_for_completion=false` uses the same path.
+ `_delete_by_query` and native `/v1` ingest still write on the receiving
+ node. A write still needs the primary. `[cluster] enabled = false` does
+ not take this path (part of
+ [#1170](https://github.com/xerj-org/xerj/issues/1170)).
+
- With `[cluster] enabled = true`, `POST /_bulk`, `POST /{index}/_bulk`, and
`POST /{index}/_update/{id}` run on the shard primary, whichever member
accepted the request. A bulk body is split by primary. An action whose
@@ -22,8 +39,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
id is the id that was routed. An update that does not change the source
is not copied again. A copy that misses quorum is
`503 replica_not_acknowledged` for that item: the primary has the write.
- `_update_by_query` and native `/v1` ingest still write on the receiving
- node. A write still needs the primary. `[cluster] enabled = false` does
+ Native `/v1` ingest still writes on the receiving node.
+ `_update_by_query` is the entry above. A write still needs the primary.
+ `[cluster] enabled = false` does
not take this path (part of
[#1170](https://github.com/xerj-org/xerj/issues/1170)).
diff --git a/engine/crates/xerj-api/src/cluster_shard.rs b/engine/crates/xerj-api/src/cluster_shard.rs
index e87df1b80..ad4923bff 100644
--- a/engine/crates/xerj-api/src/cluster_shard.rs
+++ b/engine/crates/xerj-api/src/cluster_shard.rs
@@ -4,6 +4,8 @@
//! in the same Raft entry as the create. [`dispatch_index`] forwards a
//! document to the owner [`Engine::route_write`] names. [`dispatch_update`]
//! and [`bulk_on_primaries`] send `_update` and `_bulk` to that same owner.
+//! [`update_by_query_on_primaries`] runs `_update_by_query` on every primary
+//! and each primary copies the documents it changed.
//! The primary copies an index, delete, or the document an update stored
//! onto the in-sync replicas and acknowledges the caller when a quorum of
//! the copies has confirmed. A replica that does not
@@ -30,7 +32,8 @@ use serde_json::{json, Value};
use xerj_cluster::{ApplyEffect, ClusterCommand, ShardOp, ShardResult};
use crate::es_compat::{
- apply_get_doc_source_filter, apply_local_update, search_shard_local, EsSearchBody,
+ aggregate_by_query_results, apply_get_doc_source_filter, apply_local_update,
+ pipeline_rejected_body, search_shard_local, unrunnable_pipeline_reason, EsSearchBody,
EsSearchJson, EsSearchQueryParams, EsUpdateBody, GetDocParams, UpdateDocParams,
};
use crate::responses::{EsDeleteDocResponse, EsDocResponse, EsGetResponse};
@@ -669,6 +672,29 @@ impl xerj_cluster::ShardHandler for ApiShardHandler {
)
.await
}
+ ShardOp::UpdateByQuery {
+ index,
+ query_json,
+ script_source,
+ script_params_json,
+ pipeline,
+ max_docs,
+ scroll_size,
+ check_only,
+ } => {
+ handle_update_by_query(
+ &self.state,
+ &index,
+ &query_json,
+ script_source,
+ script_params_json,
+ pipeline,
+ max_docs,
+ scroll_size,
+ check_only,
+ )
+ .await
+ }
ShardOp::Search {
index,
body_json,
@@ -2561,3 +2587,441 @@ pub(crate) async fn bulk_on_primaries(
wire,
)
}
+
+// ── `_update_by_query` on every primary ─────────────────────────────────────
+
+/// One `_update_by_query` after the accepting node has validated the query.
+///
+/// [`update_by_query_on_primaries`] borrows this. `max_docs` is the caller's
+/// cap for the whole index; the fan-out spends it across primaries.
+#[derive(Clone)]
+pub(crate) struct UpdateByQueryParts {
+ pub query: Value,
+ pub script: Option<(String, Value)>,
+ pub pipeline: Option<String>,
+ pub max_docs: Option<u64>,
+ pub scroll_size: usize,
+}
+
+/// Box the per-index fan-out so it is not inlined into `update_by_query`.
+///
+/// See `begin_cluster_update_by_query` for why the future has to live on
+/// the heap: a debug worker stack overflows when the handler and the runner
+/// are both built on it.
+pub(crate) fn pin_update_by_query_on_primaries(
+ state: AppState,
+ index: String,
+ parts: UpdateByQueryParts,
+) -> std::pin::Pin<Box<dyn std::future::Future<Output = Value> + Send>> {
+ Box::pin(async move { update_by_query_on_primaries(&state, &index, &parts).await })
+}
+
+fn update_by_query_op(
+ index: &str,
+ parts: &UpdateByQueryParts,
+ max_docs: Option<u64>,
+ check_only: bool,
+) -> ShardOp {
+ ShardOp::UpdateByQuery {
+ index: index.to_string(),
+ query_json: parts.query.to_string(),
+ script_source: parts.script.as_ref().map(|(source, _)| source.clone()),
+ script_params_json: parts.script.as_ref().map(|(_, params)| params.to_string()),
+ pipeline: parts.pipeline.clone(),
+ max_docs,
+ scroll_size: parts.scroll_size as u64,
+ check_only,
+ }
+}
+
+fn update_by_query_unavailable(index: &str, detail: &str) -> Value {
+ json!({
+ "error": {
+ "type": "shard_not_ready",
+ "reason": format!("update_by_query on [{index}] failed: {detail}")
+ },
+ "status": 503
+ })
+}
+
+fn empty_update_by_query() -> Value {
+ json!({
+ "took": 0,
+ "timed_out": false,
+ "total": 0,
+ "updated": 0,
+ "deleted": 0,
+ "batches": 0,
+ "version_conflicts": 0,
+ "noops": 0,
+ "failures": [],
+ "throttled_millis": 0,
+ "requests_per_second": -1,
+ "throttled_until_millis": 0,
+ })
+}
+
+fn value_from_shard(result: &ShardResult) -> Value {
+ serde_json::from_str(&result.body).unwrap_or_else(|_| {
+ json!({
+ "error": {
+ "type": "shard_not_ready",
+ "reason": result.body,
+ },
+ "status": result.status
+ })
+ })
+}
+
+fn is_not_ready_value(value: &Value) -> bool {
+ value.get("status").and_then(Value::as_u64) == Some(503)
+ && value.pointer("/error/type").and_then(Value::as_str) == Some("shard_not_ready")
+}
+
+fn ensure_error_status(mut value: Value) -> Value {
+ if value.get("status").is_some() {
+ return value;
+ }
+ if let Some(obj) = value.as_object_mut() {
+ obj.insert("status".to_string(), json!(503));
+ }
+ value
+}
+
+fn has_replica_nack(body: &Value) -> bool {
+ body.get("failures")
+ .and_then(Value::as_array)
+ .is_some_and(|items| {
+ items.iter().any(|item| {
+ item.pointer("/cause/type").and_then(Value::as_str)
+ == Some("replica_not_acknowledged")
+ })
+ })
+}
+
+fn stamp_replica_nack(mut body: Value) -> Value {
+ if !has_replica_nack(&body) {
+ return body;
+ }
+ if let Some(obj) = body.as_object_mut() {
+ obj.insert("status".to_string(), json!(503));
+ obj.insert(
+ "error".to_string(),
+ json!({
+ "type": "replica_not_acknowledged",
+ "reason": "primary applied the update but a replica copy missed quorum. The primary has the update; retry the update so an in-sync replica can take the copy"
+ }),
+ );
+ }
+ body
+}
+
+/// Index open, this node is a primary (when a plan exists), pipeline runnable.
+///
+/// `Err` is a body the HTTP layer can return. Nothing has been written.
+fn prepare_update_by_query(
+ state: &AppState,
+ index: &str,
+ pipeline: Option<&str>,
+) -> Result<(), Value> {
+ if state.engine.get_index(index).is_err() {
+ return Err(serde_json::from_str(&shard_not_ready_body(index))
+ .unwrap_or_else(|_| update_by_query_unavailable(index, "index is not open")));
+ }
+ let me = state.engine.routing_node_id();
+ let targets = state.engine.shard_search_targets(index);
+ if !targets.is_empty() && !targets.iter().any(|id| id == &me) {
+ let owner = targets.first().map(String::as_str).unwrap_or("unknown");
+ let body = not_this_primary(index, owner);
+ return Err(value_from_shard(&body));
+ }
+ if let Some(name) = pipeline {
+ if let Some(reason) = unrunnable_pipeline_reason(&state.engine, name) {
+ return Err(pipeline_rejected_body(&reason));
+ }
+ }
+ Ok(())
+}
+
+async fn prepare_update_by_query_wait(
+ state: &AppState,
+ index: &str,
+ pipeline: Option<&str>,
+) -> Result<(), Value> {
+ let deadline = Instant::now() + OWNER_READY_WAIT;
+ loop {
+ match prepare_update_by_query(state, index, pipeline) {
+ Err(body) if is_not_ready_value(&body) && Instant::now() < deadline => {
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ }
+ other => return other,
+ }
+ }
+}
+
+async fn forward_update_by_query(state: &AppState, index: &str, owner: &str, op: ShardOp) -> Value {
+ let deadline = Instant::now() + OWNER_READY_WAIT;
+ loop {
+ match state.engine.forward_shard(owner, op.clone()).await {
+ Ok(result) if is_shard_not_ready(&result) && Instant::now() < deadline => {
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ }
+ Ok(result) => return value_from_shard(&result),
+ Err(err) => return update_by_query_unavailable(index, &err.to_string()),
+ }
+ }
+}
+
+async fn copy_updated_document(state: &AppState, index: &str, id: &str) -> Result<(), String> {
+ if state.engine.primary_of(index, id).is_none() {
+ return Ok(());
+ }
+ if !state.engine.owns_shard(index, id) {
+ return Err(format!(
+ "primary applied the update on [{index}] but document [{id}] is not owned by this node"
+ ));
+ }
+ let op = match state.engine.get_index(index) {
+ Ok(idx) => match idx.get_document(id).await {
+ Ok(Some(doc)) => match serde_json::to_string(&doc) {
+ Ok(source_json) => ShardOp::IndexDoc {
+ index: index.to_string(),
+ id: id.to_string(),
+ source_json,
+ op: "index".to_string(),
+ if_seq_no: None,
+ if_primary_term: None,
+ version: None,
+ version_type: None,
+ replica: false,
+ shard: 0,
+ },
+ Err(err) => {
+ return Err(format!(
+ "primary applied the update on [{index}] but document [{id}] is not JSON: {err}"
+ ));
+ }
+ },
+ Ok(None) => ShardOp::DeleteDoc {
+ index: index.to_string(),
+ id: id.to_string(),
+ if_seq_no: None,
+ if_primary_term: None,
+ replica: false,
+ shard: 0,
+ },
+ Err(err) => {
+ return Err(format!(
+ "primary applied the update on [{index}] but document [{id}] could not be read: {err}"
+ ));
+ }
+ },
+ Err(err) => {
+ return Err(format!(
+ "primary applied the update but [{index}] is not open: {err}"
+ ));
+ }
+ };
+ replicate_write(state, index, id, op)
+ .await
+ .map_err(|nack| nack_reason(&nack))
+}
+
+async fn apply_update_by_query_here(
+ state: &AppState,
+ index: &str,
+ parts: &UpdateByQueryParts,
+ max_docs: Option<u64>,
+) -> Value {
+ let idx = match state.engine.get_index(index) {
+ Ok(idx) => idx,
+ Err(err) => return update_by_query_unavailable(index, &err.to_string()),
+ };
+ let recorded = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
+ let pipeline = parts
+ .pipeline
+ .as_ref()
+ .map(|name| (std::sync::Arc::clone(&state.engine), name.clone()));
+ let mut body = crate::es_compat::run_update_by_query_recorded_boxed(
+ idx,
+ parts.query.clone(),
+ max_docs,
+ parts.scroll_size,
+ parts.script.clone(),
+ pipeline,
+ Some(std::sync::Arc::clone(&recorded)),
+ )
+ .await;
+ let ids = recorded
+ .lock()
+ .unwrap_or_else(|err| err.into_inner())
+ .clone();
+ // A failed run may already have updated some documents. Copy those before
+ // returning the error so a replica does not keep the pre-update source.
+ let mut nacks = Vec::new();
+ for id in ids {
+ if let Err(reason) = copy_updated_document(state, index, &id).await {
+ nacks.push((id, reason));
+ }
+ }
+ if body.get("status").is_some() {
+ return body;
+ }
+ if nacks.is_empty() {
+ return body;
+ }
+ let Some(obj) = body.as_object_mut() else {
+ return body;
+ };
+ let failures = obj
+ .entry("failures".to_string())
+ .or_insert_with(|| json!([]));
+ if let Some(failures) = failures.as_array_mut() {
+ for (id, reason) in nacks {
+ failures.push(json!({
+ "id": id,
+ "cause": {
+ "type": "replica_not_acknowledged",
+ "reason": reason,
+ }
+ }));
+ }
+ }
+ body
+}
+
+#[allow(clippy::too_many_arguments)]
+async fn handle_update_by_query(
+ state: &AppState,
+ index: &str,
+ query_json: &str,
+ script_source: Option<String>,
+ script_params_json: Option<String>,
+ pipeline: Option<String>,
+ max_docs: Option<u64>,
+ scroll_size: u64,
+ check_only: bool,
+) -> ShardResult {
+ if let Err(body) = prepare_update_by_query_wait(state, index, pipeline.as_deref()).await {
+ let status = body
+ .get("status")
+ .and_then(Value::as_u64)
+ .and_then(|status| u16::try_from(status).ok())
+ .unwrap_or(503);
+ return ShardResult::http(status, body.to_string());
+ }
+ if check_only {
+ return ShardResult::http(200, json!({ "ok": true }).to_string());
+ }
+ let query = match serde_json::from_str::<Value>(query_json) {
+ Ok(query) => query,
+ Err(err) => {
+ return ShardResult::http(
+ 400,
+ json!({
+ "error": {
+ "type": "parsing_exception",
+ "reason": format!("update_by_query query is not JSON: {err}")
+ },
+ "status": 400
+ })
+ .to_string(),
+ );
+ }
+ };
+ let script = script_source.map(|source| {
+ let params = script_params_json
+ .as_deref()
+ .and_then(|raw| serde_json::from_str(raw).ok())
+ .unwrap_or_else(|| json!({}));
+ (source, params)
+ });
+ let parts = UpdateByQueryParts {
+ query,
+ script,
+ pipeline,
+ max_docs,
+ scroll_size: scroll_size.max(1) as usize,
+ };
+ let value = apply_update_by_query_here(state, index, &parts, max_docs).await;
+ let status = value
+ .get("status")
+ .and_then(Value::as_u64)
+ .and_then(|status| u16::try_from(status).ok())
+ .unwrap_or(200);
+ ShardResult::http(status, value.to_string())
+}
+
+/// Run `_update_by_query` on every primary of `index` and copy the updates.
+///
+/// Primaries are visited in node-id order so `max_docs` is one budget for the
+/// index. A primary that fails the preflight (missing pipeline, or not
+/// answering) stops the call before any document is written. A later primary
+/// that fails after an earlier one has already applied is returned as that
+/// error; the earlier updates stay. `[cluster] enabled = false` does not call
+/// this.
+pub(crate) async fn update_by_query_on_primaries(
+ state: &AppState,
+ index: &str,
+ parts: &UpdateByQueryParts,
+) -> Value {
+ let targets = state.engine.shard_search_targets(index);
+ if targets.is_empty() {
+ // No plan: this node holds the whole index. There is no primary to
+ // call, so the run stays here. `replicate_write` does nothing when
+ // `primary_of` is `None`.
+ return apply_update_by_query_here(state, index, parts, parts.max_docs).await;
+ }
+ // Every primary, including this node, is a shard call. The runner builds
+ // `search_inner` on the caller's stack, and that overflows a debug worker
+ // (2 MiB) once the fan-out is above it. `MemoryShardRpc` already runs a
+ // handler on an 8 MiB thread — the same thread a remote primary uses — so
+ // this node's own shard takes that path too. A primary that is not
+ // answering fails here, before any document is written.
+ for target in &targets {
+ let value = forward_update_by_query(
+ state,
+ index,
+ target,
+ update_by_query_op(index, parts, parts.max_docs, true),
+ )
+ .await;
+ if value.get("ok").and_then(Value::as_bool) != Some(true) {
+ return ensure_error_status(value);
+ }
+ }
+ if parts.max_docs == Some(0) {
+ return empty_update_by_query();
+ }
+
+ let mut remaining = parts.max_docs;
+ let mut parts_out = Vec::with_capacity(targets.len());
+ for target in &targets {
+ if remaining == Some(0) {
+ break;
+ }
+ let cap = remaining;
+ let value = forward_update_by_query(
+ state,
+ index,
+ target,
+ update_by_query_op(index, parts, cap, false),
+ )
+ .await;
+ // A shard that failed its own run (flush, script limit, or a primary
+ // that stopped answering) has no `updated` count. Replica nacks keep
+ // `updated` and sit in `failures` until the aggregate below.
+ if value.get("updated").is_none() {
+ return ensure_error_status(value);
+ }
+ if let Some(left) = remaining.as_mut() {
+ let used = value.get("total").and_then(Value::as_u64).unwrap_or(0);
+ *left = left.saturating_sub(used);
+ }
+ parts_out.push(value);
+ }
+ if parts_out.is_empty() {
+ return empty_update_by_query();
+ }
+ stamp_replica_nack(aggregate_by_query_results(parts_out, "updated"))
+}
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index aba56139c..ac3bfe4db 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -31459,7 +31459,10 @@ async fn resolve_by_query_targets(
state: &AppState,
spec: &str,
user_query: Value,
-) -> std::result::Result<(Vec<std::sync::Arc<xerj_engine::Index>>, Value), xerj_common::XerjError> {
+) -> std::result::Result<
+ (Vec<std::sync::Arc<xerj_engine::Index>>, Vec<String>, Value),
+ xerj_common::XerjError,
+> {
let member_names = resolve_index_selector(state, spec).await;
// Alias filters are keyed on the name AS WRITTEN (the alias, not its
@@ -31491,7 +31494,7 @@ async fn resolve_by_query_targets(
.map_err(xerj_common::XerjError::from)?,
);
}
- Ok((indices, effective_query))
+ Ok((indices, member_names, effective_query))
}
/// Combine the per-member results of a fanned-out by-query into one response.
@@ -31502,7 +31505,7 @@ async fn resolve_by_query_targets(
/// concatenated; if any member produced an error body (it carries a `status`),
/// that error is surfaced rather than a clean aggregate — a destructive
/// by-query that partially failed must not report success.
-fn aggregate_by_query_results(mut results: Vec<Value>, count_key: &str) -> Value {
+pub(crate) fn aggregate_by_query_results(mut results: Vec<Value>, count_key: &str) -> Value {
if results.len() == 1 {
return results.pop().unwrap();
}
@@ -31562,7 +31565,7 @@ pub async fn delete_by_query(
) -> impl IntoResponse {
// Fan the selector out to every member of a multi-index alias (#450), with
// any alias filter already ANDed into the query.
- let (indices, effective_query) =
+ let (indices, _index_names, effective_query) =
match resolve_by_query_targets(&state, &index, body.query.clone()).await {
Ok(t) => t,
Err(e) => return ApiError::new(e).into_response(),
@@ -31973,6 +31976,137 @@ pub struct UpdateByQueryBody {
pub scroll_size: Option<u64>,
}
+/// Why `pipeline` cannot run on this engine, in the same words the
+/// `_update_by_query?pipeline=` handler has always used.
+pub(crate) fn unrunnable_pipeline_reason(
+ engine: &xerj_engine::Engine,
+ pipeline: &str,
+) -> Option<String> {
+ if !engine.pipelines.contains_key(pipeline) {
+ Some(format!("pipeline with id [{pipeline}] does not exist"))
+ } else {
+ engine.unrunnable_pipelines.get(pipeline).map(|reason| {
+ format!(
+ "pipeline with id [{pipeline}] cannot be run: {}",
+ reason.value()
+ )
+ })
+ }
+}
+
+pub(crate) fn pipeline_rejected_body(reason: &str) -> Value {
+ json!({
+ "error": {
+ "root_cause": [{ "type": "illegal_argument_exception", "reason": reason }],
+ "type": "illegal_argument_exception",
+ "reason": reason,
+ },
+ "status": 400,
+ })
+}
+
+/// Build the cluster fan-out off this handler's state machine.
+///
+/// `update_by_query` is already large enough that a debug build's worker
+/// stack (2 MiB, the `#[tokio::test]` default) overflows once a second copy
+/// of the runner is inlined into the same future. `[cluster] enabled = false`
+/// never polls the fan-out, so the future is allocated here and only the
+/// pointer is stored on the handler. The server runtime is 4 MiB; this keeps
+/// the single-node path the size it was before the fan-out existed.
+fn begin_cluster_update_by_query(
+ state: AppState,
+ index_names: Vec<String>,
+ parts: crate::cluster_shard::UpdateByQueryParts,
+ wait_for_completion: bool,
+ handle: crate::state::TaskHandle,
+) -> std::pin::Pin<Box<dyn std::future::Future<Output = Response> + Send>> {
+ Box::pin(cluster_update_by_query(
+ state,
+ index_names,
+ parts,
+ wait_for_completion,
+ handle,
+ ))
+}
+
+async fn cluster_update_by_query(
+ state: AppState,
+ index_names: Vec<String>,
+ parts: crate::cluster_shard::UpdateByQueryParts,
+ wait_for_completion: bool,
+ handle: crate::state::TaskHandle,
+) -> Response {
+ if !wait_for_completion {
+ let task_key = handle.key().to_string();
+ let spawned_key = task_key.clone();
+ let tasks = state.tasks.clone();
+ let names = index_names.clone();
+ // The fan-out calls `Engine::get_index` on the detached task, so
+ // the request's visibility rule has to come with it. See
+ // `xerj_engine::index_guard`.
+ let rule = xerj_engine::index_guard::current();
+ tokio::spawn(xerj_engine::index_guard::scoped_opt(rule, async move {
+ let mut results = Vec::with_capacity(names.len());
+ for name in &names {
+ results.push(
+ crate::cluster_shard::pin_update_by_query_on_primaries(
+ state.clone(),
+ name.clone(),
+ parts.clone(),
+ )
+ .await,
+ );
+ }
+ tasks.complete(&spawned_key, aggregate_by_query_results(results, "updated"));
+ drop(handle);
+ }));
+ return Json(json!({ "task": task_key })).into_response();
+ }
+ let mut results = Vec::with_capacity(index_names.len());
+ for name in &index_names {
+ results.push(
+ crate::cluster_shard::pin_update_by_query_on_primaries(
+ state.clone(),
+ name.clone(),
+ parts.clone(),
+ )
+ .await,
+ );
+ }
+ drop(handle);
+ by_query_response(aggregate_by_query_results(results, "updated"))
+}
+
+/// Own the runner's arguments, then box it.
+///
+/// A primary's `_update_by_query` runs on the coordinator's task when this
+/// node is that primary. Inlining the runner into the fan-out future puts
+/// two copies of it on the stack under `search_inner` and overflows a debug
+/// worker. The box is the same split `MemoryShardRpc` uses for a remote
+/// primary: the heavy future is not built on the caller's state machine.
+pub(crate) fn run_update_by_query_recorded_boxed(
+ idx: std::sync::Arc<xerj_engine::Index>,
+ query: Value,
+ max_docs: Option<u64>,
+ scroll_size: usize,
+ script: Option<(String, Value)>,
+ pipeline: Option<(std::sync::Arc<xerj_engine::Engine>, String)>,
+ updated_ids: Option<std::sync::Arc<std::sync::Mutex<Vec<String>>>>,
+) -> std::pin::Pin<Box<dyn std::future::Future<Output = Value> + Send>> {
+ Box::pin(async move {
+ run_update_by_query_recorded(
+ &idx,
+ &query,
+ max_docs,
+ scroll_size,
+ script,
+ pipeline,
+ updated_ids,
+ )
+ .await
+ })
+}
+
pub async fn update_by_query(
State(state): State<AppState>,
Path(index): Path<String>,
@@ -31982,11 +32116,11 @@ pub async fn update_by_query(
let query_val = body.query.clone().unwrap_or(json!({ "match_all": {} }));
// Fan the selector out to every member of a multi-index alias (#450), with
// any alias filter already ANDed into the query.
- let (indices, effective_query) = match resolve_by_query_targets(&state, &index, query_val).await
- {
- Ok(t) => t,
- Err(e) => return ApiError::new(e).into_response(),
- };
+ let (indices, index_names, effective_query) =
+ match resolve_by_query_targets(&state, &index, query_val).await {
+ Ok(t) => t,
+ Err(e) => return ApiError::new(e).into_response(),
+ };
// Validate the (alias-filter-ANDed) query once, up front, so a malformed
// body is an immediate 400 before a task is registered or any member is
@@ -32017,34 +32151,27 @@ pub async fn update_by_query(
// rewritten every document EXACTLY as it was. Resolved and validated
// before any document is touched: an unknown or unrunnable name fails the
// whole request rather than half of it.
- let pipeline = match params.get("pipeline").map(String::as_str) {
+ //
+ // With a cluster handle the pipeline runs on each primary, which is where
+ // the current source is. The check is that primary's, before it writes.
+ // Single-node mode still refuses here, on the node that accepted the call.
+ let pipeline_name = match params.get("pipeline").map(String::as_str) {
None | Some("_none") => None,
- Some(p) => {
- let reason = if !state.engine.pipelines.contains_key(p) {
- Some(format!("pipeline with id [{p}] does not exist"))
- } else {
- state
- .engine
- .unrunnable_pipelines
- .get(p)
- .map(|r| format!("pipeline with id [{p}] cannot be run: {}", r.value()))
- };
- if let Some(reason) = reason {
- return (
- StatusCode::BAD_REQUEST,
- Json(json!({
- "error": {
- "root_cause": [{ "type": "illegal_argument_exception", "reason": reason }],
- "type": "illegal_argument_exception",
- "reason": reason,
- },
- "status": 400,
- })),
- )
- .into_response();
- }
- Some((state.engine.clone(), p.to_string()))
+ Some(p) => Some(p.to_string()),
+ };
+ let pipeline = if state.engine.has_cluster() {
+ None
+ } else if let Some(p) = pipeline_name.as_deref() {
+ if let Some(reason) = unrunnable_pipeline_reason(&state.engine, p) {
+ return (
+ StatusCode::BAD_REQUEST,
+ Json(pipeline_rejected_body(&reason)),
+ )
+ .into_response();
}
+ Some((state.engine.clone(), p.to_string()))
+ } else {
+ None
};
// ES defaults `wait_for_completion` to true (synchronous); only an
@@ -32055,6 +32182,24 @@ pub async fn update_by_query(
.unwrap_or(true);
let handle = state.tasks.register("indices:data/write/update/byquery");
+ if state.engine.has_cluster() {
+ let parts = crate::cluster_shard::UpdateByQueryParts {
+ query: effective_query,
+ script,
+ pipeline: pipeline_name,
+ max_docs: body.max_docs,
+ scroll_size,
+ };
+ return begin_cluster_update_by_query(
+ state,
+ index_names,
+ parts,
+ wait_for_completion,
+ handle,
+ )
+ .await;
+ }
+
if !wait_for_completion {
let task_key = handle.key().to_string();
let spawned_key = task_key.clone();
@@ -32183,6 +32328,15 @@ fn apply_pipeline_or_fail(
/// no-script write is blind — XERJ cannot produce a version conflict);
/// `noops` stays 0 (the mini-painless tolerates `ctx.*` statements without
/// skipping the write).
+fn record_updated(updated_ids: &Option<std::sync::Arc<std::sync::Mutex<Vec<String>>>>, id: &str) {
+ let Some(ids) = updated_ids else {
+ return;
+ };
+ ids.lock()
+ .unwrap_or_else(|err| err.into_inner())
+ .push(id.to_string());
+}
+
async fn run_update_by_query(
idx: &std::sync::Arc<xerj_engine::Index>,
query: &Value,
@@ -32190,6 +32344,18 @@ async fn run_update_by_query(
scroll_size: usize,
script: Option<(String, Value)>,
pipeline: Option<(std::sync::Arc<xerj_engine::Engine>, String)>,
+) -> Value {
+ run_update_by_query_recorded(idx, query, max_docs, scroll_size, script, pipeline, None).await
+}
+
+pub(crate) async fn run_update_by_query_recorded(
+ idx: &std::sync::Arc<xerj_engine::Index>,
+ query: &Value,
+ max_docs: Option<u64>,
+ scroll_size: usize,
+ script: Option<(String, Value)>,
+ pipeline: Option<(std::sync::Arc<xerj_engine::Engine>, String)>,
+ updated_ids: Option<std::sync::Arc<std::sync::Mutex<Vec<String>>>>,
) -> Value {
let started = Instant::now();
@@ -32300,7 +32466,10 @@ async fn run_update_by_query(
processed += 1;
if let Some(f) = transform_one(idx, id, &script, &pipeline).await {
match f {
- Ok(()) => updated += 1,
+ Ok(()) => {
+ updated += 1;
+ record_updated(&updated_ids, id);
+ }
Err(reason) => failures.push(json!({
"id": id,
"cause": { "reason": reason },
@@ -32432,6 +32601,7 @@ async fn run_update_by_query(
Ok(()) => {
updated += 1;
mutated_in_batch += 1;
+ record_updated(&updated_ids, &hit.id);
}
Err(reason) => failures.push(json!({
"id": hit.id,
@@ -32466,6 +32636,7 @@ async fn run_update_by_query(
Ok(_) => {
updated += 1;
mutated_in_batch += 1;
+ record_updated(&updated_ids, &hit.id);
}
Err(e) => {
failures.push(json!({
diff --git a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
index 4a344e10d..c74588a77 100644
--- a/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
+++ b/engine/crates/xerj-api/tests/cluster_metadata_replicates.rs
@@ -404,6 +404,46 @@ async fn cluster_disabled_create_stays_on_the_node_that_received_it() {
StatusCode::NOT_FOUND,
"single-node bulk and update do not open a replica store"
);
+
+ let (status, body) = post_json(
+ &a,
+ "/local/_update_by_query",
+ json!({
+ "query": { "ids": { "values": ["b1"] } },
+ "script": { "source": "ctx._source.title = 'ubq'" }
+ }),
+ )
+ .await;
+ assert!(
+ status.is_success(),
+ "single-node update_by_query: {status} {body}"
+ );
+ assert_eq!(
+ body.get("updated").and_then(|v| v.as_u64()),
+ Some(1),
+ "{body}"
+ );
+ let (status, got) = get(&a, "/local/_doc/b1").await;
+ assert_eq!(status, StatusCode::OK, "{got}");
+ assert_eq!(
+ got.pointer("/_source/title").and_then(|v| v.as_str()),
+ Some("ubq")
+ );
+ let (status, got) = get(&a, "/local/_doc/1").await;
+ assert_eq!(status, StatusCode::OK, "{got}");
+ assert_eq!(
+ got.pointer("/_source/title").and_then(|v| v.as_str()),
+ Some("here"),
+ "an ids query on one node still leaves the other document: {got}"
+ );
+ let (status, _) = get(&b, "/local/_doc/b1").await;
+ assert_eq!(status, StatusCode::NOT_FOUND);
+ let (status, _) = get(&a, "/.xerj_replica_local_0/_doc/b1").await;
+ assert_eq!(
+ status,
+ StatusCode::NOT_FOUND,
+ "single-node update_by_query does not open a replica store"
+ );
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
@@ -2296,3 +2336,366 @@ async fn bulk_and_update_from_a_non_owner_write_the_primary_and_the_replica() {
let _ = member.shutdown.send(true);
}
}
+
+/// `_update_by_query` accepted by a node that owns neither document updates
+/// every primary and copies the new source onto the in-sync replica.
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn update_by_query_from_a_non_owner_updates_every_primary_and_replica() {
+ let members = ring(true).await;
+ let _leader = wait_leader(&members, &[false, false, false]).await;
+
+ let (status, body) = put_json(
+ &members[0].app,
+ "/orders",
+ json!({
+ "settings": { "index": { "number_of_replicas": 1 } },
+ "mappings": { "properties": { "title": { "type": "text" } } }
+ }),
+ )
+ .await;
+ assert!(status.is_success(), "create: {status} {body}");
+ for member in &members {
+ wait_status(&member.app, "/orders", StatusCode::OK).await;
+ }
+
+ let mut placed: Vec<(String, usize, usize, u32)> = Vec::new();
+ for n in 0..128 {
+ let id = format!("u{n}");
+ let Some(primary) = members
+ .iter()
+ .position(|member| member.engine.owns_shard("orders", &id))
+ else {
+ continue;
+ };
+ let (shard, _) = members[primary]
+ .engine
+ .primary_of("orders", &id)
+ .expect("placement");
+ let replicas = members[primary].engine.replica_nodes("orders", shard);
+ if replicas.len() != 1 {
+ continue;
+ }
+ let Some(replica) = members
+ .iter()
+ .position(|member| member.handle.node_id() == replicas[0])
+ else {
+ continue;
+ };
+ if placed.iter().any(|(_, owner, _, _)| *owner == primary) {
+ continue;
+ }
+ placed.push((id, primary, replica, shard));
+ if placed.len() == 2 {
+ break;
+ }
+ }
+ assert_eq!(placed.len(), 2, "two primaries: {placed:?}");
+ let (id_a, primary_a, replica_a, shard_a) = placed[0].clone();
+ let (id_b, primary_b, replica_b, shard_b) = placed[1].clone();
+ assert_ne!(primary_a, primary_b);
+ assert_ne!(replica_a, replica_b);
+ let writer = (0..3)
+ .find(|n| *n != primary_a && *n != primary_b)
+ .expect("a node that owns neither id");
+
+ let (status, body) = put_json(
+ &members[writer].app,
+ &format!("/orders/_doc/{id_a}"),
+ json!({ "title": "alpha" }),
+ )
+ .await;
+ assert!(status.is_success(), "index {id_a}: {status} {body}");
+ let (status, body) = put_json(
+ &members[writer].app,
+ &format!("/orders/_doc/{id_b}"),
+ json!({ "title": "beta" }),
+ )
+ .await;
+ assert!(status.is_success(), "index {id_b}: {status} {body}");
+ assert_eq!(
+ user_source(&members[primary_a], &id_a)
+ .await
+ .expect("primary a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("alpha")
+ );
+ assert!(
+ user_source(&members[writer], &id_a).await.is_none(),
+ "the receiver stored {id_a}"
+ );
+
+ let (status, body) = post_json(
+ &members[writer].app,
+ "/orders/_update_by_query",
+ json!({
+ "query": { "match_all": {} },
+ "script": { "source": "ctx._source.title = 'all'" }
+ }),
+ )
+ .await;
+ assert!(
+ status.is_success(),
+ "update_by_query from non-owner: {status} {body}"
+ );
+ assert_eq!(
+ body.get("updated").and_then(|v| v.as_u64()),
+ Some(2),
+ "{body}"
+ );
+ assert_eq!(
+ body.get("failures")
+ .and_then(|v| v.as_array())
+ .map(|items| items.len()),
+ Some(0),
+ "{body}"
+ );
+ assert_eq!(
+ user_source(&members[primary_a], &id_a)
+ .await
+ .expect("primary a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("all")
+ );
+ assert_eq!(
+ user_source(&members[primary_b], &id_b)
+ .await
+ .expect("primary b")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("all")
+ );
+ assert_eq!(
+ replica_source(&members[replica_a], shard_a, &id_a)
+ .await
+ .expect("replica a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("all")
+ );
+ assert_eq!(
+ replica_source(&members[replica_b], shard_b, &id_b)
+ .await
+ .expect("replica b")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("all")
+ );
+ assert!(user_source(&members[writer], &id_a).await.is_none());
+ assert!(user_source(&members[writer], &id_b).await.is_none());
+ let (status, got) = get(&members[writer].app, &format!("/orders/_doc/{id_a}")).await;
+ assert!(
+ status.is_success(),
+ "GET after update_by_query: {status} {got}"
+ );
+ assert_eq!(
+ got.pointer("/_source/title").and_then(|v| v.as_str()),
+ Some("all"),
+ "{got}"
+ );
+
+ let (status, body) = post_json(
+ &members[writer].app,
+ "/orders/_update_by_query",
+ json!({
+ "query": { "ids": { "values": [id_a] } },
+ "script": { "source": "ctx._source.title = 'only-a'" }
+ }),
+ )
+ .await;
+ assert!(status.is_success(), "ids update_by_query: {status} {body}");
+ assert_eq!(
+ body.get("updated").and_then(|v| v.as_u64()),
+ Some(1),
+ "{body}"
+ );
+ assert_eq!(
+ user_source(&members[primary_a], &id_a)
+ .await
+ .expect("primary a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("only-a")
+ );
+ assert_eq!(
+ replica_source(&members[replica_a], shard_a, &id_a)
+ .await
+ .expect("replica a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("only-a")
+ );
+ assert_eq!(
+ user_source(&members[primary_b], &id_b)
+ .await
+ .expect("primary b")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("all"),
+ "the other shard stays as it was"
+ );
+
+ let (status, body) = post_json(
+ &members[writer].app,
+ "/orders/_update_by_query",
+ json!({
+ "query": { "match_all": {} },
+ "max_docs": 1,
+ "script": { "source": "ctx._source.title = 'capped'" }
+ }),
+ )
+ .await;
+ assert!(
+ status.is_success(),
+ "max_docs update_by_query: {status} {body}"
+ );
+ assert_eq!(
+ body.get("updated").and_then(|v| v.as_u64()),
+ Some(1),
+ "max_docs is one budget for the index, not one per shard: {body}"
+ );
+ let title_a = user_source(&members[primary_a], &id_a)
+ .await
+ .expect("primary a")
+ .get("title")
+ .and_then(|v| v.as_str())
+ .unwrap_or("")
+ .to_string();
+ let title_b = user_source(&members[primary_b], &id_b)
+ .await
+ .expect("primary b")
+ .get("title")
+ .and_then(|v| v.as_str())
+ .unwrap_or("")
+ .to_string();
+ let capped = [&title_a, &title_b]
+ .iter()
+ .filter(|title| title.as_str() == "capped")
+ .count();
+ assert_eq!(capped, 1, "titles {title_a} {title_b}");
+ assert_eq!(
+ replica_source(&members[replica_a], shard_a, &id_a)
+ .await
+ .expect("replica a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some(title_a.as_str())
+ );
+ assert_eq!(
+ replica_source(&members[replica_b], shard_b, &id_b)
+ .await
+ .expect("replica b")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some(title_b.as_str())
+ );
+
+ let (status, body) = post_json(
+ &members[writer].app,
+ "/orders/_update_by_query?wait_for_completion=false",
+ json!({
+ "query": { "match_all": {} },
+ "script": { "source": "ctx._source.title = 'async'" }
+ }),
+ )
+ .await;
+ assert!(
+ status.is_success(),
+ "async update_by_query: {status} {body}"
+ );
+ let task = body
+ .get("task")
+ .and_then(|v| v.as_str())
+ .unwrap_or_else(|| panic!("task id: {body}"))
+ .to_string();
+ let deadline = Instant::now() + Duration::from_secs(5);
+ let completed = loop {
+ let (status, got) = get(&members[writer].app, &format!("/_tasks/{task}")).await;
+ if status.is_success() && got.get("completed").and_then(|v| v.as_bool()) == Some(true) {
+ break got;
+ }
+ if Instant::now() > deadline {
+ panic!("task {task} did not finish: {status} {got}");
+ }
+ tokio::time::sleep(Duration::from_millis(20)).await;
+ };
+ assert_eq!(
+ completed
+ .pointer("/response/updated")
+ .and_then(|v| v.as_u64()),
+ Some(2),
+ "{completed}"
+ );
+ assert_eq!(
+ user_source(&members[primary_a], &id_a)
+ .await
+ .expect("primary a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("async")
+ );
+ assert_eq!(
+ replica_source(&members[replica_b], shard_b, &id_b)
+ .await
+ .expect("replica b")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("async")
+ );
+
+ members[primary_a].handle.clear_shard_handler();
+ let (status, body) = post_json(
+ &members[writer].app,
+ "/orders/_update_by_query",
+ json!({
+ "query": { "match_all": {} },
+ "script": { "source": "ctx._source.title = 'down'" }
+ }),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::SERVICE_UNAVAILABLE,
+ "a primary that does not answer refuses before writing: {body}"
+ );
+ assert_eq!(
+ user_source(&members[primary_a], &id_a)
+ .await
+ .expect("primary a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("async"),
+ "the down primary was not written"
+ );
+ assert_eq!(
+ replica_source(&members[replica_a], shard_a, &id_a)
+ .await
+ .expect("replica a")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("async"),
+ "the refused update did not change the copy"
+ );
+ assert_eq!(
+ user_source(&members[primary_b], &id_b)
+ .await
+ .expect("primary b")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("async"),
+ "a later primary is not written once an earlier one cannot be checked"
+ );
+ assert_eq!(
+ replica_source(&members[replica_b], shard_b, &id_b)
+ .await
+ .expect("replica b")
+ .get("title")
+ .and_then(|v| v.as_str()),
+ Some("async")
+ );
+
+ for member in &members {
+ let _ = member.shutdown.send(true);
+ }
+}
diff --git a/engine/crates/xerj-cluster/src/shard.rs b/engine/crates/xerj-cluster/src/shard.rs
index a630aa2ff..b8651cb59 100644
--- a/engine/crates/xerj-cluster/src/shard.rs
+++ b/engine/crates/xerj-cluster/src/shard.rs
@@ -252,6 +252,32 @@ pub enum ShardOp {
/// JSON value of the URL-level `_source` filter, when one was set.
default_source_req_json: Option<String>,
},
+ /// Run `_update_by_query` on this primary's shard only.
+ ///
+ /// The coordinating node sends one of these to every primary. This node
+ /// updates documents in its user index and copies each updated document
+ /// onto the in-sync replicas. `check_only` confirms the index is open and
+ /// the pipeline can run, and writes nothing.
+ UpdateByQuery {
+ index: String,
+ /// JSON query object. Already combined with any alias filter.
+ query_json: String,
+ /// Painless source. Absent when the request has no script.
+ #[serde(default)]
+ script_source: Option<String>,
+ /// JSON script params. Absent when there is no script.
+ #[serde(default)]
+ script_params_json: Option<String>,
+ /// Ingest pipeline name. The primary runs it; pipelines are not
+ /// replicated with the index.
+ #[serde(default)]
+ pipeline: Option<String>,
+ #[serde(default)]
+ max_docs: Option<u64>,
+ scroll_size: u64,
+ #[serde(default)]
+ check_only: bool,
+ },
/// Search this member's local shard only. The caller merges hits.
Search {
index: String,
diff --git a/engine/crates/xerj-engine/src/index_guard.rs b/engine/crates/xerj-engine/src/index_guard.rs
index 53c3b7888..ae424c964 100644
--- a/engine/crates/xerj-engine/src/index_guard.rs
+++ b/engine/crates/xerj-engine/src/index_guard.rs
@@ -65,7 +65,8 @@
//! |---|---|
//! | `xerj-api::es_compat::spawn_datafeed_task` | **carries the rule** — the one detached path that resolves an index name |
//! | `es_compat::search_impl`'s search task | owns one already-authorized `Index`; `terms`/`lookup` targets are resolved earlier, on the request's task |
-//! | `es_compat::delete_by_query` / `update_by_query` (`wait_for_completion=false`) | same: one already-authorized `Index` handle, no `Engine` |
+//! | `es_compat::delete_by_query` (`wait_for_completion=false`) | same: one already-authorized `Index` handle, no `Engine` |
+//! | `es_compat::update_by_query` (`wait_for_completion=false`) | cluster path **carries the rule** — it calls `Engine` to reach shard primaries. The single-node path owns already-authorized `Index` handles and does not |
//! | `Engine::spawn_pit_sweeper`, `spawn_search_context_sweeper` | started by `Engine::new`, capture only the PIT/scroll maps, touch no index |
//! | `Index`'s flush/merge/warm tasks (`index.rs`, `write_publication.rs`) | methods **on** an already-resolved `Index`; the name has already been through the funnel |
//! | `xerj-server::main`'s listeners, metrics loop, autoindex/brain CLIs | startup, outside any request; unrestricted by design (see `visible`) |
diff --git a/landing/docs/clustering.html b/landing/docs/clustering.html
index 839fd908e..c3103a581 100644
--- a/landing/docs/clustering.html
+++ b/landing/docs/clustering.html
@@ -236,12 +236,12 @@
<h1>Clustering</h1>
- <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create</code>, <code class="inline">POST /_bulk</code>, <code class="inline">POST /{index}/_bulk</code>, or <code class="inline">POST /{index}/_update/{id}</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log or behind a compacted prefix, installs committed metadata from the on-disk Raft log or from InstallSnapshot. A replica store that is missing the primary's live documents then copies them and rejoins the in-sync set (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>
+ <p>XERJ ships an <strong>embedded Raft implementation</strong> — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. <strong>Where the ring stands today, plainly:</strong> leader election, heartbeats, and the authenticated inter-node transport are real. <strong>Index metadata</strong> (create, delete, mapping, and settings) and the <strong>primary shard plan</strong> for that index are proposed through the Raft log and applied on every member. A document written with <code class="inline">PUT</code>, <code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create</code>, <code class="inline">POST /_bulk</code>, <code class="inline">POST /{index}/_bulk</code>, or <code class="inline">POST /{index}/_update/{id}</code> is stored on the member that owns its shard, whichever node accepted the call. <code class="inline">POST /{index}/_update_by_query</code> runs on every primary of that index and each primary copies the documents it updated. <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> follow the same owner. <code class="inline">GET</code> or <code class="inline">POST /{index}/_search</code> fans out to every owner and merges hits. <code class="inline">index.number_of_replicas</code> set at create places that many copies. The write is acknowledged when a quorum of those copies, counting the primary, has confirmed; the default is 0, which keeps a single copy. A restarted member, and a member that starts with an empty log or behind a compacted prefix, installs committed metadata from the on-disk Raft log or from InstallSnapshot. A replica store that is missing the primary's live documents then copies them and rejoins the in-sync set (public tracking: <a href="https://github.com/xerj-org/xerj/issues/1170">xerj-org/xerj#1170</a>).</p>
<h2>What cluster mode does not give you yet</h2>
<div class="enum-list">
<div class="item">Replicas follow the create setting, not a later change<span class="dsc"><code class="inline">index.number_of_replicas</code> on <code class="inline">PUT /{index}</code> places that many copies of each primary, capped at one fewer than the member count. The default is 0. A later <code class="inline">PUT /{index}/_settings</code> stores a new number and does not move copies. There is no <code class="inline">wait_for_active_shards</code>. A write is acknowledged when a quorum of the copies has confirmed: copies are the primary plus the replicas placed at create, and a quorum is <code class="inline">copies / 2 + 1</code>. The primary already counts, so with two replicas one confirmation is enough and the caller still succeeds. The replica that did not confirm is removed from the in-sync set by a Raft entry, and later reads do not use it until it has copied the primary's live documents and that enlarged set has committed. If fewer replicas confirm than that quorum, the response is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the write, and a retry of the index copies it to the replicas still in sync. <code class="inline">number_of_replicas</code> 0 or 1 does not loosen this: there is no replica to wait for, or that single replica must confirm.</span></div>
- <div class="item">Update-by-query and native ingest stay on the receiver<span class="dsc"><code class="inline">_update_by_query</code> and the native <code class="inline">/v1</code> ingest paths still write on the node that received them. They do not consult the shard plan. <code class="inline">_bulk</code> (index, create, update, and delete) and <code class="inline">POST /{index}/_update/{id}</code> run on the primary, and the primary copies the resulting document or delete to the in-sync replicas.</span></div>
+ <div class="item">Native ingest and delete-by-query stay on the receiver<span class="dsc">The native <code class="inline">/v1</code> ingest paths and <code class="inline">_delete_by_query</code> still write on the node that received them. They do not consult the shard plan. <code class="inline">POST /{index}/_update_by_query</code> runs on every primary. Each primary updates only its own shard and copies those documents to the in-sync replicas. <code class="inline">max_docs</code> counts across those primaries, in node-id order. A primary that does not answer refuses the request before any document is written. A copy that misses quorum is <code class="inline">503</code> <code class="inline">replica_not_acknowledged</code>: the primary has the update. <code class="inline">?pipeline=</code> runs on the primary. Pipelines are not replicated, so a pipeline that is not on that primary is <code class="inline">400</code> and writes nothing.</span></div>
<div class="item">Search merges hits, not every clause<span class="dsc">A query, including <code class="inline">match_all</code>, is run on each owner and the hits are merged by the per-shard score (not a global IDF), then <code class="inline">from</code>/<code class="inline">size</code> is applied. Aggregations, suggest, collapse, rescore, profile, <code class="inline">search_after</code>, and scroll are refused with <code class="inline">400</code> when the index has more than one owner — the response is not a partial merge. <code class="inline">/_search</code> with no index, and a comma-separated or wildcard index list, still search only the node that received them. The per-shard window is capped at 10,000 hits.</span></div>
<div class="item">No replica promotion<span class="dsc">A restarted member reopens <code class="inline">{data_dir}/_raft</code> and applies the snapshot plus the log suffix before it serves, so the shard plan and the in-sync set come back. After 64 applied metadata entries the prefix is snapshotted and a member whose log does not overlap receives InstallSnapshot. The snapshot is the metadata commands, not the documents. A placed replica that is missing those documents copies them from the primary and is published back into the in-sync set; a write still needs the primary, and there is no promotion. A metadata history bigger than 8 MiB stays in the log instead of being snapshotted. Changing membership does not move copies.</span></div>
<div class="item">No node-lifecycle APIs<span class="dsc">There is no drain / activate / remove endpoint and no region split-merge. Earlier revisions of this page documented them; the binary has never shipped them.</span></div>
@@ -255,7 +255,7 @@
<li><strong>Dead-peer isolation</strong> — a hung or dead peer costs one <code class="inline">WARN</code> on the transition and a retry ladder backing off 100 ms → 6.4 s; it never delays messages to live peers.</li>
<li><strong>Index metadata replication</strong> — <code class="inline">PUT /{index}</code>, <code class="inline">DELETE /{index}</code>, <code class="inline">PUT /{index}/_mapping</code>, and <code class="inline">PUT /{index}/_settings</code> are proposed on the member that received them. A follower forwards to the leader. The HTTP call succeeds only after the entry commits on a majority and this node has applied it. No elected leader, or a commit that does not land in time, is <code class="inline">503</code> and the change is not applied on the caller. With <code class="inline">[cluster] enabled = false</code> these calls stay local, as before.</li>
<li><strong>Primary shard plan</strong> — the create entry names one primary per member (member ids sorted, shard <code class="inline">i</code> on the <code class="inline">i</code>th) and, when <code class="inline">index.number_of_replicas</code> is set, the next members in that ring as replicas. <code class="inline">Engine::route_write</code> returns the primary, or nothing when this node is the primary or the index has no plan. Document id is the routing key (jump consistent hash).</li>
- <li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">POST /_bulk</code>, <code class="inline">POST /{index}/_bulk</code>, <code class="inline">POST /{index}/_update/{id}</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. A bulk body is split by primary; the receiver does not keep another member's actions. The primary copies an index, a delete, or the document an update stored to each in-sync replica and answers when a quorum of the copies has confirmed. A search of one concrete index fans out to every primary and merges hits, and reads an in-sync replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li>
+ <li><strong>Document routing</strong> — <code class="inline">PUT</code>/<code class="inline">POST /{index}/_doc</code>, <code class="inline">PUT /{index}/_create/{id}</code>, <code class="inline">POST /_bulk</code>, <code class="inline">POST /{index}/_bulk</code>, <code class="inline">POST /{index}/_update/{id}</code>, <code class="inline">POST /{index}/_update_by_query</code>, <code class="inline">GET</code> and <code class="inline">DELETE /{index}/_doc/{id}</code> run on the primary. A bulk body is split by primary; the receiver does not keep another member's actions. An update-by-query is sent to every primary, and each primary updates only the documents in its shard. The primary copies an index, a delete, or the document an update or update-by-query stored to each in-sync replica and answers when a quorum of the copies has confirmed. A search of one concrete index fans out to every primary and merges hits, and reads an in-sync replica of a primary that does not answer. The copy is a hidden <code class="inline">.xerj_replica_{index}_{shard}</code> index on the replica node, not a second document in the user index, so a live primary search does not count it twice. Replica <code class="inline">_seq_no</code> and <code class="inline">_version</code> are that node's own. The owner is reached on the cluster port (authenticated, one request and one response per connection), not by writing the document into the Raft log.</li>
<li><strong>On-disk Raft log</strong> — cluster mode opens <code class="inline">{data_dir}/_raft</code> (<code class="inline">raft.log</code>, <code class="inline">snapshot.meta</code> once the prefix has been compacted, plus the commit index, term, and vote). The directory name is not a valid index name, so deleting an index cannot delete the log. On startup the node replays the snapshot and then the committed suffix into the shard plan and the in-sync set before it serves. Indices and documents already in that data directory stay; the plan is what tells a later write where they live. A member whose log does not overlap the leader's copies the log, or installs the snapshot when the leader has compacted the prefix, and applies it.</li>
</ul>
@@ -323,10 +323,10 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre>
<h2>What replicates, and what stays on one node</h2>
<p><code class="inline">PUT /orders</code> on any member — leader or follower — creates <code class="inline">orders</code> on every member, including its mapping, the settings and aliases in that request, and a primary on each member. <code class="inline">GET /orders</code> and <code class="inline">GET /orders/_mapping</code> on another member succeed after the call returns. <code class="inline">DELETE /orders</code> removes it on every member. A settings or mapping change is one log entry per concrete index. A follower that cannot reach a leader returns <code class="inline">503</code>; it does not create the index locally and report success.</p>
- <p><code class="inline">PUT /orders/_doc/1</code>, <code class="inline">POST /orders/_update/1</code>, and each action in <code class="inline">POST /orders/_bulk</code> on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. An action whose primary is another member is not written into the accepting node's index. With <code class="inline">number_of_replicas</code> 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set until it copies the primary's live documents and the enlarged set commits, and a later read does not use it before that. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p>
+ <p><code class="inline">PUT /orders/_doc/1</code>, <code class="inline">POST /orders/_update/1</code>, and each action in <code class="inline">POST /orders/_bulk</code> on any member stores the document on the primary for that id and on enough in-sync replicas to make a quorum of the copies. <code class="inline">POST /orders/_update_by_query</code> does the same for every document it changes. An action whose primary is another member is not written into the accepting node's index. With <code class="inline">number_of_replicas</code> 1 that is the one replica. With 2, one of the two is enough; the other is left out of the in-sync set until it copies the primary's live documents and the enlarged set commits, and a later read does not use it before that. <code class="inline">GET /orders/_doc/1</code> from any other member returns it, from the primary while that member answers and from an in-sync replica when it does not. <code class="inline">POST /orders/_search</code> from any member returns documents stored on the primaries, using an in-sync replica only for a primary that does not answer. An auto-created index (a document write that names an index which does not exist yet) proposes the create with an empty schema and <code class="inline">number_of_replicas</code> 0, then routes. With no elected leader that write is <code class="inline">503</code> and the index is not created. <code class="inline">[cluster] enabled = false</code> does not place replicas: the setting is stored, and the document stays on the node that received it.</p>
<p>Several things around that path stay local, on purpose for this step:</p>
<ul>
- <li><strong><code class="inline">_update_by_query</code> and the native <code class="inline">/v1</code> ingest API</strong> do not route or replicate. <code class="inline">_bulk</code> and <code class="inline">_update</code> do. For a copy of the paths that still stay on the receiving node, point the <a href="/docs/config#wal_tap">WAL tap</a> at another XERJ or any ES-compatible target.</li>
+ <li><strong><code class="inline">_delete_by_query</code> and the native <code class="inline">/v1</code> ingest API</strong> do not route or replicate. <code class="inline">_update_by_query</code> does: it runs on every primary and the primary copies the documents it changed. For a copy of the paths that still stay on the receiving node, point the <a href="/docs/config#wal_tap">WAL tap</a> at another XERJ or any ES-compatible target.</li>
<li><strong>Index templates</strong> are not replicated. The accepting node applies its templates once and the effective schema is what the log carries.</li>
<li><strong>ILM/ISM attach</strong> from <code class="inline">index.lifecycle.name</code> runs on the accepting node only.</li>
<li><strong>Console system indices</strong> created at boot, before the ring is attached, have no shard plan and stay local.</li>
@@ -341,7 +341,7 @@ INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1</pre>
</div>
<h2>When not to cluster</h2>
- <p>When you need a replica to accept writes after the primary is gone, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a document write (<code class="inline">PUT</code>/<code class="inline">POST</code> <code class="inline">_doc</code>, <code class="inline">_bulk</code>, or <code class="inline">_update</code>) once a quorum of its copies has it, copies the primary's live documents onto a replica that joined late or missed a write, and reloads metadata when a member restarts. It does not promote a replica, and adding or removing a member does not move copies. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>
+ <p>When you need a replica to accept writes after the primary is gone, or a search that aggregates across shards. Cluster mode keeps index metadata in step, places each document on one primary, acknowledges a document write (<code class="inline">PUT</code>/<code class="inline">POST</code> <code class="inline">_doc</code>, <code class="inline">_bulk</code>, <code class="inline">_update</code>, or <code class="inline">_update_by_query</code>) once a quorum of its copies has it, copies the primary's live documents onto a replica that joined late or missed a write, and reloads metadata when a member restarts. It does not promote a replica, and adding or removing a member does not move copies. One node avoids the forwarding hop and is trivial to back up. The default config ships with <code class="inline">[cluster] enabled = false</code>.</p>
<p class="source-note">Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs</p>