AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(wal-tap): make the retention floor real, and stop calling a redelivery a stall (#320)
xerj-orgad82009e4ad6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex baca8a68b..08ea34075 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -87,13 +87,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`wal_tap.min_retained_generations` (default `0`, unchanged behaviour) buys**bounded** slack: it is a floor, not an Elasticsearch-style retentionlease, so it holds at most `n × storage.wal_max_size_mb` per WAL shard- however far behind any consumer is.+ however far behind any consumer is. The floor is enforced inside+ `IndexStore::wal_maintain_all_verified` — the prune pass the engine actually+ runs — using the same arithmetic and the same "inside the deletion pass, not+ in its callers" placement as Lucene's+ `KeepLastNCommitsDeletionPolicy.onCommit`+ (`lucene/core/src/java/org/apache/lucene/index/KeepLastNCommitsDeletionPolicy.java:51-58`,+ Apache-2.0). At the default `0` the loop is unchanged: no extra syscall, no+ behaviour change.+ - **The retention floor is live, and the API no longer promises otherwise.**+ `min_retained_generations` is the one `[wal_tap]` setting that does not live+ in the tap — it lives in every open index's `WalWriter`, seeded at open from+ `Engine.config`, an `Arc<Config>` written once at boot and never mutated.+ `PUT /_xerj/wal_tap` therefore pushes it straight onto the live writers+ (`Engine::apply_wal_retention_floor`, after Lucene's `LiveIndexWriterConfig`+ and `IndexFileDeleter.revisitPolicy`,+ `lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java:39-126`+ and `IndexFileDeleter.java:516-543`) and answers with+ `retention_floor_applied_to_indices` — the number of writers it reached —+ instead of a warning. `DELETE /_xerj/wal_tap` releases it the same way, and+ `Engine::new` folds the tap's effective configuration into the boot `Config`+ so a store opened after a restart is seeded with the persisted floor rather+ than the file's. Verified over HTTP end to end+ (`xerj-api/tests/wal_tap_retention_floor_is_applied.rs`): after+ `PUT {"min_retained_generations": 3}` an index opened before the call reads+ `3` on all 16 WAL shards, an index created after it opens with `3`, and both+ still read `3` after a restart whose config file still says `0`.- A configuration set through `PUT /_xerj/wal_tap` is persisted next to thecursors and re-applied over the config file on the next boot, so "runtimeconfig, no restart" does not quietly mean "and gone after one".- `DELETE /_xerj/wal_tap` drops the overlay and reverts to the file. The state- file holds `target_auth`, so it is written `0600` through the same writer- the API-key store uses.+ `DELETE /_xerj/wal_tap` drops the overlay and reverts to the file — to the+ value the tap kept verbatim at construction, not to a re-read of+ `Engine.config`. The state file holds `target_auth`, so it is written `0600`+ through the same writer the API-key store uses.+ - The unapplied-batch health signal counts **polls**, not `_bulk` requests.+ How many chunks a poll splits into is `max_batch_bytes` arithmetic: at+ `max_batch_bytes = 1` one legitimate at-least-once redelivery of four+ documents produced four all-conflict `_bulk` responses, tripped the+ three-in-a-row threshold on its own, and emitted `healthy: false` with+ `last_error: "… nothing is being replicated"` about a poll in which every+ document was already at the target. The field is now+ `consecutive_unapplied_polls` and rises by at most one per poll; three+ unapplied polls in a row still reports the stall.+ - `wal_tap.max_retry_backoff_secs` is range-checked (`1..=86400`) by `PUT`+ the way `poll_interval_ms` already was, and every multiply in the backoff+ computation saturates. `Config::validate` does not bound the field, so+ `xerj.toml` can put any `u64` there, and `secs.max(1) * 1000` overflowed+ above ~1.8e16: a panic under `cargo test`'s overflow-checks that killed the+ spawned tap task silently, and under a release profile a *wrapped* cap that+ can be arbitrarily small — the value `18_446_744_073_709_552` wrapped to a+ 384 ms ceiling, i.e. a retry storm aimed at an already-failing target.- XERJ's own `.xerj*` system indices are never shipped, whatever the allowlistsays, and a wildcard never expands to a hidden index.- There is no backfill. A new index is shipped whole; an established one shipsdiff --git a/engine/crates/xerj-api/src/wal_tap_api.rs b/engine/crates/xerj-api/src/wal_tap_api.rsindex 0eb38643b..7f8fb2644 100644--- a/engine/crates/xerj-api/src/wal_tap_api.rs+++ b/engine/crates/xerj-api/src/wal_tap_api.rs@@ -184,8 +184,16 @@ pub async fn put_wal_tap(config.request_timeout_secs = v;}if let Some(v) = patch.max_retry_backoff_secs {- if v == 0 {- return bad_request("wal_tap.max_retry_backoff_secs must be at least 1");+ // Range-checked like `poll_interval_ms` above, not merely `!= 0`.+ // `WalTap::arm_backoff` now saturates so no value here can overflow+ // it, but a cap of "several thousand years" is not a backoff — it is a+ // tap that stops retrying after its first failed send and reports+ // nothing beyond `last_error`. One day is already far past any+ // recovery window a `_bulk` target has.+ if !(1..=86_400).contains(&v) {+ return bad_request(+ "wal_tap.max_retry_backoff_secs must be between 1 and 86400 (one day)",+ );}config.max_retry_backoff_secs = v;}@@ -209,20 +217,26 @@ pub async fn put_wal_tap(}let body = config_body(&config);- // `min_retained_generations` reaches `WalWriter` through- // `IndexStoreConfig` when a store is opened, so a runtime change to it- // applies to indices opened after this point (and to every index after a- // restart, which this patch now survives). Said plainly in the response- // rather than left for an operator to discover.- let retention_takes_effect_on_restart =- patch.min_retained_generations.is_some() && !state.engine.index_name_list().is_empty();state.engine.wal_tap.set_config(config);+ // `min_retained_generations` is the one setting here that does not live in+ // the tap — it lives in every open index's `WalWriter`, and those were+ // seeded from `Engine.config`, an `Arc<Config>` written once at boot and+ // never mutated. Push it onto the live writers NOW, before answering.+ //+ // The previous shape acknowledged the value and attached a warning saying+ // it would take effect after a restart. It did not take effect after a+ // restart: `store_config_from` read the boot file, not the persisted+ // runtime config. An operator raising the floor to survive a target+ // outage, seeing `200`, restarting, and still losing WAL entries had been+ // told a falsehood by the API — on the one knob whose entire purpose is+ // preventing that loss.+ let applied_to = state.engine.apply_wal_retention_floor();let mut resp = json!({"acknowledged": true, "wal_tap": body, "persisted": true});- if retention_takes_effect_on_restart {- resp["warning"] = json!(- "min_retained_generations is read when an index store is opened: already-open \- indices keep the previous retention floor until this node restarts"- );+ if patch.min_retained_generations.is_some() {+ // Not a warning — a receipt. It names the number of live WAL writers+ // the new floor reached, which is the only claim here that an operator+ // cannot verify from the echoed config alone.+ resp["retention_floor_applied_to_indices"] = json!(applied_to);}Json(resp).into_response()}@@ -236,11 +250,17 @@ pub async fn put_wal_tap(/// deleting a file in the data directory. Modelled on `_cluster/settings`,/// where writing `null` clears a persistent setting.pub async fn delete_wal_tap(State(state): State<AppState>) -> Response {- // The engine's boot `Config` is the file's value: the tap took a clone of- // it at construction and only then layered the persisted runtime config- // on top, so this is genuinely "what xerj.toml says".- let file_config = state.engine.config().wal_tap.clone();+ // Taken from the tap, which kept `xerj.toml`'s value verbatim at+ // construction. It used to be read back off `state.engine.config()`, which+ // was only correct while nothing else wrote to that field —+ // `Engine::new` now folds the tap's *effective* config into the boot+ // `Config` so index stores open with the retention floor actually in+ // force, and reading it back here would have reverted a persisted runtime+ // config to itself.+ let file_config = state.engine.wal_tap.file_config();state.engine.wal_tap.clear_runtime_config(file_config);+ // The file's floor is now the live one; the open indices have to be told.+ state.engine.apply_wal_retention_floor();Json(json!({"acknowledged": true,"wal_tap": config_body(&state.engine.wal_tap.config()),diff --git a/engine/crates/xerj-api/tests/wal_tap_retention_floor_is_applied.rs b/engine/crates/xerj-api/tests/wal_tap_retention_floor_is_applied.rsnew file mode 100644index 000000000..960020496--- /dev/null+++ b/engine/crates/xerj-api/tests/wal_tap_retention_floor_is_applied.rs@@ -0,0 +1,231 @@+//! Issue #320 — `PUT /_xerj/wal_tap` must not acknowledge a retention floor+//! it does not apply.+//!+//! `wal_tap.min_retained_generations` is the knob whose entire purpose is+//! keeping WAL entries alive while a replication target is down. It reached+//! `WalWriter` through one path only: `store_config_from`, which reads+//! `Engine.config` — an `Arc<Config>` written once at boot from `xerj.toml`+//! and never mutated. `WalTap::set_config` wrote the tap's own `RwLock` and+//! the state file and nothing else.+//!+//! So a runtime `PUT` reached no writer at all: not the indices already open,+//! and — because the boot `Config` is the *file's* value, not the persisted+//! runtime overlay — not after a restart either. The endpoint answered+//! `200 {"acknowledged":true,"persisted":true,...}` plus a `warning` saying it+//! would take effect on restart. An operator who raised the floor to survive a+//! target outage, saw the `200`, restarted, and still lost WAL entries had been+//! told a falsehood by the API.+//!+//! These tests drive the whole chain over HTTP and then read the floor back off+//! the live `WalWriter`s.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use serde_json::{json, Value};+use tower::ServiceExt;++async fn request(app: &axum::Router, method: &str, path: &str, body: Value) -> (StatusCode, Value) {+ let response = app+ .clone()+ .oneshot(+ Request::builder()+ .method(method)+ .uri(path)+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .expect("request"),+ )+ .await+ .expect("response");+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .expect("response body");+ let body = serde_json::from_slice(&bytes).unwrap_or(Value::Null);+ (status, body)+}++fn app_on(dir: &std::path::Path) -> (axum::Router, xerj_api::state::AppState) {+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ let app = xerj_api::router::build_es_compat_router(state.clone());+ (app, state)+}++/// The floor the API reports must be the floor the WAL writers hold — for the+/// indices that are already open, for indices created afterwards, and across a+/// restart.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn a_retention_floor_set_over_http_reaches_the_live_wal_writers() {+ let dir = tempfile::tempdir().expect("data dir");+ let (app, state) = app_on(dir.path());++ let (status, _) = request(&app, "PUT", "/already-open", json!({})).await;+ assert_eq!(status, StatusCode::OK, "index create");++ let before = state+ .engine+ .get_index("already-open")+ .expect("index")+ .wal_min_retained_generations();+ assert!(+ before.iter().all(|&n| n == 0),+ "the default floor is 0, got {before:?}"+ );++ let (status, body) = request(+ &app,+ "PUT",+ "/_xerj/wal_tap",+ json!({"min_retained_generations": 3}),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "PUT /_xerj/wal_tap: {body}");+ assert_eq!(body["wal_tap"]["min_retained_generations"], 3);+ assert_eq!(+ body["retention_floor_applied_to_indices"], 1,+ "the response must report how many live WAL writers the floor reached: {body}"+ );+ assert!(+ body.get("warning").is_none(),+ "there is nothing left to warn about — it applied: {body}"+ );++ // The claim, checked against the writers themselves.+ let after = state+ .engine+ .get_index("already-open")+ .expect("index")+ .wal_min_retained_generations();+ assert!(+ after.iter().all(|&n| n == 3),+ "an index open BEFORE the PUT must hold the new floor on every WAL shard, got {after:?}"+ );++ // An index created after the PUT opens with it too, rather than inheriting+ // the boot file's 0.+ let (status, _) = request(&app, "PUT", "/created-after", json!({})).await;+ assert_eq!(status, StatusCode::OK, "index create");+ let created_after = state+ .engine+ .get_index("created-after")+ .expect("index")+ .wal_min_retained_generations();+ assert!(+ created_after.iter().all(|&n| n == 3),+ "an index created AFTER the PUT must open with the floor, got {created_after:?}"+ );++ // And a restart. The config FILE still says 0 — nothing wrote to+ // `xerj.toml` — so every part of this depends on the persisted runtime+ // overlay reaching the storage layer.+ drop(app);+ drop(state);+ let (_app2, state2) = app_on(dir.path());+ assert_eq!(+ state2.engine.wal_tap.config().min_retained_generations,+ 3,+ "the persisted runtime floor must survive the restart"+ );+ // The value every `IndexStore` is SEEDED with at open, before anything+ // re-applies it. `Engine::new` folds the tap's effective configuration+ // into the boot `Config` for exactly this: `store_config_from` reads it+ // inside `IndexStore::open`, which replays the WAL, and a store must not+ // spend its replay holding a floor the operator has already changed.+ assert_eq!(+ state2.engine.config().wal_tap.min_retained_generations,+ 3,+ "the boot Config a store opens with must carry the effective floor, not the file's"+ );+ let reopened = state2+ .engine+ .get_index("already-open")+ .expect("index")+ .wal_min_retained_generations();+ assert!(+ reopened.iter().all(|&n| n == 3),+ "after a restart the reopened index must hold the persisted floor, not the config \+ file's — this is the exact case the old `warning` claimed to cover and did not, \+ got {reopened:?}"+ );+}++/// `DELETE /_xerj/wal_tap` reverts to the config file, and that has to reach+/// the writers as well — otherwise clearing an expensive floor would leave the+/// disk cost behind with nothing in the API to show for it.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn deleting_the_runtime_config_releases_the_retention_floor() {+ let dir = tempfile::tempdir().expect("data dir");+ let (app, state) = app_on(dir.path());+ let (status, _) = request(&app, "PUT", "/logs", json!({})).await;+ assert_eq!(status, StatusCode::OK);++ let (status, _) = request(+ &app,+ "PUT",+ "/_xerj/wal_tap",+ json!({"min_retained_generations": 5}),+ )+ .await;+ assert_eq!(status, StatusCode::OK);+ assert!(state+ .engine+ .get_index("logs")+ .expect("index")+ .wal_min_retained_generations()+ .iter()+ .all(|&n| n == 5));++ let (status, body) = request(&app, "DELETE", "/_xerj/wal_tap", json!({})).await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["source"], "config_file");+ assert_eq!(body["wal_tap"]["min_retained_generations"], 0);+ let released = state+ .engine+ .get_index("logs")+ .expect("index")+ .wal_min_retained_generations();+ assert!(+ released.iter().all(|&n| n == 0),+ "reverting to the config file must release the floor on the live writers too, \+ got {released:?}"+ );+}++/// `max_retry_backoff_secs` is range-checked like `poll_interval_ms`, not+/// merely `!= 0`. `WalTap::arm_backoff` saturates so no value can overflow it,+/// but a multi-millennium cap is a tap that stops retrying, not a backoff.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn an_absurd_retry_cap_is_refused_at_the_api() {+ let dir = tempfile::tempdir().expect("data dir");+ let (app, _state) = app_on(dir.path());++ for bad in [0u64, 86_401, u64::MAX] {+ let (status, body) = request(+ &app,+ "PUT",+ "/_xerj/wal_tap",+ json!({"max_retry_backoff_secs": bad}),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::BAD_REQUEST,+ "max_retry_backoff_secs={bad} must be refused, got {status} {body}"+ );+ }++ let (status, body) = request(+ &app,+ "PUT",+ "/_xerj/wal_tap",+ json!({"max_retry_backoff_secs": 86_400}),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "the bound itself is valid: {body}");+ assert_eq!(body["wal_tap"]["max_retry_backoff_secs"], 86_400);+}diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 653e27a0e..64911a48e 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1823,6 +1823,12 @@ pub struct WalTapConfig {/// It is deliberately **not** an Elasticsearch-style retention lease,/// which holds generations for as long as a follower is behind and is how/// a dead follower wedges a leader's disk.+ ///+ /// Unlike every other field here, this one does not live in the tap: it+ /// lives in each index's `WalWriter`. `PUT /_xerj/wal_tap` therefore+ /// pushes it onto the open writers itself+ /// (`Engine::apply_wal_retention_floor`) and reports how many it reached,+ /// rather than acknowledging a value that reaches nothing.pub min_retained_generations: u64,}diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex 0e92ec6de..524acaef7 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -702,6 +702,16 @@ impl Engine {// reading the frozen boot config.let wal_tap_config = config.wal_tap.clone();+ // Built HERE, ahead of the struct literal, because `WalTap::new` is+ // what loads the persisted runtime configuration and it can override+ // `min_retained_generations` — which `store_config_from` reads out of+ // this `Config` when every index opens a few lines below. Freezing the+ // *file's* floor into the `Arc` while the tap ran with a different one+ // is how a floor set through `PUT /_xerj/wal_tap` survived a restart+ // in `_stats` and still reached no `WalWriter`.+ let wal_tap = Arc::new(crate::wal_tap::WalTap::new(&data_dir, wal_tap_config));+ config.wal_tap = wal_tap.config();+// Apply operator-tunable aggregation bucket cap. Stored in a static// AtomicUsize inside aggs.rs so all per-bucket-allocator hot loops// can read it with no plumbing through every agg signature.@@ -755,7 +765,7 @@ impl Engine {frozen_indices: Arc::new(DashMap::new()),rollup_jobs: Arc::new(DashMap::new()),ccr_auto_follow: Arc::new(DashMap::new()),- wal_tap: Arc::new(crate::wal_tap::WalTap::new(&data_dir, wal_tap_config)),+ wal_tap,api_keys: Arc::new(DashMap::new()),application_privileges: Arc::new(DashMap::new()),legacy_templates: Arc::new(DashMap::new()),@@ -828,6 +838,13 @@ impl Engine {match Index::open(index_name.clone(), &engine.config, &data_dir) {Ok(idx) => {info!(name = name_str.as_str(), "opened existing index");+ // #320 — `engine.config.wal_tap` is already the+ // tap's effective configuration (see `WalTap::new`+ // above), so the store opened with the right+ // floor; this makes that true by construction+ // rather than by argument, and covers a tap+ // reconfigured between two boot-loop iterations.+ idx.set_wal_min_retained_generations(engine.wal_retention_floor());// Restore the raw ES mapping blob (analyzers, formats,// dims — full fidelity) BEFORE any ingest/query can run,// so GET /_mapping and mapping-dependent code paths see@@ -1240,6 +1257,11 @@ impl Engine {&self.config,&self.data_dir,)?;+ // #320 — the store was seeded from the boot `Config`; the tap's live+ // floor is what an operator last asked for. Applied before the index+ // is published, so no ingest (and therefore no prune) can run against+ // the wrong floor.+ idx.set_wal_min_retained_generations(self.wal_retention_floor());self.indices.insert(name.to_string(), idx);self.record_index_created_at(name);info!(name, "index created with custom settings");@@ -2663,6 +2685,9 @@ impl Engine {let index_name = IndexName::new(name).map_err(EngineError::Common)?;match Index::open(index_name, &self.config, &self.data_dir) {Ok(idx) => {+ // #320 — a reopened index gets the tap's live retention floor,+ // not the boot file's.+ idx.set_wal_min_retained_generations(self.wal_retention_floor());// The store opened, but the full-fidelity mapping blob is part// of what makes this index serveable (#202). Putting the index// back with a silently reduced mapping is exactly the defect@@ -2832,6 +2857,48 @@ impl Engine {.collect()}+ /// The WAL-consumer retention floor the tap is currently configured with.+ ///+ /// Read from the tap, never from `self.config`: the tap's copy is the one+ /// an operator can change at runtime, and after a restart it is the one+ /// that was persisted. `self.config` is the boot file, frozen in an `Arc`.+ fn wal_retention_floor(&self) -> u64 {+ self.wal_tap.config().min_retained_generations+ }++ /// Push the tap's live retention floor onto every open index's WAL shards.+ ///+ /// Returns the number of indices it reached, so the `PUT /_xerj/wal_tap`+ /// handler can report what actually happened instead of guessing.+ ///+ /// #320. A floor set through the API previously reached nothing: it lived+ /// in the tap's own `RwLock` and in the state file, while every+ /// `WalWriter` had been seeded from `Engine.config` — an `Arc<Config>`+ /// written once at boot from `xerj.toml` and never mutated. The API+ /// answered `200` with the new value and a warning that it would apply+ /// after a restart; it did not apply after a restart either. This is the+ /// half that makes it true for indices that are already open.+ ///+ /// Re-applying a policy to already-live state on change, rather than only+ /// at construction, is Lucene's `IndexFileDeleter.revisitPolicy`+ /// (lucene/core/src/java/org/apache/lucene/index/IndexFileDeleter.java:516-543,+ /// Apache-2.0), which re-runs `policy.onCommit(commits)` over the commits+ /// that already exist rather than waiting for the next one.+ pub fn apply_wal_retention_floor(&self) -> usize {+ let floor = self.wal_retention_floor();+ let mut applied = 0usize;+ for entry in self.indices.iter() {+ entry.value().set_wal_min_retained_generations(floor);+ applied += 1;+ }+ tracing::debug!(+ floor,+ indices = applied,+ "WAL tap: retention floor applied to open indices"+ );+ applied+ }+/// Sum the internal query-result cache hit/miss counters across every open/// index (RC4-W4 item 4). Returns `(hits, misses)`. Reconciled into the/// `xerj_query_cache_{hits,misses}` gauges by the `/v1/metrics` handler at@@ -3592,6 +3659,9 @@ impl Engine {// Reopen the index.match Index::open(index_name, &self.config, &self.data_dir) {Ok(idx) => {+ // #320 — a restored index gets the tap's live retention+ // floor, not the boot file's.+ idx.set_wal_min_retained_generations(self.wal_retention_floor());// Snapshot dirs carry es_mapping.json — reload it so the// restored index serves the same mapping it was saved with.// A corrupt blob in the snapshot fails the restore of thatdiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 85927eaca..1afc4b49a 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -5747,6 +5747,22 @@ pub struct Index {}impl Index {+ /// Apply a WAL-consumer retention floor to this index's WAL shards, live.+ ///+ /// #320. Called by [`Engine`](crate::engine::Engine) immediately after+ /// every open/create, and again on every index when an operator changes+ /// `wal_tap.min_retained_generations` through `PUT /_xerj/wal_tap`. See+ /// [`xerj_storage::index_store::IndexStore::set_wal_min_retained_generations`]+ /// for why the open-time config value alone was not enough.+ pub fn set_wal_min_retained_generations(&self, n: u64) {+ self.store.set_wal_min_retained_generations(n);+ }++ /// The retention floor in force on each of this index's WAL shards.+ pub fn wal_min_retained_generations(&self) -> Vec<u64> {+ self.store.wal_min_retained_generations()+ }+fn begin_hnsw_publication(&self) -> HnswPublicationGuard {HnswPublicationGuard::begin(&self.hnsw_publications_in_flight,@@ -28174,13 +28190,18 @@ fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> Indschema_version: 1,storage_mode: xerj_storage::StorageMode::Local,num_wal_shards: wal_shards_override.unwrap_or(config.engine.ingest_shards),- // #320 — the retention floor a WAL consumer needs. Read here, from- // the boot config, because it is consumed once when a `WalWriter` is- // opened; a runtime `PUT /_xerj/wal_tap` that changes it therefore- // applies to stores opened after the change (and to every store after- // a restart, which the persisted runtime config now survives). The- // `PUT` handler says so in its response rather than leaving it to be- // discovered.+ // #320 — the retention floor a WAL consumer needs, as the store's+ // *seed* value.+ //+ // `Engine::new` overwrites `config.wal_tap` with the tap's effective+ // configuration before freezing the `Config` into its `Arc`, so this+ // is the persisted runtime value when one exists and the file's value+ // otherwise — i.e. correct from byte zero for every store opened at+ // boot. A runtime `PUT /_xerj/wal_tap` cannot reach an `Arc<Config>`+ // at all, so it is applied by+ // `Engine::apply_wal_retention_floor` straight onto the live+ // `WalWriter`s instead. Both halves are required: this one alone+ // acknowledged a floor that never reached a writer.wal_min_retained_generations: config.wal_tap.min_retained_generations,}}diff --git a/engine/crates/xerj-engine/src/wal_tap.rs b/engine/crates/xerj-engine/src/wal_tap.rsindex c82bcbb25..74fbe7d0a 100644--- a/engine/crates/xerj-engine/src/wal_tap.rs+++ b/engine/crates/xerj-engine/src/wal_tap.rs@@ -64,10 +64,13 @@//! version for every `_id` because a second writer got there first, or one//! rejecting every document with a `mapper_parsing_exception` — is a real//! state and used to be indistinguishable from a healthy tap.-//! `consecutive_unapplied_batches`, `last_item_rejection` and+//! `consecutive_unapplied_polls`, `last_item_rejection` and//! [`IndexTapStats::healthy`] exist so it is not. The counter is deliberately//! blind to *why* nothing applied: conflict and rejection are opposite-//! diagnoses but the same fact, and it is the fact that `healthy` reports.+//! diagnoses but the same fact, and it is the fact that `healthy` reports. It+//! counts **polls**, not `_bulk` requests: how many chunks a poll splits into+//! is `max_batch_bytes` arithmetic, and charging each of them separately made+//! one ordinary redelivery look like a dead replication stream.//!//! **3. Backpressure.** Bounded by construction: one in-flight request per//! index, batches capped by `max_batch_docs` and `max_batch_bytes`, and@@ -84,10 +87,11 @@//! Seed the target with snapshot/restore if it needs the existing documents.//!//! **4b. What falling behind costs.** WAL retention is *not* coupled to the-//! tap, deliberately: `prune_verified` deletes a rotated generation as soon as-//! its entries are durable in a segment, and making that wait on a remote-//! target is how a slow consumer turns into a disk-full incident. So a tap-//! that falls far enough behind loses entries — and always finds out. The+//! tap, deliberately: `IndexStore::wal_maintain_all_verified` deletes a+//! rotated generation as soon as its entries are durable in a segment, and+//! making that wait on a remote target is how a slow consumer turns into a+//! disk-full incident. So a tap that falls far enough behind loses entries —+//! and always finds out. The//! cursor carries the generation it sits in, a pruned-out-from-under-us//! generation is detected exactly ([`WalShardTail::gap`]), and each occurrence//! logs at `warn` and increments `gaps` in `GET /_xerj/wal_tap/_stats`. The@@ -97,13 +101,29 @@//! `storage.flush_interval_secs = 30` the WAL window is about one flush — so a//! minute of target downtime on a hot index loses data with nothing but a log//! line to offer. `wal_tap.min_retained_generations` is the middle path: a-//! **bounded** floor under `prune_verified` that holds the newest N rotated-//! generations whatever any consumer is doing. Deliberately not a retention-//! lease (Elasticsearch's `ReplicationTracker.addRetentionLease`, Lucene's+//! **bounded** floor inside that maintenance pass, holding the newest N+//! rotated generations whatever any consumer is doing. Same arithmetic and+//! same placement — inside the deletion pass, not in its callers — as Lucene's+//! `KeepLastNCommitsDeletionPolicy.onCommit`+//! (`lucene/core/src/java/org/apache/lucene/index/KeepLastNCommitsDeletionPolicy.java:51-58`,+//! Apache-2.0). Deliberately **not** a retention lease (Elasticsearch's+//! `ReplicationTracker.addRetentionLease`, Lucene's//! `SnapshotDeletionPolicy.snapshot`): a lease is held until the consumer//! releases it, which is exactly how a dead follower wedges a leader's disk.//! A floor costs `n * wal_max_size_mb` per shard and not one byte more.//!+//! The floor is **live**: `PUT /_xerj/wal_tap` pushes it straight onto the+//! open `WalWriter`s (`Engine::apply_wal_retention_floor`), because the value+//! that reaches a store at open time comes from `Engine.config`, an+//! `Arc<Config>` written once at boot and never mutated. Setting it through+//! the API used to reach no writer at all — not the open ones, and not after a+//! restart either — on the one knob whose whole purpose is preventing loss+//! during a target outage. Live reconfiguration of an already-open writer is+//! Lucene's `LiveIndexWriterConfig`+//! (`lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java:39-126`),+//! and re-applying a policy to state that already exists rather than only at+//! construction is its `IndexFileDeleter.revisitPolicy` (`:516-543`).+//!//! **4c. Where the configuration lives.** In the state file next to the//! cursors, not only in memory. `PUT /_xerj/wal_tap` is durable and is//! re-applied over the config file on the next boot; `DELETE /_xerj/wal_tap`@@ -268,8 +288,18 @@ pub struct IndexTapStats {/// instead is what let a target that rejected every single action report/// `lag_seq: 0` while `docs_shipped` climbed.pub last_shipped_seq: u64,- /// Consecutive `_bulk` responses in which the target accepted **nothing**- /// — every action inside came back with an error.+ /// Consecutive **polls** in which the target accepted **nothing** — every+ /// action of every `_bulk` the poll sent came back with an error.+ ///+ /// Counted per poll, not per `_bulk`. A poll splits its read into as many+ /// chunks as `max_batch_bytes` dictates, and those chunks are one logical+ /// delivery: at `max_batch_bytes = 1` a single legitimate at-least-once+ /// redelivery of four documents produced four all-conflict `_bulk`+ /// responses and tripped [`UNAPPLIED_UNHEALTHY_AFTER`] on its own, with a+ /// `last_error` claiming "nothing is being replicated" about a poll in+ /// which every document was already at the target. The chunk count is a+ /// transport detail; the operator-visible unit of "did this tap make+ /// progress" is the poll.////// Deliberately blind to *why*. A version conflict and a/// `mapper_parsing_exception` are opposite diagnoses but the same@@ -279,11 +309,11 @@ pub struct IndexTapStats {/// read `healthy: true` with `docs_shipped: 0` — the exact class of lie/// the per-item accounting exists to remove.///- /// One such batch is expected and benign: it is how external versioning+ /// One such poll is expected and benign: it is how external versioning/// absorbs an at-least-once redelivery. A *sustained run* is not — it means/// the tap is a no-op and the two sides are diverging. See/// [`IndexTapStats::healthy`].- pub consecutive_unapplied_batches: u32,+ pub consecutive_unapplied_polls: u32,/// Error `type` of the most recent item the target rejected for a reason/// other than a version conflict, e.g. `mapper_parsing_exception`.///@@ -293,13 +323,17 @@ pub struct IndexTapStats {pub last_item_rejection: Option<String>,}-/// How many batches in a row that applied nothing stop counting as "a+/// How many **polls** in a row that applied nothing stop counting as "a/// redelivery landed" and start counting as "this target is not taking our/// writes".///-/// A redelivery after a failed send re-ships exactly the chunks that did not-/// land, so it clears on the next batch that contains anything new. Three in a-/// row cannot be that.+/// Polls, not `_bulk` requests. A poll that applies nothing still advances the+/// cursor past the entries it read (rejected items are not retried — see+/// [`WalTap::send_bulk`]), so the *next* poll reads new entries and a healthy+/// stream clears the counter immediately. A redelivery — a rewound cursor, or+/// a send that failed part-way through and re-ships its earlier chunks on the+/// next poll — is at most one such poll, however many chunks it took.+/// Three in a row cannot be that.const UNAPPLIED_UNHEALTHY_AFTER: u32 = 3;impl IndexTapStats {@@ -310,7 +344,7 @@ impl IndexTapStats {/// every request while rejecting every action inside it — for any reason,/// conflict or otherwise.pub fn healthy(&self) -> bool {- self.last_error.is_none() && self.consecutive_unapplied_batches < UNAPPLIED_UNHEALTHY_AFTER+ self.last_error.is_none() && self.consecutive_unapplied_polls < UNAPPLIED_UNHEALTHY_AFTER}}@@ -322,6 +356,16 @@ impl IndexTapStats {/// surface can read stats and edit the allowlist without a restart.pub struct WalTap {config: RwLock<WalTapConfig>,+ /// Exactly what `xerj.toml` said, kept verbatim so `DELETE+ /// /_xerj/wal_tap` has something true to revert to.+ ///+ /// It used to be read back off `Engine.config`, which was safe only while+ /// nothing else ever wrote to that field. `Engine::new` now folds the+ /// tap's *effective* configuration into the boot `Config` (so index stores+ /// open with the retention floor actually in force), which would have made+ /// `DELETE` revert a persisted runtime config to… itself. Owning the file+ /// value here removes the coupling instead of documenting around it.+ file_config: WalTapConfig,state: RwLock<PersistedState>,stats: RwLock<BTreeMap<String, IndexTapStats>>,state_path: PathBuf,@@ -383,6 +427,8 @@ impl WalTap {Err(_) => PersistedState::default(),};+ let file_config = config.clone();+// A configuration set through the API outlives the process that took// it. Without this a restart reverted to the file config — usually// `enabled = false` — and the tap went quiet while its cursors froze@@ -408,6 +454,7 @@ impl WalTap {Self {config: RwLock::new(config),+ file_config,state: RwLock::new(state),stats: RwLock::new(BTreeMap::new()),state_path,@@ -425,6 +472,12 @@ impl WalTap {self.config.read().clone()}+ /// What `xerj.toml` said at boot — the value `DELETE /_xerj/wal_tap`+ /// reverts to.+ pub fn file_config(&self) -> WalTapConfig {+ self.file_config.clone()+ }+/// Replace the live configuration **durably**.////// Takes effect on the next poll — no restart — and cursors are preserved@@ -603,20 +656,34 @@ impl WalTap {/// Capped exponential backoff with jitter, keyed off the worst/// consecutive-failure count across indices.///+ /// **Every multiply here saturates**, including the two that used to be+ /// plain `*`. `max_retry_backoff_secs` and `poll_interval_ms` reach this+ /// function from two places, and only one of them is range-checked:+ /// `PUT /_xerj/wal_tap` bounds both, but `Config::validate` bounds+ /// neither, so `xerj.toml` can put any `u64` in either field. At+ /// `max_retry_backoff_secs > u64::MAX / 1000` (~1.8e16) the old+ /// `secs.max(1) * 1000` overflowed — a panic under `cargo test`'s+ /// overflow-checks that killed the spawned tap task with nothing in+ /// `_stats` to say why, and in a release profile a *wrapped* cap that can+ /// be arbitrarily small: the concrete value 18_446_744_073_709_552 wraps+ /// to a 384 ms ceiling, i.e. a retry storm aimed at a target that is+ /// already failing. `uncapped` on the line above already saturated for+ /// exactly this reason.+ ////// Approach (not code) from Elasticsearch's `ShardFollowNodeTask.computeDelay`.fn arm_backoff(&self, config: &WalTapConfig, failures: u32) {let base = config.poll_interval_ms.max(50);let shift = failures.saturating_sub(1).min(20);let uncapped = base.saturating_mul(1u64 << shift);- let cap = config.max_retry_backoff_secs.max(1) * 1000;+ let cap = config.max_retry_backoff_secs.max(1).saturating_mul(1000);let delay = uncapped.min(cap);// Deterministic, dependency-free jitter in [50%, 100%] of the delay:// enough to stop N indices from re-hammering a recovering target in// lockstep, without pulling in an RNG.let jitter = self.now_ms() % 50 + 50;- let delay = delay * jitter / 100;+ let delay = delay.saturating_mul(jitter) / 100;self.retry_not_before_ms- .store(self.now_ms() + delay, Ordering::Relaxed);+ .store(self.now_ms().saturating_add(delay), Ordering::Relaxed);}fn clear_backoff(&self) {@@ -1037,7 +1104,11 @@ impl WalTap {let mut item_failures = 0u64;let mut version_conflicts = 0u64;let mut batches = 0u64;- let mut unapplied_batches = 0u32;+ // Chunks in THIS poll that applied nothing. Kept separate from the+ // health counter on purpose: that counter is per poll (see+ // `UNAPPLIED_UNHEALTHY_AFTER`), and adding this into it is what made+ // one legitimate multi-chunk redelivery report the tap unhealthy.+ let mut unapplied_chunks = 0u32;let mut rejection_kind: Option<String> = None;for chunk in chunks {@@ -1071,7 +1142,7 @@ impl WalTap {// let an all-`mapper_parsing_exception` response read as a// healthy zero-lag tap with `docs_shipped: 0`.if accepted_here == 0 && !actions.is_empty() {- unapplied_batches += 1;+ unapplied_chunks += 1;}if rejection.is_some() {rejection_kind = rejection;@@ -1117,17 +1188,24 @@ impl WalTap {if let Some(kind) = &rejection_kind {s.last_item_rejection = Some(kind.clone());}- // A run of batches that applied nothing is not "replication- // working". Only a batch that landed something clears it.- if unapplied_batches > 0 && docs == 0 && deletes == 0 {- s.consecutive_unapplied_batches = s- .consecutive_unapplied_batches- .saturating_add(unapplied_batches);+ // A run of POLLS that applied nothing is not "replication+ // working". Only a poll that landed something clears it.+ //+ // `+= 1`, not `+= unapplied_chunks`. How many `_bulk` requests a+ // poll needed is a function of `max_batch_bytes`, not of whether+ // replication is progressing: at `max_batch_bytes = 1` the old+ // shape turned ONE legitimate at-least-once redelivery of four+ // documents into `consecutive_unapplied = 4`, tripping the+ // three-in-a-row threshold on the documented-benign path and+ // emitting a `last_error` saying "nothing is being replicated"+ // about a poll in which every document was already at the target.+ if unapplied_chunks > 0 && docs == 0 && deletes == 0 {+ s.consecutive_unapplied_polls = s.consecutive_unapplied_polls.saturating_add(1);} else {- s.consecutive_unapplied_batches = 0;+ s.consecutive_unapplied_polls = 0;}- if s.consecutive_unapplied_batches >= UNAPPLIED_UNHEALTHY_AFTER {- let run = s.consecutive_unapplied_batches;+ if s.consecutive_unapplied_polls >= UNAPPLIED_UNHEALTHY_AFTER {+ let run = s.consecutive_unapplied_polls;// Name the cause the target gave *this poll*, not whatever it// last gave: a conflict and a mapping error need opposite// responses from the operator, and a stale reason would send@@ -1138,13 +1216,13 @@ impl WalTap {.or(s.last_item_rejection.as_deref()).unwrap_or("an unnamed error");format!(- "target accepted nothing in the last {run} batches — it rejected \+ "target accepted nothing in the last {run} polls — it rejected \every action, most recently with {kind}. Nothing is being \replicated; these documents are being dropped, not retried.")} else {format!(- "target rejected every action of the last {run} batches with \+ "target rejected every action of the last {run} polls with \version_conflict_engine_exception — it holds a higher version for \every document this tap sends, so nothing is being replicated")@@ -1403,4 +1481,36 @@ mod tests {"backoff must still be substantial, got {delay}ms");}++ /// `max_retry_backoff_secs` reaches [`WalTap::arm_backoff`] from the config+ /// file as well as from `PUT /_xerj/wal_tap`, and `Config::validate` does+ /// not range-check it — so any `u64` in `xerj.toml` gets here.+ ///+ /// The value below is chosen so the OLD `secs.max(1) * 1000` fails in both+ /// profiles rather than only one: `18_446_744_073_709_552 * 1000` is+ /// `18_446_744_073_709_552_000`, which is `2^64 + 384`, so it panics under+ /// `cargo test`'s overflow-checks (killing the spawned tap task with+ /// nothing in `_stats` to say why) and **wraps to a 384 ms cap** under+ /// `--profile ci-test` — a retry storm aimed at a target that is already+ /// failing. Saturating leaves the exponential backoff itself in charge.+ #[test]+ fn an_unbounded_retry_cap_saturates_instead_of_wrapping() {+ let dir = tempfile::tempdir().unwrap();+ let config = WalTapConfig {+ poll_interval_ms: 500,+ max_retry_backoff_secs: 18_446_744_073_709_552,+ ..Default::default()+ };+ let tap = WalTap::new(dir.path(), config.clone());+ // failures = 3 → shift 2 → uncapped = 500 * 4 = 2_000 ms, then jitter+ // in [50%, 100%].+ tap.arm_backoff(&config, 3);+ let armed = tap.retry_not_before_ms.load(Ordering::Relaxed);+ let delay = armed.saturating_sub(tap.now_ms());+ assert!(+ (900..=2_000).contains(&delay),+ "a cap too large to express in ms must not become a SMALL cap; the wrapped \+ value gives at most 384ms. got {delay}ms"+ );+ }}diff --git a/engine/crates/xerj-engine/tests/wal_tap_push.rs b/engine/crates/xerj-engine/tests/wal_tap_push.rsindex 5a3b783f0..6ab1f9bd2 100644--- a/engine/crates/xerj-engine/tests/wal_tap_push.rs+++ b/engine/crates/xerj-engine/tests/wal_tap_push.rs@@ -908,6 +908,99 @@ async fn redelivery_converges_at_a_target_that_honours_external_versioning() {);}+/// The health signal counts **polls** that applied nothing, not `_bulk`+/// requests.+///+/// How many chunks a poll splits into is `max_batch_bytes` arithmetic, not a+/// statement about replication. Charging the unapplied counter once per chunk+/// meant one legitimate at-least-once redelivery — the path the module+/// documents as expected and benign, and the entire reason every action+/// carries `version_type: external` — tripped the three-in-a-row threshold on+/// its own and reported the tap as broken.+///+/// Fails before the fix with `healthy = false`,+/// `consecutive_unapplied_polls = 4`, and+/// `last_error = "target rejected every action of the last 4 polls with+/// version_conflict_engine_exception … so nothing is being replicated"` — a+/// message that is factually wrong, because all four documents are at the+/// target and this test asserts so.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn a_multi_chunk_redelivery_is_one_unapplied_poll_not_four() {+ let target = StubTarget::start_with(TargetSemantics::ExternalVersioning).await;+ let dir = TempDir::new().unwrap();+ let mut tap = tap_config(&target.url, &["edge-logs"]);+ // One action per `_bulk`: the smallest legal cap, and the shape any+ // operator with a low `http.max_content_length` at the target gets.+ tap.max_batch_bytes = 1;+ let engine = engine_with_tap(&dir, tap);++ write_docs(&engine, "edge-logs", &["a", "b", "c", "d"]).await;+ engine.wal_tap.tick(&engine).await;+ let landed = target.stored_versions();+ assert_eq!(+ landed.len(),+ 4,+ "the first delivery must land all four documents: {landed:?}"+ );+ let first_batches = engine.wal_tap.stats()["edge-logs"].batches_sent;+ assert!(+ first_batches >= 4,+ "the point of this test is a MULTI-chunk poll; got {first_batches} _bulk request(s)"+ );++ // One rewind, one poll: four all-conflict `_bulk` responses that are+ // together a single, correct, no-op redelivery.+ engine.wal_tap.rewind_for_test("edge-logs");+ engine.wal_tap.tick(&engine).await;++ let stats = engine.wal_tap.stats();+ let edge = stats.get("edge-logs").expect("stats");+ assert_eq!(+ edge.version_conflicts, 4,+ "every redelivered action must still be reported as a conflict"+ );+ assert_eq!(+ edge.consecutive_unapplied_polls, 1,+ "one poll that applied nothing is ONE, however many chunks it took"+ );+ assert!(+ edge.healthy(),+ "a redelivery is the documented-benign path and must not report the tap broken: \+ last_error = {:?}",+ edge.last_error+ );+ assert_eq!(+ edge.last_error, None,+ "…and must not leave an operator-facing message behind either"+ );+ assert_eq!(+ target.stored_versions(),+ landed,+ "the target still holds exactly the four documents it already had"+ );++ // The signal must still fire on a genuinely stuck stream: three polls in a+ // row that apply nothing is not a redelivery.+ for _ in 0..2 {+ engine.wal_tap.rewind_for_test("edge-logs");+ engine.wal_tap.tick(&engine).await;+ }+ let stats = engine.wal_tap.stats();+ let edge = stats.get("edge-logs").expect("stats");+ assert_eq!(edge.consecutive_unapplied_polls, 3);+ assert!(+ !edge.healthy(),+ "three unapplied polls in a row is a real stall and must still be reported"+ );+ assert!(+ edge.last_error+ .as_deref()+ .is_some_and(|e| e.contains("last 3 polls")),+ "and the message must count the unit it actually counts: {:?}",+ edge.last_error+ );+}+/// A target that holds a higher version for every `_id` — a second writer got/// there first — turns every action into a rejected no-op. `_stats` used to/// call that healthy with `lag_seq: 0` and a climbing `docs_shipped`, because@@ -1084,7 +1177,7 @@ async fn health_recovers_once_the_target_accepts_again() {edge.last_error);assert_eq!(- edge.consecutive_unapplied_batches, 0,+ edge.consecutive_unapplied_polls, 0,"the run counter must reset, not decay");assert!(diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rsindex d5f494c42..5e5a38b5b 100644--- a/engine/crates/xerj-storage/src/index_store.rs+++ b/engine/crates/xerj-storage/src/index_store.rs@@ -2695,7 +2695,43 @@ impl IndexStore {}// Prune pass over all rotated generations, cache-first.- for gen in wal.rotated_generations()? {+ //+ // #320 — the WAL-consumer retention floor is applied HERE, not+ // only in `WalWriter::prune_verified`. `prune_verified` has no+ // production caller (grep: `wal.rs` and its own unit tests are the+ // only hits); this loop is the prune the engine actually runs, so+ // a floor enforced only there was fully unit-tested and still had+ // zero effect on a live node.+ //+ // `rotated_generations` is sorted ascending, so the newest are the+ // tail and the prunable prefix is `len - keep`. Same arithmetic,+ // and the same "enforce it inside the deletion pass rather than in+ // its callers" placement, as Lucene's+ // `KeepLastNCommitsDeletionPolicy.onCommit`+ // (lucene/core/src/java/org/apache/lucene/index/KeepLastNCommitsDeletionPolicy.java:51-58,+ // Apache-2.0): "The commits list is already sorted from oldest to+ // newest / for (i = 0; i < size - numCommitsToKeep; i++)+ // commits.get(i).delete()". Adapted, not copied.+ //+ // At the default floor of 0 this is `keep = 0` and the loop is+ // byte-for-byte the old one: no extra syscall, no behaviour change.+ let rotated = wal.rotated_generations()?;+ let keep = (wal.min_retained_generations() as usize).min(rotated.len());+ let prunable_prefix = rotated.len() - keep;+ for (pos, gen) in rotated.into_iter().enumerate() {+ if pos >= prunable_prefix {+ // Held for a WAL consumer. Deliberately skipped BEFORE the+ // cache lookup: decoding a generation we are not going to+ // delete is the O(retained WAL bytes)/tick cost the prune+ // cache exists to avoid.+ debug!(+ gen,+ shard = ws_idx,+ keep,+ "WAL generation retained: consumer retention floor"+ );+ continue;+ }let mut cache = self.wal_prune_cache.lock().unwrap();let verdict = cache.entry((ws_idx, gen)).or_insert_with(|| {// Cache miss: a generation rotated by the size-based@@ -2762,6 +2798,46 @@ impl IndexStore {.collect()}+ /// Change the WAL-consumer retention floor on every shard of this store,+ /// **live**.+ ///+ /// #320. `IndexStoreConfig.wal_min_retained_generations` is read once, at+ /// open, from the boot config file — and `Engine.config` is an+ /// `Arc<Config>` that is never mutated, so without this a floor set+ /// through `PUT /_xerj/wal_tap` reached no writer at all: not the ones+ /// already open, and not the ones opened after a restart either. The knob+ /// exists to stop a target outage turning into data loss, so "acknowledged+ /// but inert" is the one failure mode it must not have.+ ///+ /// Live reconfiguration of an already-open writer is Lucene's+ /// `LiveIndexWriterConfig`+ /// (lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java:39-126,+ /// Apache-2.0): "Holds all the configuration used by IndexWriter with few+ /// setters for settings that can be changed on an IndexWriter instance+ /// 'live'." Everything else there is explicitly documented as "Only takes+ /// effect when IndexWriter is first created"+ /// (`IndexWriterConfig.setCodec`, :297-305). A setting is one or the+ /// other, and says which; it is never claimed live and implemented at+ /// create.+ pub fn set_wal_min_retained_generations(&self, n: u64) {+ for shard in &self.wal_shards {+ shard.lock().unwrap().set_min_retained_generations(n);+ }+ }++ /// The retention floor in force on each WAL shard, in shard order.+ ///+ /// Instrumentation for the property that+ /// [`set_wal_min_retained_generations`](Self::set_wal_min_retained_generations)+ /// exists to provide: it must be assertable that a runtime change reached+ /// every writer, not just the first.+ pub fn wal_min_retained_generations(&self) -> Vec<u64> {+ self.wal_shards+ .iter()+ .map(|s| s.lock().unwrap().min_retained_generations())+ .collect()+ }+/// Unconditionally run verified WAL maintenance across all shards./// Bypasses the `WAL_MAINTENANCE_INTERVAL_MS` gate that/// `finalize_flush_with_publisher` uses on the hot flush path.@@ -5212,6 +5288,93 @@ mod tests {}}+ /// #320 — the WAL-consumer retention floor must be honoured by the prune+ /// pass the ENGINE runs, which is `wal_maintain_all_verified`, not+ /// `WalWriter::prune_verified`.+ ///+ /// `prune_verified` has no production caller anywhere in the tree+ /// (`rg 'prune_verified\('` hits `wal.rs` and its own unit tests only), so+ /// a floor implemented and unit-tested there alone was decoration: on a+ /// live node every rotated generation was still deleted the moment a flush+ /// made it redundant, and a tap whose target was down for longer than one+ /// flush interval still lost entries.+ ///+ /// Fails before the fix with `rotated after maintenance: 0`.+ #[test]+ fn the_retention_floor_holds_generations_through_engine_wal_maintenance() {+ fn rotated_wal_files(store: &IndexStore) -> usize {+ // One file per shard is the ACTIVE generation and is never a prune+ // candidate; everything beyond that is retained-rotated.+ walk_wal_files(&store.wal_dir()).len() - store.wal_min_retained_generations().len()+ }++ // Control: the default floor of 0 reclaims everything it can prove+ // durable, exactly as before this change.+ let bare_dir = tempfile::tempdir().unwrap();+ let bare = open_test_store(bare_dir.path());+ for round in 0..4 {+ for i in 0..5 {+ bare.index(format!("b{round}_{i}"), serde_json::json!({"v": i}))+ .unwrap();+ }+ bare.flush().unwrap();+ bare.force_wal_maintenance().unwrap();+ }+ assert_eq!(+ rotated_wal_files(&bare),+ 0,+ "floor 0 must keep pruning to the active generation, files: {:?}",+ walk_wal_files(&bare.wal_dir())+ );++ // Floor 2: the newest two rotated generations survive every+ // maintenance pass, so a consumer two generations behind still finds+ // its entries.+ let dir = tempfile::tempdir().unwrap();+ let store = open_test_store(dir.path());+ store.set_wal_min_retained_generations(2);+ for round in 0..4 {+ for i in 0..5 {+ store+ .index(format!("d{round}_{i}"), serde_json::json!({"v": i}))+ .unwrap();+ }+ store.flush().unwrap();+ store.force_wal_maintenance().unwrap();+ }+ let retained = rotated_wal_files(&store);+ assert_eq!(+ retained,+ 2 * store.wal_min_retained_generations().len(),+ "the floor must hold 2 rotated generations per shard through the maintenance \+ path the engine actually runs; files on disk: {:?}",+ walk_wal_files(&store.wal_dir())+ );++ // And it is a floor, not a lease: it holds the same two however many+ // more passes run, so a stalled consumer cannot grow the WAL without+ // bound.+ for _ in 0..3 {+ store.force_wal_maintenance().unwrap();+ }+ assert_eq!(+ rotated_wal_files(&store),+ retained,+ "a floor must not grow with the number of maintenance passes"+ );++ // Lowering it back to 0 releases them on the next pass — the knob is+ // live in both directions.+ store.set_wal_min_retained_generations(0);+ store.force_wal_maintenance().unwrap();+ assert_eq!(+ rotated_wal_files(&store),+ 0,+ "clearing the floor must release the held generations, files: {:?}",+ walk_wal_files(&store.wal_dir())+ );+ }+/// RC4 W2 #14 regression — a plain, never-re-indexed DELETE must not/// pin its WAL shard forever. Pre-fix: the delete's tombstone lived/// only in RAM + WAL, `sweep_pending_deletes` never unpinned it, anddiff --git a/engine/crates/xerj-storage/src/wal.rs b/engine/crates/xerj-storage/src/wal.rsindex 1df629504..c4adc1636 100644--- a/engine/crates/xerj-storage/src/wal.rs+++ b/engine/crates/xerj-storage/src/wal.rs@@ -342,9 +342,15 @@ pub struct WalWriter {/// append first re-attempts the fresh-generation heal so the writer/// self-recovers once space frees up.poisoned: bool,- /// Rotated generations [`prune_verified`](Self::prune_verified) must keep- /// even when every entry in them is proven durable — the retention floor- /// a WAL consumer (the #320 tap) needs to survive a target outage.+ /// Rotated generations a prune pass must keep even when every entry in+ /// them is proven durable — the retention floor a WAL consumer (the #320+ /// tap) needs to survive a target outage.+ ///+ /// Read by **both** prune paths: [`prune_verified`](Self::prune_verified)+ /// and — through [`min_retained_generations`](Self::min_retained_generations)+ /// — `IndexStore::wal_maintain_all_verified`, which is the one the engine+ /// actually runs. Enforcing it in only one of them is how this knob+ /// managed to be fully tested and still have no effect on a live node.////// `0` (the default) is the historical behaviour: retention never waits/// for anyone. Anything above it is a *bounded* floor, not a lease: the@@ -476,16 +482,30 @@ impl WalWriter {})}- /// Set the retention floor: rotated generations `prune_verified` keeps- /// even when it can prove every entry in them durable.+ /// Set the retention floor: rotated generations a prune pass keeps even+ /// when it can prove every entry in them durable.+ ///+ /// Seeded by [`crate::index_store::IndexStore::open`] from+ /// `wal_tap.min_retained_generations`, and re-set **live** by+ /// [`IndexStore::set_wal_min_retained_generations`] when an operator+ /// changes it through `PUT /_xerj/wal_tap`.///- /// Set once by [`crate::index_store::IndexStore::open`] from- /// `wal_tap.min_retained_generations`, right after the writer is placed- /// in its `Arc`.+ /// [`IndexStore::set_wal_min_retained_generations`]:+ /// crate::index_store::IndexStore::set_wal_min_retained_generationspub fn set_min_retained_generations(&mut self, n: u64) {self.min_retained_generations = n;}+ /// The retention floor currently in force on this shard.+ ///+ /// Exposed because the floor has to be enforced by the prune pass the+ /// engine actually runs (`IndexStore::wal_maintain_all_verified`), not+ /// only by [`prune_verified`](Self::prune_verified) — which no production+ /// path calls.+ pub fn min_retained_generations(&self) -> u64 {+ self.min_retained_generations+ }+/// Install (or clear) this shard's registration with the batched-fsync/// scheduler. Set once, right after the writer is placed in its `Arc`/// by [`crate::index_store::IndexStore::open`].