Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(wal-tap): make the retention floor real, and stop calling a redelivery a stall (#320)

xerj-orgad82009e4ad6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index baca8a68b..08ea34075 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -87,13 +87,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`wal_tap.min_retained_generations` (default `0`, unchanged behaviour) buys
**bounded** slack: it is a floor, not an Elasticsearch-style retention
lease, so it holds at most `n × storage.wal_max_size_mb` per WAL shard
- however far behind any consumer is.
+ however far behind any consumer is. The floor is enforced inside
+ `IndexStore::wal_maintain_all_verified` — the prune pass the engine actually
+ runs — using the same arithmetic and the same "inside the deletion pass, not
+ in its callers" placement as Lucene's
+ `KeepLastNCommitsDeletionPolicy.onCommit`
+ (`lucene/core/src/java/org/apache/lucene/index/KeepLastNCommitsDeletionPolicy.java:51-58`,
+ Apache-2.0). At the default `0` the loop is unchanged: no extra syscall, no
+ behaviour change.
+ - **The retention floor is live, and the API no longer promises otherwise.**
+ `min_retained_generations` is the one `[wal_tap]` setting that does not live
+ in the tap — it lives in every open index's `WalWriter`, seeded at open from
+ `Engine.config`, an `Arc<Config>` written once at boot and never mutated.
+ `PUT /_xerj/wal_tap` therefore pushes it straight onto the live writers
+ (`Engine::apply_wal_retention_floor`, after Lucene's `LiveIndexWriterConfig`
+ and `IndexFileDeleter.revisitPolicy`,
+ `lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java:39-126`
+ and `IndexFileDeleter.java:516-543`) and answers with
+ `retention_floor_applied_to_indices` — the number of writers it reached —
+ instead of a warning. `DELETE /_xerj/wal_tap` releases it the same way, and
+ `Engine::new` folds the tap's effective configuration into the boot `Config`
+ so a store opened after a restart is seeded with the persisted floor rather
+ than the file's. Verified over HTTP end to end
+ (`xerj-api/tests/wal_tap_retention_floor_is_applied.rs`): after
+ `PUT {"min_retained_generations": 3}` an index opened before the call reads
+ `3` on all 16 WAL shards, an index created after it opens with `3`, and both
+ still read `3` after a restart whose config file still says `0`.
- A configuration set through `PUT /_xerj/wal_tap` is persisted next to the
cursors and re-applied over the config file on the next boot, so "runtime
config, no restart" does not quietly mean "and gone after one".
- `DELETE /_xerj/wal_tap` drops the overlay and reverts to the file. The state
- file holds `target_auth`, so it is written `0600` through the same writer
- the API-key store uses.
+ `DELETE /_xerj/wal_tap` drops the overlay and reverts to the file — to the
+ value the tap kept verbatim at construction, not to a re-read of
+ `Engine.config`. The state file holds `target_auth`, so it is written `0600`
+ through the same writer the API-key store uses.
+ - The unapplied-batch health signal counts **polls**, not `_bulk` requests.
+ How many chunks a poll splits into is `max_batch_bytes` arithmetic: at
+ `max_batch_bytes = 1` one legitimate at-least-once redelivery of four
+ documents produced four all-conflict `_bulk` responses, tripped the
+ three-in-a-row threshold on its own, and emitted `healthy: false` with
+ `last_error: "… nothing is being replicated"` about a poll in which every
+ document was already at the target. The field is now
+ `consecutive_unapplied_polls` and rises by at most one per poll; three
+ unapplied polls in a row still reports the stall.
+ - `wal_tap.max_retry_backoff_secs` is range-checked (`1..=86400`) by `PUT`
+ the way `poll_interval_ms` already was, and every multiply in the backoff
+ computation saturates. `Config::validate` does not bound the field, so
+ `xerj.toml` can put any `u64` there, and `secs.max(1) * 1000` overflowed
+ above ~1.8e16: a panic under `cargo test`'s overflow-checks that killed the
+ spawned tap task silently, and under a release profile a *wrapped* cap that
+ can be arbitrarily small — the value `18_446_744_073_709_552` wrapped to a
+ 384 ms ceiling, i.e. a retry storm aimed at an already-failing target.
- XERJ's own `.xerj*` system indices are never shipped, whatever the allowlist
says, and a wildcard never expands to a hidden index.
- There is no backfill. A new index is shipped whole; an established one ships
diff --git a/engine/crates/xerj-api/src/wal_tap_api.rs b/engine/crates/xerj-api/src/wal_tap_api.rs
index 0eb38643b..7f8fb2644 100644
--- a/engine/crates/xerj-api/src/wal_tap_api.rs
+++ b/engine/crates/xerj-api/src/wal_tap_api.rs
@@ -184,8 +184,16 @@ pub async fn put_wal_tap(
config.request_timeout_secs = v;
}
if let Some(v) = patch.max_retry_backoff_secs {
- if v == 0 {
- return bad_request("wal_tap.max_retry_backoff_secs must be at least 1");
+ // Range-checked like `poll_interval_ms` above, not merely `!= 0`.
+ // `WalTap::arm_backoff` now saturates so no value here can overflow
+ // it, but a cap of "several thousand years" is not a backoff — it is a
+ // tap that stops retrying after its first failed send and reports
+ // nothing beyond `last_error`. One day is already far past any
+ // recovery window a `_bulk` target has.
+ if !(1..=86_400).contains(&v) {
+ return bad_request(
+ "wal_tap.max_retry_backoff_secs must be between 1 and 86400 (one day)",
+ );
}
config.max_retry_backoff_secs = v;
}
@@ -209,20 +217,26 @@ pub async fn put_wal_tap(
}
let body = config_body(&config);
- // `min_retained_generations` reaches `WalWriter` through
- // `IndexStoreConfig` when a store is opened, so a runtime change to it
- // applies to indices opened after this point (and to every index after a
- // restart, which this patch now survives). Said plainly in the response
- // rather than left for an operator to discover.
- let retention_takes_effect_on_restart =
- patch.min_retained_generations.is_some() && !state.engine.index_name_list().is_empty();
state.engine.wal_tap.set_config(config);
+ // `min_retained_generations` is the one setting here that does not live in
+ // the tap — it lives in every open index's `WalWriter`, and those were
+ // seeded from `Engine.config`, an `Arc<Config>` written once at boot and
+ // never mutated. Push it onto the live writers NOW, before answering.
+ //
+ // The previous shape acknowledged the value and attached a warning saying
+ // it would take effect after a restart. It did not take effect after a
+ // restart: `store_config_from` read the boot file, not the persisted
+ // runtime config. An operator raising the floor to survive a target
+ // outage, seeing `200`, restarting, and still losing WAL entries had been
+ // told a falsehood by the API — on the one knob whose entire purpose is
+ // preventing that loss.
+ let applied_to = state.engine.apply_wal_retention_floor();
let mut resp = json!({"acknowledged": true, "wal_tap": body, "persisted": true});
- if retention_takes_effect_on_restart {
- resp["warning"] = json!(
- "min_retained_generations is read when an index store is opened: already-open \
- indices keep the previous retention floor until this node restarts"
- );
+ if patch.min_retained_generations.is_some() {
+ // Not a warning — a receipt. It names the number of live WAL writers
+ // the new floor reached, which is the only claim here that an operator
+ // cannot verify from the echoed config alone.
+ resp["retention_floor_applied_to_indices"] = json!(applied_to);
}
Json(resp).into_response()
}
@@ -236,11 +250,17 @@ pub async fn put_wal_tap(
/// deleting a file in the data directory. Modelled on `_cluster/settings`,
/// where writing `null` clears a persistent setting.
pub async fn delete_wal_tap(State(state): State<AppState>) -> Response {
- // The engine's boot `Config` is the file's value: the tap took a clone of
- // it at construction and only then layered the persisted runtime config
- // on top, so this is genuinely "what xerj.toml says".
- let file_config = state.engine.config().wal_tap.clone();
+ // Taken from the tap, which kept `xerj.toml`'s value verbatim at
+ // construction. It used to be read back off `state.engine.config()`, which
+ // was only correct while nothing else wrote to that field —
+ // `Engine::new` now folds the tap's *effective* config into the boot
+ // `Config` so index stores open with the retention floor actually in
+ // force, and reading it back here would have reverted a persisted runtime
+ // config to itself.
+ let file_config = state.engine.wal_tap.file_config();
state.engine.wal_tap.clear_runtime_config(file_config);
+ // The file's floor is now the live one; the open indices have to be told.
+ state.engine.apply_wal_retention_floor();
Json(json!({
"acknowledged": true,
"wal_tap": config_body(&state.engine.wal_tap.config()),
diff --git a/engine/crates/xerj-api/tests/wal_tap_retention_floor_is_applied.rs b/engine/crates/xerj-api/tests/wal_tap_retention_floor_is_applied.rs
new file mode 100644
index 000000000..960020496
--- /dev/null
+++ b/engine/crates/xerj-api/tests/wal_tap_retention_floor_is_applied.rs
@@ -0,0 +1,231 @@
+//! Issue #320 — `PUT /_xerj/wal_tap` must not acknowledge a retention floor
+//! it does not apply.
+//!
+//! `wal_tap.min_retained_generations` is the knob whose entire purpose is
+//! keeping WAL entries alive while a replication target is down. It reached
+//! `WalWriter` through one path only: `store_config_from`, which reads
+//! `Engine.config` — an `Arc<Config>` written once at boot from `xerj.toml`
+//! and never mutated. `WalTap::set_config` wrote the tap's own `RwLock` and
+//! the state file and nothing else.
+//!
+//! So a runtime `PUT` reached no writer at all: not the indices already open,
+//! and — because the boot `Config` is the *file's* value, not the persisted
+//! runtime overlay — not after a restart either. The endpoint answered
+//! `200 {"acknowledged":true,"persisted":true,...}` plus a `warning` saying it
+//! would take effect on restart. An operator who raised the floor to survive a
+//! target outage, saw the `200`, restarted, and still lost WAL entries had been
+//! told a falsehood by the API.
+//!
+//! These tests drive the whole chain over HTTP and then read the floor back off
+//! the live `WalWriter`s.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use serde_json::{json, Value};
+use tower::ServiceExt;
+
+async fn request(app: &axum::Router, method: &str, path: &str, body: Value) -> (StatusCode, Value) {
+ let response = app
+ .clone()
+ .oneshot(
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .expect("request"),
+ )
+ .await
+ .expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("response body");
+ let body = serde_json::from_slice(&bytes).unwrap_or(Value::Null);
+ (status, body)
+}
+
+fn app_on(dir: &std::path::Path) -> (axum::Router, xerj_api::state::AppState) {
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+ let app = xerj_api::router::build_es_compat_router(state.clone());
+ (app, state)
+}
+
+/// The floor the API reports must be the floor the WAL writers hold — for the
+/// indices that are already open, for indices created afterwards, and across a
+/// restart.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_retention_floor_set_over_http_reaches_the_live_wal_writers() {
+ let dir = tempfile::tempdir().expect("data dir");
+ let (app, state) = app_on(dir.path());
+
+ let (status, _) = request(&app, "PUT", "/already-open", json!({})).await;
+ assert_eq!(status, StatusCode::OK, "index create");
+
+ let before = state
+ .engine
+ .get_index("already-open")
+ .expect("index")
+ .wal_min_retained_generations();
+ assert!(
+ before.iter().all(|&n| n == 0),
+ "the default floor is 0, got {before:?}"
+ );
+
+ let (status, body) = request(
+ &app,
+ "PUT",
+ "/_xerj/wal_tap",
+ json!({"min_retained_generations": 3}),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "PUT /_xerj/wal_tap: {body}");
+ assert_eq!(body["wal_tap"]["min_retained_generations"], 3);
+ assert_eq!(
+ body["retention_floor_applied_to_indices"], 1,
+ "the response must report how many live WAL writers the floor reached: {body}"
+ );
+ assert!(
+ body.get("warning").is_none(),
+ "there is nothing left to warn about — it applied: {body}"
+ );
+
+ // The claim, checked against the writers themselves.
+ let after = state
+ .engine
+ .get_index("already-open")
+ .expect("index")
+ .wal_min_retained_generations();
+ assert!(
+ after.iter().all(|&n| n == 3),
+ "an index open BEFORE the PUT must hold the new floor on every WAL shard, got {after:?}"
+ );
+
+ // An index created after the PUT opens with it too, rather than inheriting
+ // the boot file's 0.
+ let (status, _) = request(&app, "PUT", "/created-after", json!({})).await;
+ assert_eq!(status, StatusCode::OK, "index create");
+ let created_after = state
+ .engine
+ .get_index("created-after")
+ .expect("index")
+ .wal_min_retained_generations();
+ assert!(
+ created_after.iter().all(|&n| n == 3),
+ "an index created AFTER the PUT must open with the floor, got {created_after:?}"
+ );
+
+ // And a restart. The config FILE still says 0 — nothing wrote to
+ // `xerj.toml` — so every part of this depends on the persisted runtime
+ // overlay reaching the storage layer.
+ drop(app);
+ drop(state);
+ let (_app2, state2) = app_on(dir.path());
+ assert_eq!(
+ state2.engine.wal_tap.config().min_retained_generations,
+ 3,
+ "the persisted runtime floor must survive the restart"
+ );
+ // The value every `IndexStore` is SEEDED with at open, before anything
+ // re-applies it. `Engine::new` folds the tap's effective configuration
+ // into the boot `Config` for exactly this: `store_config_from` reads it
+ // inside `IndexStore::open`, which replays the WAL, and a store must not
+ // spend its replay holding a floor the operator has already changed.
+ assert_eq!(
+ state2.engine.config().wal_tap.min_retained_generations,
+ 3,
+ "the boot Config a store opens with must carry the effective floor, not the file's"
+ );
+ let reopened = state2
+ .engine
+ .get_index("already-open")
+ .expect("index")
+ .wal_min_retained_generations();
+ assert!(
+ reopened.iter().all(|&n| n == 3),
+ "after a restart the reopened index must hold the persisted floor, not the config \
+ file's — this is the exact case the old `warning` claimed to cover and did not, \
+ got {reopened:?}"
+ );
+}
+
+/// `DELETE /_xerj/wal_tap` reverts to the config file, and that has to reach
+/// the writers as well — otherwise clearing an expensive floor would leave the
+/// disk cost behind with nothing in the API to show for it.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn deleting_the_runtime_config_releases_the_retention_floor() {
+ let dir = tempfile::tempdir().expect("data dir");
+ let (app, state) = app_on(dir.path());
+ let (status, _) = request(&app, "PUT", "/logs", json!({})).await;
+ assert_eq!(status, StatusCode::OK);
+
+ let (status, _) = request(
+ &app,
+ "PUT",
+ "/_xerj/wal_tap",
+ json!({"min_retained_generations": 5}),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK);
+ assert!(state
+ .engine
+ .get_index("logs")
+ .expect("index")
+ .wal_min_retained_generations()
+ .iter()
+ .all(|&n| n == 5));
+
+ let (status, body) = request(&app, "DELETE", "/_xerj/wal_tap", json!({})).await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["source"], "config_file");
+ assert_eq!(body["wal_tap"]["min_retained_generations"], 0);
+ let released = state
+ .engine
+ .get_index("logs")
+ .expect("index")
+ .wal_min_retained_generations();
+ assert!(
+ released.iter().all(|&n| n == 0),
+ "reverting to the config file must release the floor on the live writers too, \
+ got {released:?}"
+ );
+}
+
+/// `max_retry_backoff_secs` is range-checked like `poll_interval_ms`, not
+/// merely `!= 0`. `WalTap::arm_backoff` saturates so no value can overflow it,
+/// but a multi-millennium cap is a tap that stops retrying, not a backoff.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn an_absurd_retry_cap_is_refused_at_the_api() {
+ let dir = tempfile::tempdir().expect("data dir");
+ let (app, _state) = app_on(dir.path());
+
+ for bad in [0u64, 86_401, u64::MAX] {
+ let (status, body) = request(
+ &app,
+ "PUT",
+ "/_xerj/wal_tap",
+ json!({"max_retry_backoff_secs": bad}),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::BAD_REQUEST,
+ "max_retry_backoff_secs={bad} must be refused, got {status} {body}"
+ );
+ }
+
+ let (status, body) = request(
+ &app,
+ "PUT",
+ "/_xerj/wal_tap",
+ json!({"max_retry_backoff_secs": 86_400}),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "the bound itself is valid: {body}");
+ assert_eq!(body["wal_tap"]["max_retry_backoff_secs"], 86_400);
+}
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 653e27a0e..64911a48e 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -1823,6 +1823,12 @@ pub struct WalTapConfig {
/// It is deliberately **not** an Elasticsearch-style retention lease,
/// which holds generations for as long as a follower is behind and is how
/// a dead follower wedges a leader's disk.
+ ///
+ /// Unlike every other field here, this one does not live in the tap: it
+ /// lives in each index's `WalWriter`. `PUT /_xerj/wal_tap` therefore
+ /// pushes it onto the open writers itself
+ /// (`Engine::apply_wal_retention_floor`) and reports how many it reached,
+ /// rather than acknowledging a value that reaches nothing.
pub min_retained_generations: u64,
}
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index 0e92ec6de..524acaef7 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -702,6 +702,16 @@ impl Engine {
// reading the frozen boot config.
let wal_tap_config = config.wal_tap.clone();
+ // Built HERE, ahead of the struct literal, because `WalTap::new` is
+ // what loads the persisted runtime configuration and it can override
+ // `min_retained_generations` — which `store_config_from` reads out of
+ // this `Config` when every index opens a few lines below. Freezing the
+ // *file's* floor into the `Arc` while the tap ran with a different one
+ // is how a floor set through `PUT /_xerj/wal_tap` survived a restart
+ // in `_stats` and still reached no `WalWriter`.
+ let wal_tap = Arc::new(crate::wal_tap::WalTap::new(&data_dir, wal_tap_config));
+ config.wal_tap = wal_tap.config();
+
// Apply operator-tunable aggregation bucket cap. Stored in a static
// AtomicUsize inside aggs.rs so all per-bucket-allocator hot loops
// can read it with no plumbing through every agg signature.
@@ -755,7 +765,7 @@ impl Engine {
frozen_indices: Arc::new(DashMap::new()),
rollup_jobs: Arc::new(DashMap::new()),
ccr_auto_follow: Arc::new(DashMap::new()),
- wal_tap: Arc::new(crate::wal_tap::WalTap::new(&data_dir, wal_tap_config)),
+ wal_tap,
api_keys: Arc::new(DashMap::new()),
application_privileges: Arc::new(DashMap::new()),
legacy_templates: Arc::new(DashMap::new()),
@@ -828,6 +838,13 @@ impl Engine {
match Index::open(index_name.clone(), &engine.config, &data_dir) {
Ok(idx) => {
info!(name = name_str.as_str(), "opened existing index");
+ // #320 — `engine.config.wal_tap` is already the
+ // tap's effective configuration (see `WalTap::new`
+ // above), so the store opened with the right
+ // floor; this makes that true by construction
+ // rather than by argument, and covers a tap
+ // reconfigured between two boot-loop iterations.
+ idx.set_wal_min_retained_generations(engine.wal_retention_floor());
// Restore the raw ES mapping blob (analyzers, formats,
// dims — full fidelity) BEFORE any ingest/query can run,
// so GET /_mapping and mapping-dependent code paths see
@@ -1240,6 +1257,11 @@ impl Engine {
&self.config,
&self.data_dir,
)?;
+ // #320 — the store was seeded from the boot `Config`; the tap's live
+ // floor is what an operator last asked for. Applied before the index
+ // is published, so no ingest (and therefore no prune) can run against
+ // the wrong floor.
+ idx.set_wal_min_retained_generations(self.wal_retention_floor());
self.indices.insert(name.to_string(), idx);
self.record_index_created_at(name);
info!(name, "index created with custom settings");
@@ -2663,6 +2685,9 @@ impl Engine {
let index_name = IndexName::new(name).map_err(EngineError::Common)?;
match Index::open(index_name, &self.config, &self.data_dir) {
Ok(idx) => {
+ // #320 — a reopened index gets the tap's live retention floor,
+ // not the boot file's.
+ idx.set_wal_min_retained_generations(self.wal_retention_floor());
// The store opened, but the full-fidelity mapping blob is part
// of what makes this index serveable (#202). Putting the index
// back with a silently reduced mapping is exactly the defect
@@ -2832,6 +2857,48 @@ impl Engine {
.collect()
}
+ /// The WAL-consumer retention floor the tap is currently configured with.
+ ///
+ /// Read from the tap, never from `self.config`: the tap's copy is the one
+ /// an operator can change at runtime, and after a restart it is the one
+ /// that was persisted. `self.config` is the boot file, frozen in an `Arc`.
+ fn wal_retention_floor(&self) -> u64 {
+ self.wal_tap.config().min_retained_generations
+ }
+
+ /// Push the tap's live retention floor onto every open index's WAL shards.
+ ///
+ /// Returns the number of indices it reached, so the `PUT /_xerj/wal_tap`
+ /// handler can report what actually happened instead of guessing.
+ ///
+ /// #320. A floor set through the API previously reached nothing: it lived
+ /// in the tap's own `RwLock` and in the state file, while every
+ /// `WalWriter` had been seeded from `Engine.config` — an `Arc<Config>`
+ /// written once at boot from `xerj.toml` and never mutated. The API
+ /// answered `200` with the new value and a warning that it would apply
+ /// after a restart; it did not apply after a restart either. This is the
+ /// half that makes it true for indices that are already open.
+ ///
+ /// Re-applying a policy to already-live state on change, rather than only
+ /// at construction, is Lucene's `IndexFileDeleter.revisitPolicy`
+ /// (lucene/core/src/java/org/apache/lucene/index/IndexFileDeleter.java:516-543,
+ /// Apache-2.0), which re-runs `policy.onCommit(commits)` over the commits
+ /// that already exist rather than waiting for the next one.
+ pub fn apply_wal_retention_floor(&self) -> usize {
+ let floor = self.wal_retention_floor();
+ let mut applied = 0usize;
+ for entry in self.indices.iter() {
+ entry.value().set_wal_min_retained_generations(floor);
+ applied += 1;
+ }
+ tracing::debug!(
+ floor,
+ indices = applied,
+ "WAL tap: retention floor applied to open indices"
+ );
+ applied
+ }
+
/// Sum the internal query-result cache hit/miss counters across every open
/// index (RC4-W4 item 4). Returns `(hits, misses)`. Reconciled into the
/// `xerj_query_cache_{hits,misses}` gauges by the `/v1/metrics` handler at
@@ -3592,6 +3659,9 @@ impl Engine {
// Reopen the index.
match Index::open(index_name, &self.config, &self.data_dir) {
Ok(idx) => {
+ // #320 — a restored index gets the tap's live retention
+ // floor, not the boot file's.
+ idx.set_wal_min_retained_generations(self.wal_retention_floor());
// Snapshot dirs carry es_mapping.json — reload it so the
// restored index serves the same mapping it was saved with.
// A corrupt blob in the snapshot fails the restore of that
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 85927eaca..1afc4b49a 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -5747,6 +5747,22 @@ pub struct Index {
}
impl Index {
+ /// Apply a WAL-consumer retention floor to this index's WAL shards, live.
+ ///
+ /// #320. Called by [`Engine`](crate::engine::Engine) immediately after
+ /// every open/create, and again on every index when an operator changes
+ /// `wal_tap.min_retained_generations` through `PUT /_xerj/wal_tap`. See
+ /// [`xerj_storage::index_store::IndexStore::set_wal_min_retained_generations`]
+ /// for why the open-time config value alone was not enough.
+ pub fn set_wal_min_retained_generations(&self, n: u64) {
+ self.store.set_wal_min_retained_generations(n);
+ }
+
+ /// The retention floor in force on each of this index's WAL shards.
+ pub fn wal_min_retained_generations(&self) -> Vec<u64> {
+ self.store.wal_min_retained_generations()
+ }
+
fn begin_hnsw_publication(&self) -> HnswPublicationGuard {
HnswPublicationGuard::begin(
&self.hnsw_publications_in_flight,
@@ -28174,13 +28190,18 @@ fn store_config_from(config: &Config, wal_shards_override: Option<usize>) -> Ind
schema_version: 1,
storage_mode: xerj_storage::StorageMode::Local,
num_wal_shards: wal_shards_override.unwrap_or(config.engine.ingest_shards),
- // #320 — the retention floor a WAL consumer needs. Read here, from
- // the boot config, because it is consumed once when a `WalWriter` is
- // opened; a runtime `PUT /_xerj/wal_tap` that changes it therefore
- // applies to stores opened after the change (and to every store after
- // a restart, which the persisted runtime config now survives). The
- // `PUT` handler says so in its response rather than leaving it to be
- // discovered.
+ // #320 — the retention floor a WAL consumer needs, as the store's
+ // *seed* value.
+ //
+ // `Engine::new` overwrites `config.wal_tap` with the tap's effective
+ // configuration before freezing the `Config` into its `Arc`, so this
+ // is the persisted runtime value when one exists and the file's value
+ // otherwise — i.e. correct from byte zero for every store opened at
+ // boot. A runtime `PUT /_xerj/wal_tap` cannot reach an `Arc<Config>`
+ // at all, so it is applied by
+ // `Engine::apply_wal_retention_floor` straight onto the live
+ // `WalWriter`s instead. Both halves are required: this one alone
+ // acknowledged a floor that never reached a writer.
wal_min_retained_generations: config.wal_tap.min_retained_generations,
}
}
diff --git a/engine/crates/xerj-engine/src/wal_tap.rs b/engine/crates/xerj-engine/src/wal_tap.rs
index c82bcbb25..74fbe7d0a 100644
--- a/engine/crates/xerj-engine/src/wal_tap.rs
+++ b/engine/crates/xerj-engine/src/wal_tap.rs
@@ -64,10 +64,13 @@
//! version for every `_id` because a second writer got there first, or one
//! rejecting every document with a `mapper_parsing_exception` — is a real
//! state and used to be indistinguishable from a healthy tap.
-//! `consecutive_unapplied_batches`, `last_item_rejection` and
+//! `consecutive_unapplied_polls`, `last_item_rejection` and
//! [`IndexTapStats::healthy`] exist so it is not. The counter is deliberately
//! blind to *why* nothing applied: conflict and rejection are opposite
-//! diagnoses but the same fact, and it is the fact that `healthy` reports.
+//! diagnoses but the same fact, and it is the fact that `healthy` reports. It
+//! counts **polls**, not `_bulk` requests: how many chunks a poll splits into
+//! is `max_batch_bytes` arithmetic, and charging each of them separately made
+//! one ordinary redelivery look like a dead replication stream.
//!
//! **3. Backpressure.** Bounded by construction: one in-flight request per
//! index, batches capped by `max_batch_docs` and `max_batch_bytes`, and
@@ -84,10 +87,11 @@
//! Seed the target with snapshot/restore if it needs the existing documents.
//!
//! **4b. What falling behind costs.** WAL retention is *not* coupled to the
-//! tap, deliberately: `prune_verified` deletes a rotated generation as soon as
-//! its entries are durable in a segment, and making that wait on a remote
-//! target is how a slow consumer turns into a disk-full incident. So a tap
-//! that falls far enough behind loses entries — and always finds out. The
+//! tap, deliberately: `IndexStore::wal_maintain_all_verified` deletes a
+//! rotated generation as soon as its entries are durable in a segment, and
+//! making that wait on a remote target is how a slow consumer turns into a
+//! disk-full incident. So a tap that falls far enough behind loses entries —
+//! and always finds out. The
//! cursor carries the generation it sits in, a pruned-out-from-under-us
//! generation is detected exactly ([`WalShardTail::gap`]), and each occurrence
//! logs at `warn` and increments `gaps` in `GET /_xerj/wal_tap/_stats`. The
@@ -97,13 +101,29 @@
//! `storage.flush_interval_secs = 30` the WAL window is about one flush — so a
//! minute of target downtime on a hot index loses data with nothing but a log
//! line to offer. `wal_tap.min_retained_generations` is the middle path: a
-//! **bounded** floor under `prune_verified` that holds the newest N rotated
-//! generations whatever any consumer is doing. Deliberately not a retention
-//! lease (Elasticsearch's `ReplicationTracker.addRetentionLease`, Lucene's
+//! **bounded** floor inside that maintenance pass, holding the newest N
+//! rotated generations whatever any consumer is doing. Same arithmetic and
+//! same placement — inside the deletion pass, not in its callers — as Lucene's
+//! `KeepLastNCommitsDeletionPolicy.onCommit`
+//! (`lucene/core/src/java/org/apache/lucene/index/KeepLastNCommitsDeletionPolicy.java:51-58`,
+//! Apache-2.0). Deliberately **not** a retention lease (Elasticsearch's
+//! `ReplicationTracker.addRetentionLease`, Lucene's
//! `SnapshotDeletionPolicy.snapshot`): a lease is held until the consumer
//! releases it, which is exactly how a dead follower wedges a leader's disk.
//! A floor costs `n * wal_max_size_mb` per shard and not one byte more.
//!
+//! The floor is **live**: `PUT /_xerj/wal_tap` pushes it straight onto the
+//! open `WalWriter`s (`Engine::apply_wal_retention_floor`), because the value
+//! that reaches a store at open time comes from `Engine.config`, an
+//! `Arc<Config>` written once at boot and never mutated. Setting it through
+//! the API used to reach no writer at all — not the open ones, and not after a
+//! restart either — on the one knob whose whole purpose is preventing loss
+//! during a target outage. Live reconfiguration of an already-open writer is
+//! Lucene's `LiveIndexWriterConfig`
+//! (`lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java:39-126`),
+//! and re-applying a policy to state that already exists rather than only at
+//! construction is its `IndexFileDeleter.revisitPolicy` (`:516-543`).
+//!
//! **4c. Where the configuration lives.** In the state file next to the
//! cursors, not only in memory. `PUT /_xerj/wal_tap` is durable and is
//! re-applied over the config file on the next boot; `DELETE /_xerj/wal_tap`
@@ -268,8 +288,18 @@ pub struct IndexTapStats {
/// instead is what let a target that rejected every single action report
/// `lag_seq: 0` while `docs_shipped` climbed.
pub last_shipped_seq: u64,
- /// Consecutive `_bulk` responses in which the target accepted **nothing**
- /// — every action inside came back with an error.
+ /// Consecutive **polls** in which the target accepted **nothing** — every
+ /// action of every `_bulk` the poll sent came back with an error.
+ ///
+ /// Counted per poll, not per `_bulk`. A poll splits its read into as many
+ /// chunks as `max_batch_bytes` dictates, and those chunks are one logical
+ /// delivery: at `max_batch_bytes = 1` a single legitimate at-least-once
+ /// redelivery of four documents produced four all-conflict `_bulk`
+ /// responses and tripped [`UNAPPLIED_UNHEALTHY_AFTER`] on its own, with a
+ /// `last_error` claiming "nothing is being replicated" about a poll in
+ /// which every document was already at the target. The chunk count is a
+ /// transport detail; the operator-visible unit of "did this tap make
+ /// progress" is the poll.
///
/// Deliberately blind to *why*. A version conflict and a
/// `mapper_parsing_exception` are opposite diagnoses but the same
@@ -279,11 +309,11 @@ pub struct IndexTapStats {
/// read `healthy: true` with `docs_shipped: 0` — the exact class of lie
/// the per-item accounting exists to remove.
///
- /// One such batch is expected and benign: it is how external versioning
+ /// One such poll is expected and benign: it is how external versioning
/// absorbs an at-least-once redelivery. A *sustained run* is not — it means
/// the tap is a no-op and the two sides are diverging. See
/// [`IndexTapStats::healthy`].
- pub consecutive_unapplied_batches: u32,
+ pub consecutive_unapplied_polls: u32,
/// Error `type` of the most recent item the target rejected for a reason
/// other than a version conflict, e.g. `mapper_parsing_exception`.
///
@@ -293,13 +323,17 @@ pub struct IndexTapStats {
pub last_item_rejection: Option<String>,
}
-/// How many batches in a row that applied nothing stop counting as "a
+/// How many **polls** in a row that applied nothing stop counting as "a
/// redelivery landed" and start counting as "this target is not taking our
/// writes".
///
-/// A redelivery after a failed send re-ships exactly the chunks that did not
-/// land, so it clears on the next batch that contains anything new. Three in a
-/// row cannot be that.
+/// Polls, not `_bulk` requests. A poll that applies nothing still advances the
+/// cursor past the entries it read (rejected items are not retried — see
+/// [`WalTap::send_bulk`]), so the *next* poll reads new entries and a healthy
+/// stream clears the counter immediately. A redelivery — a rewound cursor, or
+/// a send that failed part-way through and re-ships its earlier chunks on the
+/// next poll — is at most one such poll, however many chunks it took.
+/// Three in a row cannot be that.
const UNAPPLIED_UNHEALTHY_AFTER: u32 = 3;
impl IndexTapStats {
@@ -310,7 +344,7 @@ impl IndexTapStats {
/// every request while rejecting every action inside it — for any reason,
/// conflict or otherwise.
pub fn healthy(&self) -> bool {
- self.last_error.is_none() && self.consecutive_unapplied_batches < UNAPPLIED_UNHEALTHY_AFTER
+ self.last_error.is_none() && self.consecutive_unapplied_polls < UNAPPLIED_UNHEALTHY_AFTER
}
}
@@ -322,6 +356,16 @@ impl IndexTapStats {
/// surface can read stats and edit the allowlist without a restart.
pub struct WalTap {
config: RwLock<WalTapConfig>,
+ /// Exactly what `xerj.toml` said, kept verbatim so `DELETE
+ /// /_xerj/wal_tap` has something true to revert to.
+ ///
+ /// It used to be read back off `Engine.config`, which was safe only while
+ /// nothing else ever wrote to that field. `Engine::new` now folds the
+ /// tap's *effective* configuration into the boot `Config` (so index stores
+ /// open with the retention floor actually in force), which would have made
+ /// `DELETE` revert a persisted runtime config to… itself. Owning the file
+ /// value here removes the coupling instead of documenting around it.
+ file_config: WalTapConfig,
state: RwLock<PersistedState>,
stats: RwLock<BTreeMap<String, IndexTapStats>>,
state_path: PathBuf,
@@ -383,6 +427,8 @@ impl WalTap {
Err(_) => PersistedState::default(),
};
+ let file_config = config.clone();
+
// A configuration set through the API outlives the process that took
// it. Without this a restart reverted to the file config — usually
// `enabled = false` — and the tap went quiet while its cursors froze
@@ -408,6 +454,7 @@ impl WalTap {
Self {
config: RwLock::new(config),
+ file_config,
state: RwLock::new(state),
stats: RwLock::new(BTreeMap::new()),
state_path,
@@ -425,6 +472,12 @@ impl WalTap {
self.config.read().clone()
}
+ /// What `xerj.toml` said at boot — the value `DELETE /_xerj/wal_tap`
+ /// reverts to.
+ pub fn file_config(&self) -> WalTapConfig {
+ self.file_config.clone()
+ }
+
/// Replace the live configuration **durably**.
///
/// Takes effect on the next poll — no restart — and cursors are preserved
@@ -603,20 +656,34 @@ impl WalTap {
/// Capped exponential backoff with jitter, keyed off the worst
/// consecutive-failure count across indices.
///
+ /// **Every multiply here saturates**, including the two that used to be
+ /// plain `*`. `max_retry_backoff_secs` and `poll_interval_ms` reach this
+ /// function from two places, and only one of them is range-checked:
+ /// `PUT /_xerj/wal_tap` bounds both, but `Config::validate` bounds
+ /// neither, so `xerj.toml` can put any `u64` in either field. At
+ /// `max_retry_backoff_secs > u64::MAX / 1000` (~1.8e16) the old
+ /// `secs.max(1) * 1000` overflowed — a panic under `cargo test`'s
+ /// overflow-checks that killed the spawned tap task with nothing in
+ /// `_stats` to say why, and in a release profile a *wrapped* cap that can
+ /// be arbitrarily small: the concrete value 18_446_744_073_709_552 wraps
+ /// to a 384 ms ceiling, i.e. a retry storm aimed at a target that is
+ /// already failing. `uncapped` on the line above already saturated for
+ /// exactly this reason.
+ ///
/// Approach (not code) from Elasticsearch's `ShardFollowNodeTask.computeDelay`.
fn arm_backoff(&self, config: &WalTapConfig, failures: u32) {
let base = config.poll_interval_ms.max(50);
let shift = failures.saturating_sub(1).min(20);
let uncapped = base.saturating_mul(1u64 << shift);
- let cap = config.max_retry_backoff_secs.max(1) * 1000;
+ let cap = config.max_retry_backoff_secs.max(1).saturating_mul(1000);
let delay = uncapped.min(cap);
// Deterministic, dependency-free jitter in [50%, 100%] of the delay:
// enough to stop N indices from re-hammering a recovering target in
// lockstep, without pulling in an RNG.
let jitter = self.now_ms() % 50 + 50;
- let delay = delay * jitter / 100;
+ let delay = delay.saturating_mul(jitter) / 100;
self.retry_not_before_ms
- .store(self.now_ms() + delay, Ordering::Relaxed);
+ .store(self.now_ms().saturating_add(delay), Ordering::Relaxed);
}
fn clear_backoff(&self) {
@@ -1037,7 +1104,11 @@ impl WalTap {
let mut item_failures = 0u64;
let mut version_conflicts = 0u64;
let mut batches = 0u64;
- let mut unapplied_batches = 0u32;
+ // Chunks in THIS poll that applied nothing. Kept separate from the
+ // health counter on purpose: that counter is per poll (see
+ // `UNAPPLIED_UNHEALTHY_AFTER`), and adding this into it is what made
+ // one legitimate multi-chunk redelivery report the tap unhealthy.
+ let mut unapplied_chunks = 0u32;
let mut rejection_kind: Option<String> = None;
for chunk in chunks {
@@ -1071,7 +1142,7 @@ impl WalTap {
// let an all-`mapper_parsing_exception` response read as a
// healthy zero-lag tap with `docs_shipped: 0`.
if accepted_here == 0 && !actions.is_empty() {
- unapplied_batches += 1;
+ unapplied_chunks += 1;
}
if rejection.is_some() {
rejection_kind = rejection;
@@ -1117,17 +1188,24 @@ impl WalTap {
if let Some(kind) = &rejection_kind {
s.last_item_rejection = Some(kind.clone());
}
- // A run of batches that applied nothing is not "replication
- // working". Only a batch that landed something clears it.
- if unapplied_batches > 0 && docs == 0 && deletes == 0 {
- s.consecutive_unapplied_batches = s
- .consecutive_unapplied_batches
- .saturating_add(unapplied_batches);
+ // A run of POLLS that applied nothing is not "replication
+ // working". Only a poll that landed something clears it.
+ //
+ // `+= 1`, not `+= unapplied_chunks`. How many `_bulk` requests a
+ // poll needed is a function of `max_batch_bytes`, not of whether
+ // replication is progressing: at `max_batch_bytes = 1` the old
+ // shape turned ONE legitimate at-least-once redelivery of four
+ // documents into `consecutive_unapplied = 4`, tripping the
+ // three-in-a-row threshold on the documented-benign path and
+ // emitting a `last_error` saying "nothing is being replicated"
+ // about a poll in which every document was already at the target.
+ if unapplied_chunks > 0 && docs == 0 && deletes == 0 {
+ s.consecutive_unapplied_polls = s.consecutive_unapplied_polls.saturating_add(1);
} else {
- s.consecutive_unapplied_batches = 0;
+ s.consecutive_unapplied_polls = 0;
}
- if s.consecutive_unapplied_batches >= UNAPPLIED_UNHEALTHY_AFTER {
- let run = s.consecutive_unapplied_batches;
+ if s.consecutive_unapplied_polls >= UNAPPLIED_UNHEALTHY_AFTER {
+ let run = s.consecutive_unapplied_polls;
// Name the cause the target gave *this poll*, not whatever it
// last gave: a conflict and a mapping error need opposite
// responses from the operator, and a stale reason would send
@@ -1138,13 +1216,13 @@ impl WalTap {
.or(s.last_item_rejection.as_deref())
.unwrap_or("an unnamed error");
format!(
- "target accepted nothing in the last {run} batches — it rejected \
+ "target accepted nothing in the last {run} polls — it rejected \
every action, most recently with {kind}. Nothing is being \
replicated; these documents are being dropped, not retried."
)
} else {
format!(
- "target rejected every action of the last {run} batches with \
+ "target rejected every action of the last {run} polls with \
version_conflict_engine_exception — it holds a higher version for \
every document this tap sends, so nothing is being replicated"
)
@@ -1403,4 +1481,36 @@ mod tests {
"backoff must still be substantial, got {delay}ms"
);
}
+
+ /// `max_retry_backoff_secs` reaches [`WalTap::arm_backoff`] from the config
+ /// file as well as from `PUT /_xerj/wal_tap`, and `Config::validate` does
+ /// not range-check it — so any `u64` in `xerj.toml` gets here.
+ ///
+ /// The value below is chosen so the OLD `secs.max(1) * 1000` fails in both
+ /// profiles rather than only one: `18_446_744_073_709_552 * 1000` is
+ /// `18_446_744_073_709_552_000`, which is `2^64 + 384`, so it panics under
+ /// `cargo test`'s overflow-checks (killing the spawned tap task with
+ /// nothing in `_stats` to say why) and **wraps to a 384 ms cap** under
+ /// `--profile ci-test` — a retry storm aimed at a target that is already
+ /// failing. Saturating leaves the exponential backoff itself in charge.
+ #[test]
+ fn an_unbounded_retry_cap_saturates_instead_of_wrapping() {
+ let dir = tempfile::tempdir().unwrap();
+ let config = WalTapConfig {
+ poll_interval_ms: 500,
+ max_retry_backoff_secs: 18_446_744_073_709_552,
+ ..Default::default()
+ };
+ let tap = WalTap::new(dir.path(), config.clone());
+ // failures = 3 → shift 2 → uncapped = 500 * 4 = 2_000 ms, then jitter
+ // in [50%, 100%].
+ tap.arm_backoff(&config, 3);
+ let armed = tap.retry_not_before_ms.load(Ordering::Relaxed);
+ let delay = armed.saturating_sub(tap.now_ms());
+ assert!(
+ (900..=2_000).contains(&delay),
+ "a cap too large to express in ms must not become a SMALL cap; the wrapped \
+ value gives at most 384ms. got {delay}ms"
+ );
+ }
}
diff --git a/engine/crates/xerj-engine/tests/wal_tap_push.rs b/engine/crates/xerj-engine/tests/wal_tap_push.rs
index 5a3b783f0..6ab1f9bd2 100644
--- a/engine/crates/xerj-engine/tests/wal_tap_push.rs
+++ b/engine/crates/xerj-engine/tests/wal_tap_push.rs
@@ -908,6 +908,99 @@ async fn redelivery_converges_at_a_target_that_honours_external_versioning() {
);
}
+/// The health signal counts **polls** that applied nothing, not `_bulk`
+/// requests.
+///
+/// How many chunks a poll splits into is `max_batch_bytes` arithmetic, not a
+/// statement about replication. Charging the unapplied counter once per chunk
+/// meant one legitimate at-least-once redelivery — the path the module
+/// documents as expected and benign, and the entire reason every action
+/// carries `version_type: external` — tripped the three-in-a-row threshold on
+/// its own and reported the tap as broken.
+///
+/// Fails before the fix with `healthy = false`,
+/// `consecutive_unapplied_polls = 4`, and
+/// `last_error = "target rejected every action of the last 4 polls with
+/// version_conflict_engine_exception … so nothing is being replicated"` — a
+/// message that is factually wrong, because all four documents are at the
+/// target and this test asserts so.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn a_multi_chunk_redelivery_is_one_unapplied_poll_not_four() {
+ let target = StubTarget::start_with(TargetSemantics::ExternalVersioning).await;
+ let dir = TempDir::new().unwrap();
+ let mut tap = tap_config(&target.url, &["edge-logs"]);
+ // One action per `_bulk`: the smallest legal cap, and the shape any
+ // operator with a low `http.max_content_length` at the target gets.
+ tap.max_batch_bytes = 1;
+ let engine = engine_with_tap(&dir, tap);
+
+ write_docs(&engine, "edge-logs", &["a", "b", "c", "d"]).await;
+ engine.wal_tap.tick(&engine).await;
+ let landed = target.stored_versions();
+ assert_eq!(
+ landed.len(),
+ 4,
+ "the first delivery must land all four documents: {landed:?}"
+ );
+ let first_batches = engine.wal_tap.stats()["edge-logs"].batches_sent;
+ assert!(
+ first_batches >= 4,
+ "the point of this test is a MULTI-chunk poll; got {first_batches} _bulk request(s)"
+ );
+
+ // One rewind, one poll: four all-conflict `_bulk` responses that are
+ // together a single, correct, no-op redelivery.
+ engine.wal_tap.rewind_for_test("edge-logs");
+ engine.wal_tap.tick(&engine).await;
+
+ let stats = engine.wal_tap.stats();
+ let edge = stats.get("edge-logs").expect("stats");
+ assert_eq!(
+ edge.version_conflicts, 4,
+ "every redelivered action must still be reported as a conflict"
+ );
+ assert_eq!(
+ edge.consecutive_unapplied_polls, 1,
+ "one poll that applied nothing is ONE, however many chunks it took"
+ );
+ assert!(
+ edge.healthy(),
+ "a redelivery is the documented-benign path and must not report the tap broken: \
+ last_error = {:?}",
+ edge.last_error
+ );
+ assert_eq!(
+ edge.last_error, None,
+ "…and must not leave an operator-facing message behind either"
+ );
+ assert_eq!(
+ target.stored_versions(),
+ landed,
+ "the target still holds exactly the four documents it already had"
+ );
+
+ // The signal must still fire on a genuinely stuck stream: three polls in a
+ // row that apply nothing is not a redelivery.
+ for _ in 0..2 {
+ engine.wal_tap.rewind_for_test("edge-logs");
+ engine.wal_tap.tick(&engine).await;
+ }
+ let stats = engine.wal_tap.stats();
+ let edge = stats.get("edge-logs").expect("stats");
+ assert_eq!(edge.consecutive_unapplied_polls, 3);
+ assert!(
+ !edge.healthy(),
+ "three unapplied polls in a row is a real stall and must still be reported"
+ );
+ assert!(
+ edge.last_error
+ .as_deref()
+ .is_some_and(|e| e.contains("last 3 polls")),
+ "and the message must count the unit it actually counts: {:?}",
+ edge.last_error
+ );
+}
+
/// A target that holds a higher version for every `_id` — a second writer got
/// there first — turns every action into a rejected no-op. `_stats` used to
/// call that healthy with `lag_seq: 0` and a climbing `docs_shipped`, because
@@ -1084,7 +1177,7 @@ async fn health_recovers_once_the_target_accepts_again() {
edge.last_error
);
assert_eq!(
- edge.consecutive_unapplied_batches, 0,
+ edge.consecutive_unapplied_polls, 0,
"the run counter must reset, not decay"
);
assert!(
diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rs
index d5f494c42..5e5a38b5b 100644
--- a/engine/crates/xerj-storage/src/index_store.rs
+++ b/engine/crates/xerj-storage/src/index_store.rs
@@ -2695,7 +2695,43 @@ impl IndexStore {
}
// Prune pass over all rotated generations, cache-first.
- for gen in wal.rotated_generations()? {
+ //
+ // #320 — the WAL-consumer retention floor is applied HERE, not
+ // only in `WalWriter::prune_verified`. `prune_verified` has no
+ // production caller (grep: `wal.rs` and its own unit tests are the
+ // only hits); this loop is the prune the engine actually runs, so
+ // a floor enforced only there was fully unit-tested and still had
+ // zero effect on a live node.
+ //
+ // `rotated_generations` is sorted ascending, so the newest are the
+ // tail and the prunable prefix is `len - keep`. Same arithmetic,
+ // and the same "enforce it inside the deletion pass rather than in
+ // its callers" placement, as Lucene's
+ // `KeepLastNCommitsDeletionPolicy.onCommit`
+ // (lucene/core/src/java/org/apache/lucene/index/KeepLastNCommitsDeletionPolicy.java:51-58,
+ // Apache-2.0): "The commits list is already sorted from oldest to
+ // newest / for (i = 0; i < size - numCommitsToKeep; i++)
+ // commits.get(i).delete()". Adapted, not copied.
+ //
+ // At the default floor of 0 this is `keep = 0` and the loop is
+ // byte-for-byte the old one: no extra syscall, no behaviour change.
+ let rotated = wal.rotated_generations()?;
+ let keep = (wal.min_retained_generations() as usize).min(rotated.len());
+ let prunable_prefix = rotated.len() - keep;
+ for (pos, gen) in rotated.into_iter().enumerate() {
+ if pos >= prunable_prefix {
+ // Held for a WAL consumer. Deliberately skipped BEFORE the
+ // cache lookup: decoding a generation we are not going to
+ // delete is the O(retained WAL bytes)/tick cost the prune
+ // cache exists to avoid.
+ debug!(
+ gen,
+ shard = ws_idx,
+ keep,
+ "WAL generation retained: consumer retention floor"
+ );
+ continue;
+ }
let mut cache = self.wal_prune_cache.lock().unwrap();
let verdict = cache.entry((ws_idx, gen)).or_insert_with(|| {
// Cache miss: a generation rotated by the size-based
@@ -2762,6 +2798,46 @@ impl IndexStore {
.collect()
}
+ /// Change the WAL-consumer retention floor on every shard of this store,
+ /// **live**.
+ ///
+ /// #320. `IndexStoreConfig.wal_min_retained_generations` is read once, at
+ /// open, from the boot config file — and `Engine.config` is an
+ /// `Arc<Config>` that is never mutated, so without this a floor set
+ /// through `PUT /_xerj/wal_tap` reached no writer at all: not the ones
+ /// already open, and not the ones opened after a restart either. The knob
+ /// exists to stop a target outage turning into data loss, so "acknowledged
+ /// but inert" is the one failure mode it must not have.
+ ///
+ /// Live reconfiguration of an already-open writer is Lucene's
+ /// `LiveIndexWriterConfig`
+ /// (lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java:39-126,
+ /// Apache-2.0): "Holds all the configuration used by IndexWriter with few
+ /// setters for settings that can be changed on an IndexWriter instance
+ /// 'live'." Everything else there is explicitly documented as "Only takes
+ /// effect when IndexWriter is first created"
+ /// (`IndexWriterConfig.setCodec`, :297-305). A setting is one or the
+ /// other, and says which; it is never claimed live and implemented at
+ /// create.
+ pub fn set_wal_min_retained_generations(&self, n: u64) {
+ for shard in &self.wal_shards {
+ shard.lock().unwrap().set_min_retained_generations(n);
+ }
+ }
+
+ /// The retention floor in force on each WAL shard, in shard order.
+ ///
+ /// Instrumentation for the property that
+ /// [`set_wal_min_retained_generations`](Self::set_wal_min_retained_generations)
+ /// exists to provide: it must be assertable that a runtime change reached
+ /// every writer, not just the first.
+ pub fn wal_min_retained_generations(&self) -> Vec<u64> {
+ self.wal_shards
+ .iter()
+ .map(|s| s.lock().unwrap().min_retained_generations())
+ .collect()
+ }
+
/// Unconditionally run verified WAL maintenance across all shards.
/// Bypasses the `WAL_MAINTENANCE_INTERVAL_MS` gate that
/// `finalize_flush_with_publisher` uses on the hot flush path.
@@ -5212,6 +5288,93 @@ mod tests {
}
}
+ /// #320 — the WAL-consumer retention floor must be honoured by the prune
+ /// pass the ENGINE runs, which is `wal_maintain_all_verified`, not
+ /// `WalWriter::prune_verified`.
+ ///
+ /// `prune_verified` has no production caller anywhere in the tree
+ /// (`rg 'prune_verified\('` hits `wal.rs` and its own unit tests only), so
+ /// a floor implemented and unit-tested there alone was decoration: on a
+ /// live node every rotated generation was still deleted the moment a flush
+ /// made it redundant, and a tap whose target was down for longer than one
+ /// flush interval still lost entries.
+ ///
+ /// Fails before the fix with `rotated after maintenance: 0`.
+ #[test]
+ fn the_retention_floor_holds_generations_through_engine_wal_maintenance() {
+ fn rotated_wal_files(store: &IndexStore) -> usize {
+ // One file per shard is the ACTIVE generation and is never a prune
+ // candidate; everything beyond that is retained-rotated.
+ walk_wal_files(&store.wal_dir()).len() - store.wal_min_retained_generations().len()
+ }
+
+ // Control: the default floor of 0 reclaims everything it can prove
+ // durable, exactly as before this change.
+ let bare_dir = tempfile::tempdir().unwrap();
+ let bare = open_test_store(bare_dir.path());
+ for round in 0..4 {
+ for i in 0..5 {
+ bare.index(format!("b{round}_{i}"), serde_json::json!({"v": i}))
+ .unwrap();
+ }
+ bare.flush().unwrap();
+ bare.force_wal_maintenance().unwrap();
+ }
+ assert_eq!(
+ rotated_wal_files(&bare),
+ 0,
+ "floor 0 must keep pruning to the active generation, files: {:?}",
+ walk_wal_files(&bare.wal_dir())
+ );
+
+ // Floor 2: the newest two rotated generations survive every
+ // maintenance pass, so a consumer two generations behind still finds
+ // its entries.
+ let dir = tempfile::tempdir().unwrap();
+ let store = open_test_store(dir.path());
+ store.set_wal_min_retained_generations(2);
+ for round in 0..4 {
+ for i in 0..5 {
+ store
+ .index(format!("d{round}_{i}"), serde_json::json!({"v": i}))
+ .unwrap();
+ }
+ store.flush().unwrap();
+ store.force_wal_maintenance().unwrap();
+ }
+ let retained = rotated_wal_files(&store);
+ assert_eq!(
+ retained,
+ 2 * store.wal_min_retained_generations().len(),
+ "the floor must hold 2 rotated generations per shard through the maintenance \
+ path the engine actually runs; files on disk: {:?}",
+ walk_wal_files(&store.wal_dir())
+ );
+
+ // And it is a floor, not a lease: it holds the same two however many
+ // more passes run, so a stalled consumer cannot grow the WAL without
+ // bound.
+ for _ in 0..3 {
+ store.force_wal_maintenance().unwrap();
+ }
+ assert_eq!(
+ rotated_wal_files(&store),
+ retained,
+ "a floor must not grow with the number of maintenance passes"
+ );
+
+ // Lowering it back to 0 releases them on the next pass — the knob is
+ // live in both directions.
+ store.set_wal_min_retained_generations(0);
+ store.force_wal_maintenance().unwrap();
+ assert_eq!(
+ rotated_wal_files(&store),
+ 0,
+ "clearing the floor must release the held generations, files: {:?}",
+ walk_wal_files(&store.wal_dir())
+ );
+ }
+
/// RC4 W2 #14 regression — a plain, never-re-indexed DELETE must not
/// pin its WAL shard forever. Pre-fix: the delete's tombstone lived
/// only in RAM + WAL, `sweep_pending_deletes` never unpinned it, and
diff --git a/engine/crates/xerj-storage/src/wal.rs b/engine/crates/xerj-storage/src/wal.rs
index 1df629504..c4adc1636 100644
--- a/engine/crates/xerj-storage/src/wal.rs
+++ b/engine/crates/xerj-storage/src/wal.rs
@@ -342,9 +342,15 @@ pub struct WalWriter {
/// append first re-attempts the fresh-generation heal so the writer
/// self-recovers once space frees up.
poisoned: bool,
- /// Rotated generations [`prune_verified`](Self::prune_verified) must keep
- /// even when every entry in them is proven durable — the retention floor
- /// a WAL consumer (the #320 tap) needs to survive a target outage.
+ /// Rotated generations a prune pass must keep even when every entry in
+ /// them is proven durable — the retention floor a WAL consumer (the #320
+ /// tap) needs to survive a target outage.
+ ///
+ /// Read by **both** prune paths: [`prune_verified`](Self::prune_verified)
+ /// and — through [`min_retained_generations`](Self::min_retained_generations)
+ /// — `IndexStore::wal_maintain_all_verified`, which is the one the engine
+ /// actually runs. Enforcing it in only one of them is how this knob
+ /// managed to be fully tested and still have no effect on a live node.
///
/// `0` (the default) is the historical behaviour: retention never waits
/// for anyone. Anything above it is a *bounded* floor, not a lease: the
@@ -476,16 +482,30 @@ impl WalWriter {
})
}
- /// Set the retention floor: rotated generations `prune_verified` keeps
- /// even when it can prove every entry in them durable.
+ /// Set the retention floor: rotated generations a prune pass keeps even
+ /// when it can prove every entry in them durable.
+ ///
+ /// Seeded by [`crate::index_store::IndexStore::open`] from
+ /// `wal_tap.min_retained_generations`, and re-set **live** by
+ /// [`IndexStore::set_wal_min_retained_generations`] when an operator
+ /// changes it through `PUT /_xerj/wal_tap`.
///
- /// Set once by [`crate::index_store::IndexStore::open`] from
- /// `wal_tap.min_retained_generations`, right after the writer is placed
- /// in its `Arc`.
+ /// [`IndexStore::set_wal_min_retained_generations`]:
+ /// crate::index_store::IndexStore::set_wal_min_retained_generations
pub fn set_min_retained_generations(&mut self, n: u64) {
self.min_retained_generations = n;
}
+ /// The retention floor currently in force on this shard.
+ ///
+ /// Exposed because the floor has to be enforced by the prune pass the
+ /// engine actually runs (`IndexStore::wal_maintain_all_verified`), not
+ /// only by [`prune_verified`](Self::prune_verified) — which no production
+ /// path calls.
+ pub fn min_retained_generations(&self) -> u64 {
+ self.min_retained_generations
+ }
+
/// Install (or clear) this shard's registration with the batched-fsync
/// scheduler. Set once, right after the writer is placed in its `Arc`
/// by [`crate::index_store::IndexStore::open`].