AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
perf(engine): event-driven merge scheduling — idle indices cost zero wakeups (#871) (#881)
xerj-org438ecebe82cd
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex ad82d1ee0..4999e78c9 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -3515,18 +3515,21 @@ impl Engine {/// startup — and if startup index-discovery doesn't pick the index up/// (e.g. WAL-only indexes), the data is lost.///- /// First aborts every index's per-Index merge background task — those- /// tasks are spawned via `tokio::spawn` and use a `tokio::time::sleep`- /// loop, which keeps the tokio runtime alive even after axum has- /// stopped accepting connections. Without aborting them up-front,- /// the process stays at 100% CPU until the next sleep wake notices- /// the index is dropped (or a merge fires post-shutdown — either way- /// SIGTERM hangs). See bench `engine/reports/2026-04-25T03-30-00`- /// for the captured regression introduced by B-2b (commit 605ac7b).+ /// First stops every index's background merge scheduling (#871:+ /// event-driven debounced checks; historically a per-index tick task)+ /// and aborts any armed check task — a pending debounce+ /// `tokio::time::sleep` keeps the tokio runtime alive even after axum+ /// has stopped accepting connections. Without stopping them up-front,+ /// the process waits for the sleep to wake (or a merge fires+ /// post-shutdown — either way SIGTERM hangs). Stopping FIRST also+ /// guarantees the shutdown flushes below cannot arm new checks through+ /// the segments-changed hook. See bench+ /// `engine/reports/2026-04-25T03-30-00` for the captured regression+ /// introduced by B-2b (commit 605ac7b).pub async fn flush_all_force(&self) {- // 1. Stop all background merges so the runtime can exit once the- // flush is done. Aborts are non-blocking; the spawned task is- // unwound by tokio without us needing to await it.+ // 1. Stop all background merge scheduling so the runtime can exit+ // once the flush is done. Aborts are non-blocking; the armed+ // task is unwound by tokio without us needing to await it.for entry in self.indices.iter() {entry.value().abort_background_tasks();}diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 21a9f9d79..ac22ff770 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -3405,14 +3405,16 @@ mod merge_publication_transaction_tests {config}- /// Open an engine plus one index with the background merge loop already+ /// Open an engine plus one index with background merge scheduling already/// stopped, for a test that drives every merge itself.///- /// `Index::create` (index.rs:6191) and `Index::open` (index.rs:6554) both- /// start a 5-second merge pass, and with this module's- /// `min_merge_count = max_merge_count = 2` a single tick rewrites the+ /// `Index::create` and `Index::open` wire the store's segments-changed+ /// hook (#871), so every flush a test performs arms a debounced+ /// background merge check — and with this module's+ /// `min_merge_count = max_merge_count = 2` one check rewrites the/// segment set two segments at a time. Every test here calls- /// `run_merge_once()` by hand, so that tick is pure interference with two+ /// `run_merge_once()` by hand, so that background pass is pure+ /// interference with two/// distinct failure shapes: it can win the `merge_in_progress` CAS and turn/// the test's own `run_merge_once()` into `Ok(0)` (the flake `fixture`/// already guarded against), and it can collapse fixture segments before the@@ -3447,11 +3449,13 @@ mod merge_publication_transaction_tests {(engine, index)}- /// Reopen `name` from `dir` with the background merge loop already stopped.+ /// Reopen `name` from `dir` with background merge scheduling already+ /// stopped.////// Same contract as `hand_driven_index`: the restart half of these tests- /// asserts on a recovered segment set, and `Index::open` starts its own- /// merge pass too (`spawn_merge_task(5)`, index.rs:6554).+ /// asserts on a recovered segment set, and `Index::open` schedules its+ /// own open-time merge check too (`request_merge_check`, #871 — an+ /// opened index may carry pre-existing merge debt).fn reopen_hand_driven(dir: &TempDir, name: &str) -> (Engine, Arc<Index>) {let engine = Engine::new(config(dir)).unwrap();let index = engine.get_index(name).unwrap();@@ -4033,13 +4037,14 @@ mod merge_publication_transaction_tests {);}- /// Regression for #372: a background merge tick must not be able to move a+ /// Regression for #372: a background merge pass must not be able to move a/// hand-driven fixture's segment set.////// `start_paused` is what makes this deterministic instead of a 5-second/// sleep: with tokio's clock paused the runtime auto-advances to the next/// timer deadline whenever it goes idle, so the `sleep` below fires exactly- /// the merge tick that `spawn_merge_task` scheduled — the tick that, on a+ /// the debounced merge check each flush would arm (#871; historically, the+ /// 5-s tick `spawn_merge_task` scheduled) — the pass that, on a/// real clock, only lands when a full-parallelism run stretches a test body/// past five seconds. Without `hand_driven_index`'s/// `abort_background_tasks()` this collapses two of the four segments and@@ -4074,6 +4079,69 @@ mod merge_publication_transaction_tests {);}+ /// #871 — merge scheduling must be event-driven: after an index's merge+ /// work settles, ZERO further merge-policy evaluations may run until the+ /// segment set changes again.+ ///+ /// On the pre-fix code this fails at the flat-counter assertion: every+ /// index owns a tokio task that wakes every 5 s forever and evaluates the+ /// merge policy (CAS + `SizeTieredMergePolicy` + registry snapshot) with+ /// no writes — 93 wakeups/s at the customer-reported 464 indices. Same+ /// paused-clock technique as the #372 guard above: auto-advance fires+ /// exactly the timers that are scheduled, so an idle 60-minute window+ /// runs every evaluation the scheduler would ever run.+ #[tokio::test(start_paused = true)]+ async fn idle_index_schedules_no_merge_evaluations_without_segment_changes() {+ let dir = TempDir::new().unwrap();+ // Deliberately NOT `hand_driven_index`: the background scheduler is+ // the subject under test, so it must stay running.+ let engine = Engine::new(config(&dir)).unwrap();+ engine.create_index("idle-871", Schema::empty()).unwrap();+ let index = engine.get_index("idle-871").unwrap();+ for id in ["a", "b", "c", "d"] {+ index+ .index_document(Some(id.into()), serde_json::json!({"value": id}))+ .await+ .unwrap();+ index.flush().await.unwrap();+ }+ // Let the scheduled checks and their cascading merges run to+ // quiescence (min/max_merge_count = 2 collapses two at a time).+ for _ in 0..8 {+ tokio::time::sleep(std::time::Duration::from_secs(30)).await;+ tokio::task::yield_now().await;+ }+ let settled = index.merge_evaluation_count();+ assert!(+ settled >= 1,+ "background merge scheduling never evaluated the index"+ );+ // IDLE: a full hour of clock with zero segment-set changes.+ for _ in 0..60 {+ tokio::time::sleep(std::time::Duration::from_secs(60)).await;+ tokio::task::yield_now().await;+ }+ assert_eq!(+ index.merge_evaluation_count(),+ settled,+ "merge evaluations ran on an idle index (#871 per-index tick)"+ );+ // A segment-set change (flush publishes a segment) wakes it back up.+ index+ .index_document(Some("e".into()), serde_json::json!({"value": "e"}))+ .await+ .unwrap();+ index.flush().await.unwrap();+ for _ in 0..8 {+ tokio::time::sleep(std::time::Duration::from_secs(30)).await;+ tokio::task::yield_now().await;+ }+ assert!(+ index.merge_evaluation_count() > settled,+ "a published segment no longer schedules a merge check"+ );+ }+#[tokio::test]async fn partial_multi_batch_failure_is_reported_without_losing_either_batch() {let dir = TempDir::new().unwrap();@@ -7350,17 +7418,41 @@ pub struct Index {/// the response.pub external_versions: Arc<dashmap::DashMap<String, u64>>,- /// Handle to the per-Index merge background task. Held so the- /// shutdown path (`Engine::flush_all_force` → SIGTERM exit) can- /// abort the task explicitly — without it the tokio runtime stays- /// alive after axum has stopped accepting connections, because- /// `tokio::spawn` keeps the runtime non-empty while the task's- /// 5-s `tokio::time::sleep` is pending. The bench at 2026-04-25- /// caught this as a SIGTERM hang regression introduced by B-2b.+ /// Handle to the currently armed debounced merge-check task, if any+ /// (#871 — armed by [`Index::request_merge_check`] when the segment+ /// set changes; an idle index holds `None`). Held so the shutdown+ /// path (`Engine::flush_all_force` → SIGTERM exit) can abort the task+ /// explicitly — without it the tokio runtime stays alive after axum+ /// has stopped accepting connections, because a spawned task keeps+ /// the runtime non-empty while its debounce `tokio::time::sleep` is+ /// pending. The bench at 2026-04-25 caught this as a SIGTERM hang+ /// regression introduced by B-2b (against the old always-on tick+ /// task; the contract is unchanged)./// `parking_lot::Mutex` (sync, no async, no `.await`) is fine/// because we only ever take/replace under it, never hold across/// any awaits.pub(crate) merge_task: Arc<parking_lot::Mutex<Option<tokio::task::JoinHandle<()>>>>,+ /// #871 — true while a debounced merge check is scheduled (armed).+ /// Set by [`Index::request_merge_check`], cleared by the check task+ /// when it quiesces; makes arming idempotent so any number of+ /// segment-set changes inside one debounce window cost one task.+ merge_check_armed: std::sync::atomic::AtomicBool,+ /// #871 — count of segment-set change events (flush published a+ /// segment, merge applied, tombstone-only segment persisted). The+ /// debounced check task compares this epoch before/after a pass to+ /// decide whether to run another cycle or quiesce.+ merge_events: AtomicU64,+ /// #871 — once true, no further merge checks are ever scheduled.+ /// Set by [`Index::abort_background_tasks`] (SIGTERM shutdown and+ /// hand-driven test fixtures). Matches the old semantics where the+ /// aborted per-index tick task was never respawned.+ merge_scheduling_stopped: std::sync::atomic::AtomicBool,+ /// #871 — number of merge-policy evaluations actually run (passes+ /// that won the `merge_in_progress` flag and snapshotted the segment+ /// registry). Observability for the idle-cost contract: on an idle+ /// index this counter must stay flat — see+ /// `idle_index_schedules_no_merge_evaluations_without_segment_changes`.+ merge_evaluations: AtomicU64,/// Failed writes of `schema.json` during dynamic-mapping evolution./// See [`Index::persist_evolved_schema`] (issue #204).schema_persist_failures: AtomicU64,@@ -7727,12 +7819,26 @@ impl Index {flush_signal: Arc::new(SyncFlushCoord::new()),external_versions: Arc::new(dashmap::DashMap::new()),merge_task: Arc::new(parking_lot::Mutex::new(None)),+ merge_check_armed: std::sync::atomic::AtomicBool::new(false),+ merge_events: AtomicU64::new(0),+ merge_scheduling_stopped: std::sync::atomic::AtomicBool::new(false),+ merge_evaluations: AtomicU64::new(0),schema_persist_failures: AtomicU64::new(0),});- // Kick off the background merge pass. 5 s cadence is aggressive- // enough to collapse a burst of flushes quickly without burning a- // core — every pass is cheap when there's nothing to merge.- index.spawn_merge_task(5);+ // #871 — event-driven merge scheduling: the store fires this hook on+ // every segment-set change (flush publication, merge application,+ // tombstone-only segment, orphan recovery) and the debounced check+ // task does the rest — see `request_merge_check`. No per-index tick+ // task: a fresh index has no segments and costs no timer and no+ // wakeup until its first flush publishes one.+ let merge_hook_index = Arc::downgrade(&index);+ index+ .store+ .set_segments_changed_hook(Some(Arc::new(move || {+ if let Some(idx) = merge_hook_index.upgrade() {+ idx.on_segments_changed();+ }+ })));Ok(index)}@@ -8096,9 +8202,28 @@ impl Index {flush_signal: Arc::new(SyncFlushCoord::new()),external_versions: Arc::new(dashmap::DashMap::new()),merge_task: Arc::new(parking_lot::Mutex::new(None)),+ merge_check_armed: std::sync::atomic::AtomicBool::new(false),+ merge_events: AtomicU64::new(0),+ merge_scheduling_stopped: std::sync::atomic::AtomicBool::new(false),+ merge_evaluations: AtomicU64::new(0),schema_persist_failures: AtomicU64::new(0),});- index.spawn_merge_task(5);+ // #871 — event-driven merge scheduling (see `Index::create` and+ // `request_merge_check`): wire the store's segments-changed hook.+ let merge_hook_index = Arc::downgrade(&index);+ index+ .store+ .set_segments_changed_hook(Some(Arc::new(move || {+ if let Some(idx) = merge_hook_index.upgrade() {+ idx.on_segments_changed();+ }+ })));+ // An opened index can carry pre-existing merge debt — a crash+ // mid-cascade, or merge config lowered since the segments were+ // written — that no future event would pay off on a read-only+ // index. Schedule one debounced check now; it quiesces after a+ // single evaluation when there is nothing to do.+ index.request_merge_check();// RC4 W2 item 17: a flush-time-stale HNSW snapshot used to stay// stale for the rest of the process lifetime (ingest kept paying// full graph-maintenance cost while the ANN path never served).@@ -10064,9 +10189,25 @@ impl Index {////// Returns the number of merge batches that succeeded.///- /// This is called from a background task (see `spawn_merge_task`) and may- /// also be invoked explicitly by tests.+ /// This is called from the debounced background merge check (see+ /// `request_merge_check`, #871) and may also be invoked explicitly by+ /// tests.pub async fn run_merge_once(&self) -> Result<usize> {+ match self.try_run_merge_once().await {+ Some(result) => result,+ // Another pass holds the merge flag — same "skip" contract the+ // public signature always had.+ None => Ok(0),+ }+ }++ /// [`Self::run_merge_once`], but a lost `merge_in_progress` race is+ /// distinguishable from "evaluated, nothing to merge": `None` means the+ /// flag was held by another pass (background or `_forcemerge`) and NO+ /// policy evaluation happened. The #871 debounced check task needs the+ /// distinction — on `None` it must retry a cycle later instead of+ /// quiescing with a possibly-unevaluated segment set.+ pub(crate) async fn try_run_merge_once(&self) -> Option<Result<usize>> {use std::sync::atomic::Ordering as AtomicOrdering;// Serialise merges with ourselves (skip if another pass is running).@@ -10075,7 +10216,7 @@ impl Index {.compare_exchange(false, true, AtomicOrdering::AcqRel, AtomicOrdering::Relaxed).is_err(){- return Ok(0);+ return None;}// Ensure we clear the flag on every exit path.let _clear = MergeFlagClear(&self.merge_in_progress);@@ -10091,7 +10232,7 @@ impl Index {}}}- result+ Some(result)}/// True while a merge pass (background or forced) holds the merge flag.@@ -10102,6 +10243,15 @@ impl Index {.load(std::sync::atomic::Ordering::Acquire)}+ /// #871 — how many merge-policy evaluations have run on this index+ /// (background debounced checks, hand-driven `run_merge_once`, and+ /// `_forcemerge` passes all count). The idle-cost contract is that this+ /// stays flat while the segment set does not change.+ pub fn merge_evaluation_count(&self) -> u64 {+ self.merge_evaluations+ .load(std::sync::atomic::Ordering::Relaxed)+ }+/// ES-style `_forcemerge`: run SYNCHRONOUSLY until the index has at/// most `max_num_segments` segments.///@@ -10179,6 +10329,13 @@ impl Index {use xerj_storage::merge::{MergePolicy, SizeTieredMergePolicy};use xerj_storage::segment::{SectionType, SegmentId, SegmentReader, SegmentWriter};+ // #871 idle-cost observability: every pass that reaches this point+ // builds a merge policy and snapshots the segment registry — that IS+ // the per-tick idle cost the event-driven scheduler exists to remove,+ // so it is exactly what the counter measures. Background debounced+ // checks, hand-driven `run_merge_once`, and `_forcemerge` all count.+ self.merge_evaluations.fetch_add(1, AtomicOrdering::Relaxed);+// V4 M4.5 — larger merge batches so a bursty-ingest index with// thousands of small segments converges in far fewer passes.// Override via `XERJ_MAX_MERGE_COUNT` for battle tests that@@ -11251,62 +11408,172 @@ impl Index {Ok(merged_batches)}- /// Spawn a background tokio task that runs the merge pass every- /// `interval` seconds until the index is dropped.- ///- /// Uses a Weak pointer so the task exits naturally when the last Arc to- /// the index is released — no explicit shutdown plumbing needed for the- /// happy path. The returned `JoinHandle` is stored in `self.merge_task`- /// so the SIGTERM path can call `abort_background_tasks` to break the- /// 5-second `tokio::time::sleep` and let the runtime exit promptly- /// (otherwise tokio waits for the sleep to wake before noticing the- /// engine has been dropped — that is the bench-found shutdown hang).- pub fn spawn_merge_task(self: &Arc<Self>, interval_secs: u64) {- let weak = Arc::downgrade(self);- // Allow override via env for benchmarks / battle tests so merges- // don't interfere with ingest-rate measurements.- let effective = std::env::var("XERJ_MERGE_INTERVAL_SECS")+ /// #871 — the debounce window between a segment-set change and the+ /// merge check it schedules. `XERJ_MERGE_INTERVAL_SECS` keeps its+ /// historical meaning (benchmarks push it up so merges don't pollute+ /// ingest-rate measurements): it is now the debounce delay instead of a+ /// free-running tick period. Default 5 s — the old tick cadence, so a+ /// burst of flushes still collapses on the same schedule as before.+ fn merge_check_delay() -> std::time::Duration {+ let secs = std::env::var("XERJ_MERGE_INTERVAL_SECS").ok().and_then(|s| s.parse::<u64>().ok())- .unwrap_or(interval_secs);- let interval = std::time::Duration::from_secs(effective.max(1));- let handle = tokio::spawn(async move {- tracing::info!(interval_secs, "merge background task started");+ .unwrap_or(5);+ std::time::Duration::from_secs(secs.max(1))+ }++ /// #871 — the segment set changed (the [`IndexStore`+ /// segments-changed hook](xerj_storage::index_store::IndexStore::set_segments_changed_hook)+ /// lands here): bump the event epoch, then make sure a debounced merge+ /// check is scheduled.+ fn on_segments_changed(self: &Arc<Self>) {+ self.merge_events+ .fetch_add(1, std::sync::atomic::Ordering::SeqCst);+ self.request_merge_check();+ }++ /// #871 — schedule a debounced merge-policy check for this index.+ ///+ /// Replaces the per-index tick task (`spawn_merge_task`) that gave+ /// EVERY index a tokio timer waking every 5 s forever, writes or no+ /// writes — measured 93 wakeups/s and 15-27% of a core at the+ /// customer-reported 464 idle indices. Merge scheduling is now+ /// event-driven, mirroring the #334 WAL-fsync scheduler+ /// (`xerj-storage/src/wal_fsync.rs`): the storage layer fires its+ /// segments-changed hook on every segment-set change — flush+ /// publication, merge application, tombstone-only segment, orphan+ /// recovery: the only events that can create a merge candidate — and+ /// that hook lands here via [`Self::on_segments_changed`]. Lucene's+ /// writer works the same way: `IndexWriter.maybeMerge` runs on flush+ /// (IndexWriter.java:706,:3681 — MergeTrigger.FULL_FLUSH) and on merge+ /// completion (IndexWriter.java:2452 — MERGE_FINISHED), never on a+ /// clock (lucene corpus, Apache-2.0; design mirrored, no code copied).+ ///+ /// Contract:+ /// * Arming is idempotent — any number of events inside one debounce+ /// window cost one task (`merge_check_armed` swap).+ /// * The armed task sleeps [`Self::merge_check_delay`], evaluates once,+ /// and loops only while there is a reason to: the pass lost the merge+ /// flag to `_forcemerge` (`try_run_merge_once` → `None`, nothing was+ /// evaluated), the pass errored (the same every-pass retry the old+ /// tick gave a damaged segment — see the LOSS FIREWALL note in+ /// `merge_pass_locked`), or the segment set changed since the cycle+ /// started (`merge_events` epoch). Otherwise it disarms and EXITS —+ /// an idle index holds zero tasks, zero timers, zero wakeups.+ /// * Merge cascades need no special case: applying a merge fires the+ /// hook, so the follow-up evaluation schedules itself.+ /// * The disarm race (an event between the epoch check and the disarm+ /// observed `armed == true` and spawned nothing) is closed by+ /// re-checking the epoch after the disarm and re-arming on the+ /// event's behalf.+ ///+ /// The task handle is stored in `self.merge_task` so the SIGTERM path+ /// (`abort_background_tasks`, via `Engine::flush_all_force`) can break+ /// the debounce `tokio::time::sleep` — the same shutdown-hang contract+ /// the tick task had; `merge_scheduling_stopped` then keeps any later+ /// event from re-arming.+ pub fn request_merge_check(self: &Arc<Self>) {+ use std::sync::atomic::Ordering as AtomicOrdering;+ if self.merge_scheduling_stopped.load(AtomicOrdering::SeqCst) {+ return;+ }+ // Resolve the runtime BEFORE arming: if there is nowhere to spawn,+ // leaving `armed` set would eat the next event's arm. The hook can+ // fire from a blocking-pool thread (flush finalize), where+ // `try_current` still resolves; the coord's cached handle covers+ // any path outside every runtime context.+ let rt = match tokio::runtime::Handle::try_current() {+ Ok(handle) => handle,+ Err(_) => match self.flush_signal.runtime() {+ Some(handle) => handle.clone(),+ None => return, // no runtime; the next event retries+ },+ };+ if self.merge_check_armed.swap(true, AtomicOrdering::SeqCst) {+ return; // already scheduled — debounced+ }+ let weak = Arc::downgrade(self);+ let delay = Self::merge_check_delay();+ let handle = rt.spawn(async move {loop {- tokio::time::sleep(interval).await;- let idx = match weak.upgrade() {- Some(a) => a,- None => {- tracing::info!("merge background task exiting (index dropped)");- return;+ // Epoch BEFORE the sleep: any event after this read either+ // re-runs this loop (epoch mismatch below) or re-arms after+ // the exit path's disarm.+ let seen = match weak.upgrade() {+ Some(idx) => idx.merge_events.load(AtomicOrdering::SeqCst),+ None => return,+ };+ tokio::time::sleep(delay).await;+ let Some(idx) = weak.upgrade() else { return };+ if idx.merge_scheduling_stopped.load(AtomicOrdering::SeqCst) {+ idx.merge_check_armed.store(false, AtomicOrdering::SeqCst);+ return;+ }+ let outcome = idx.try_run_merge_once().await;+ let retry = match &outcome {+ // Lost the flag to `_forcemerge`/another pass: nothing+ // was evaluated, so we still owe the arming event a+ // pass.+ None => true,+ Some(Err(e)) => {+ tracing::warn!("merge pass failed: {e}");+ true+ }+ Some(Ok(n)) => {+ if *n > 0 {+ tracing::debug!(batches = n, "merge pass ran");+ }+ false}};- match idx.run_merge_once().await {- Ok(0) => {}- Ok(n) => tracing::debug!(batches = n, "merge pass ran"),- Err(e) => tracing::warn!("merge pass failed: {e}"),+ if retry || idx.merge_events.load(AtomicOrdering::SeqCst) != seen {+ continue;+ }+ idx.merge_check_armed.store(false, AtomicOrdering::SeqCst);+ if idx.merge_events.load(AtomicOrdering::SeqCst) != seen {+ // An event slipped in between the epoch check and the+ // disarm: it saw `armed` and spawned nothing. Re-arm on+ // its behalf.+ idx.request_merge_check();}+ return;}});- // Replace any previous handle and abort it (defensive — should be- // None at first construction; only matters if a caller ever calls- // spawn_merge_task twice on the same Index).- let mut slot = self.merge_task.lock();- if let Some(prev) = slot.replace(handle) {- prev.abort();+ // Replace the previous handle. It is either None or a task on its+ // way out (a new arm is only possible after the old task disarmed,+ // i.e. past its last await) — dropping it detaches those final+ // instructions, never a pending sleep.+ *self.merge_task.lock() = Some(handle);+ if self.merge_scheduling_stopped.load(AtomicOrdering::SeqCst) {+ // Lost a race with `abort_background_tasks` between the check at+ // the top and the handle store above: finish the abort's job so+ // no debounce sleep outlives shutdown.+ if let Some(handle) = self.merge_task.lock().take() {+ handle.abort();+ }+ self.merge_check_armed.store(false, AtomicOrdering::SeqCst);}}- /// Abort the merge background task spawned by `spawn_merge_task`.+ /// Stop this index's background merge scheduling permanently and abort+ /// the armed check task, if any.////// Idempotent and safe to call from a signal handler. After this- /// returns, the merge task is no longer holding the tokio runtime- /// alive, so once axum's listeners stop and the final flush completes,- /// the runtime exits cleanly instead of waiting up to 5 s for the- /// merge sleep to wake on its own. The aborted task itself is still- /// being unwound, so ingest-side callers MUST NOT rely on the merge- /// loop running after this. Called from `Engine::flush_all_force`.+ /// returns, no merge task is holding the tokio runtime alive, so once+ /// axum's listeners stop and the final flush completes, the runtime+ /// exits cleanly instead of waiting up to the debounce delay for a+ /// `tokio::time::sleep` to wake on its own (the bench-found SIGTERM+ /// hang). `merge_scheduling_stopped` is set FIRST so a segments-changed+ /// hook firing concurrently (e.g. from the shutdown flush that follows+ /// in `Engine::flush_all_force`) cannot re-arm after the abort; the+ /// re-check at the end of `request_merge_check` closes the remaining+ /// interleaving. The aborted task itself is still being unwound, so+ /// ingest-side callers MUST NOT rely on merges running after this.+ /// Called from `Engine::flush_all_force` and by hand-driven test+ /// fixtures that drive every merge themselves.pub fn abort_background_tasks(&self) {+ self.merge_scheduling_stopped+ .store(true, std::sync::atomic::Ordering::SeqCst);if let Some(handle) = self.merge_task.lock().take() {handle.abort();}diff --git a/engine/crates/xerj-storage/src/index_store.rs b/engine/crates/xerj-storage/src/index_store.rsindex 5e5a38b5b..6bc86d312 100644--- a/engine/crates/xerj-storage/src/index_store.rs+++ b/engine/crates/xerj-storage/src/index_store.rs@@ -549,6 +549,16 @@ pub struct IndexStore {/// is already unlinked at retire time, so `recover_orphaned_segments`/// can never resurrect them as duplicates).retired_segments: Mutex<Vec<SegmentId>>,+ /// #871 — fired after every `self.snapshot` swap that changes the+ /// segment set: flush publication, merge application, tombstone-only+ /// segment persistence, orphan recovery. These are the only events+ /// that can create a merge candidate, so the engine installs a hook+ /// here that debounce-schedules a merge-policy check — mirroring the+ /// #334 wal_fsync design (work is scheduled by the event that creates+ /// it; an idle index costs no timer and no wakeup). `None` (e.g.+ /// storage-crate unit tests) is a no-op. The hook is cloned out of the+ /// mutex before it runs, so it never executes under a store lock.+ segments_changed_hook: Mutex<Option<Arc<dyn Fn() + Send + Sync>>>,/// Delete-durability fix (2026-07): `doc_id → (delete seq_no, wal/// shard)` for every acknowledged delete whose ONLY durable record/// is still its `WalEntry::Delete` in the WAL.@@ -723,6 +733,7 @@ impl IndexStore {last_wal_maintenance_ms: AtomicU64::new(0),read_leases: std::sync::atomic::AtomicUsize::new(0),retired_segments: Mutex::new(Vec::new()),+ segments_changed_hook: Mutex::new(None),pending_deletes: Mutex::new(std::collections::HashMap::new()),wal_prune_cache: Mutex::new(std::collections::HashMap::new()),#[cfg(test)]@@ -1052,6 +1063,7 @@ impl IndexStore {new_snap = new_snap.with_new_segment(meta.clone());}self.snapshot.store(Arc::new(new_snap));+ self.notify_segments_changed();// Persist immediately so a second restart doesn't need to re-recover.self.save_snapshot()?;@@ -2242,6 +2254,9 @@ impl IndexStore {segment_id,"post-publication panic caught; maintenance deferred");+ // #871 — the segment IS in the snapshot; merge scheduling+ // must hear about it even on this degraded path.+ self.notify_segments_changed();return Ok(FlushFinalizeOutcome::Published {meta,maintenance_deferred: true,@@ -2256,6 +2271,11 @@ impl IndexStore {));return Err(self.abandon_unpublished_segment(&segment_id, publication_error));}+ // #871 — the rcu above published a new segment: fire the segment-set+ // change hook so the engine debounce-schedules a merge check. This is+ // the flush-side analogue of Lucene's maybeMerge-on-flush+ // (IndexWriter.java:706, MergeTrigger.FULL_FLUSH).+ self.notify_segments_changed();if prof {eprintln!("XERJ_PROF finalize docs={} ser_us={} encode_us={} writer_finish_us={} post_finish_us={} vm_us={} total_so_far_us={}",@@ -2417,6 +2437,31 @@ impl IndexStore {}}+ /// #871 — install (or clear) the segment-set change hook. See the+ /// `segments_changed_hook` field docs; the engine wires this to its+ /// debounced merge-check scheduler right after constructing an index.+ pub fn set_segments_changed_hook(&self, hook: Option<Arc<dyn Fn() + Send + Sync>>) {+ *self+ .segments_changed_hook+ .lock()+ .unwrap_or_else(|e| e.into_inner()) = hook;+ }++ /// Fire the #871 segment-set change hook, if installed. Called on every+ /// path that swaps `self.snapshot` with a different segment list. The+ /// hook is cloned out first so it never runs under this store's locks+ /// (it may spawn a tokio task).+ fn notify_segments_changed(&self) {+ let hook = self+ .segments_changed_hook+ .lock()+ .unwrap_or_else(|e| e.into_inner())+ .clone();+ if let Some(hook) = hook {+ hook();+ }+ }+/// Return the current WAL sequence number (the next value that/// `wal_append_batch` would assign). Used by `Index::flush` to/// write a global checkpoint covering ALL shards after a@@ -2538,6 +2583,9 @@ impl IndexStore {// Publish so the caller's save_snapshot() registers it on disk.self.snapshot.rcu(|old| Arc::new(old.with_new_segment(meta.clone())));+ // #871 — a tombstone-only segment is a segment-set change (and a+ // merge candidate: merges are what fold tombstones away).+ self.notify_segments_changed();info!(segment_id = meta.id.as_str(),@@ -3751,6 +3799,14 @@ impl IndexStore {// segment swap. rcu retries on contention.self.snapshot.rcu(|old| Arc::new(old.replace_segments(merged_ids, new_meta.clone())));+ // #871 — merge application changed the segment set: fire the hook so+ // a cascading follow-up merge schedules itself (Lucene's+ // MergeTrigger.MERGE_FINISHED, IndexWriter.java:2452). Fired before+ // the fallible persistence below because the in-memory set HAS+ // changed on every path from here — including the rollback, which+ // swaps it twice more; a spurious debounced policy check is cheap,+ // a missed one strands merge debt until the next event.+ self.notify_segments_changed();if let Err(error) = self.save_snapshot() {// Publication changed memory before manifest persistence. Restore// the exact inputs only while this output remains authoritative;diff --git a/engine/crates/xerj-storage/src/merge.rs b/engine/crates/xerj-storage/src/merge.rsindex 2ea6d38fd..0bf2c932f 100644--- a/engine/crates/xerj-storage/src/merge.rs+++ b/engine/crates/xerj-storage/src/merge.rs@@ -203,8 +203,9 @@ impl Default for MergeConfig {/// **Not part of the supported public API — the server does NOT use this.**////// The running engine merges through its own path-/// (`xerj-engine::index::Index::merge_pass_locked`, driven by-/// `spawn_merge_task`); `MergeExecutor` is a legacy, storage-crate-only+/// (`xerj-engine::index::Index::merge_pass_locked`, driven by the+/// event-driven debounced checks of `Index::request_merge_check`, #871);+/// `MergeExecutor` is a legacy, storage-crate-only/// helper kept solely as a convenience for in-crate unit tests. It is/// `#[doc(hidden)]` and no longer re-exported at the crate root because it/// has two footguns that make it unsafe to call on a real index: