AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): issue #372 — both parallel-run failures are test isolation, not durability
xerj-orgb7ffcfdb6167
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 1345da72e..31ddd21c7 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -230,6 +230,40 @@ jobs:working-directory: enginerun: cargo test --profile ci-test --workspace -- --test-threads=2+ # Issue #372. `--test-threads=2` above is a deliberate configuration, and+ # it is not the one a contributor gets: plain `cargo test` runs one thread+ # per core. Two xerj-engine lib tests failed on a clean checkout at that+ # default and this workflow could not see either of them, because two+ # threads serialise enough of the suite to hide the interleaving. The+ # general rule the third such defect in one cycle earns: CI must run at+ # least one configuration that matches what a contributor actually types.+ #+ # Honest about its reach, because "default parallelism" is not a fixed+ # number: `ubuntu-latest` is a 2-4 vCPU runner, so the default here is+ # 2-4 threads -- barely above the `--test-threads=2` that hid #372, and+ # far from the 20 that surfaced it (11 of 16 clean-main runs red on a+ # 20-core box). This step would most likely NOT have caught #372 on this+ # runner. It is cheap insurance and the place a wider, better-sized gate+ # (#384) can land -- not a claim that the class is now covered.+ #+ # Scoped to the one crate whose lib tests have been *measured* green at+ # default parallelism (10 dev-profile and 5 ci-test-profile clean runs at+ # this commit, 20 cores). `--workspace --lib` is the destination, not this+ # commit: at default parallelism it is currently red in xerj-autoindex+ # (5 failures in 10 runs), where the `incremental_reconcile_http_tests`+ # module reaches the global `SNAPSHOT_FAILPOINT` without holding the+ # `SNAPSHOT_TEST_LOCK` that guards it and steals another test's injected+ # failure -- the same defect class as #372, in a different crate. That is+ # issue #385; widening this step past xerj-engine is #384. A gate is+ # worth nothing if it is merged red, so it goes in at the width it+ # passes at.+ #+ # Reuses the binaries the step above already built: the added cost is test+ # runtime, not another compile.+ - name: Engine lib tests at default parallelism (contributor's `cargo test`)+ working-directory: engine+ run: cargo test --profile ci-test -p xerj-engine --lib+# Fat LTO + codegen-units=1 + panic=abort over every workspace member is a# failure mode of its own (LTO ICEs, symbol collisions, panic-strategy# mismatches) and is not covered by `cargo clippy --all-targets` in `lint`diff --git a/engine/crates/xerj-engine/Cargo.toml b/engine/crates/xerj-engine/Cargo.tomlindex 826245fe7..e710362df 100644--- a/engine/crates/xerj-engine/Cargo.toml+++ b/engine/crates/xerj-engine/Cargo.toml@@ -80,7 +80,10 @@ stackprobe = [][dev-dependencies]xerj-storage = { path = "../xerj-storage", features = ["test-hooks"] }tempfile = { workspace = true }-tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }+# `test-util` is what lets a test pause tokio's clock (`start_paused = true`)+# and step a background timer deterministically instead of sleeping through it.+# Dev-only: with resolver = "2" it never reaches the release binary's graph.+tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util"] }criterion = { workspace = true }tracing-subscriber = { workspace = true }diff --git a/engine/crates/xerj-engine/src/cluster_state.rs b/engine/crates/xerj-engine/src/cluster_state.rsindex f6bf27565..715e38307 100644--- a/engine/crates/xerj-engine/src/cluster_state.rs+++ b/engine/crates/xerj-engine/src/cluster_state.rs@@ -85,6 +85,71 @@ impl PreparedClusterState {}}+/// Serialises every test in this crate's `--lib` binary that can reach the+/// `error!` in `PreparedClusterState::blocked` above.+///+/// `tracing-core` keeps one process-global `Interest` per callsite, and while a+/// single dispatcher is registered it recomputes that slot from the *calling+/// thread's* default subscriber — `rebuild_callsite_interest` ->+/// `Rebuilder::JustOne` -> `dispatcher::get_default`, tracing-core-0.1.36+/// `src/callsite.rs:490` and `:564`-`:565`. A thread with no subscriber+/// resolves to `NoSubscriber`, whose `register_callsite` returns+/// `Interest::never()` (`src/subscriber.rs:676`). So a test that reaches+/// `blocked` with no subscriber installed pins the boot diagnostic to "never"+/// for the whole process if it is the thread that first registers that+/// callsite — and a `capture_preflight_log` running concurrently comes back+/// with an empty string. That is the `cluster_state` half of issue #372:+/// `every_representative_block_class_logs_kind_detail_and_operator_path_at_boot`+/// failed with an empty log, and only at full test parallelism, because+/// `--test-threads=2` rarely produces the losing interleaving.+///+/// A poison that lands *before* a capture starts is harmless — installing a+/// subscriber is `Dispatch::new`, which unconditionally rebuilds every+/// registered callsite against the live dispatchers (`src/dispatcher.rs:479`+/// -> `src/callsite.rs:484`), and our subscriber is one of them. Only a poison+/// that lands *inside* the install-subscriber-then-log window can blank the+/// capture, so mutual exclusion is the whole fix.+///+/// Scope, exactly. This is `pub(crate)` rather than private to+/// `cluster_state::tests` because `blocked` is reachable from a *second* test+/// module in the same binary, so "no interleaving remains" is only true if that+/// one takes the lock too. `blocked` has exactly one production caller —+/// `preflight`, below — and `preflight` has exactly one production caller,+/// `Engine::new` (engine.rs:763), so the reachers are enumerable by grep. All+/// of them, as of this commit:+///+/// - `tests::capture_preflight_log` — holds it across+/// install-subscriber-then-log. The victim.+/// - `tests::preflight_serialised`, and `tests::preflight_bytes` through it —+/// no subscriber installed, so a poisoner.+/// - `tests::poison_boot_diagnostic_callsite` — the regression test's+/// deliberate model of a subscriber-less sibling.+/// - `lifecycle::tests::cluster_state_format1_lifecycle_delete_and_detach_guard_before_side_effects`+/// — writes a `format_version: 2` state and calls `Engine::new`, which+/// reaches `preflight`, classifies `UnsupportedFormat` and lands here.+/// Also subscriber-less, so also a poisoner.+///+/// That last one is latent rather than live, and the reason is test-name+/// ordering, not design: libtest runs a byte-sorted list, in which+/// `cluster_state::tests::*` ends at #106 of 601 and the `lifecycle` case is+/// #436 (measured with `--list` at this commit), and a callsite's registration+/// is one-shot per process — the first capture wins it and nothing `lifecycle`+/// does afterwards re-registers it. Nothing *enforces* that ordering, so the+/// lock is taken there rather than argued away. Any new test that boots an+/// `Engine` over a cluster state which classifies as blocked owes the same+/// call.+#[cfg(test)]+pub(crate) static PREFLIGHT_CALLSITE: std::sync::Mutex<()> = std::sync::Mutex::new(());++/// Lock `PREFLIGHT_CALLSITE`, ignoring poisoning.+///+/// A test that panics while holding it has already failed; propagating the+/// poison would convert one real failure into a cascade of unrelated ones.+#[cfg(test)]+pub(crate) fn lock_preflight_callsite() -> std::sync::MutexGuard<'static, ()> {+ PREFLIGHT_CALLSITE.lock().unwrap_or_else(|e| e.into_inner())+}+/// Exact seven-field envelope emitted by the only shipped format-1 writer.#[derive(Debug, Clone, Serialize, Deserialize)]#[serde(deny_unknown_fields)]@@ -436,6 +501,7 @@ mod tests {}fn capture_preflight_log(dir: &Path) -> (PreparedClusterState, String) {+ let _serialised = lock_preflight_callsite();let buffer = LogBuffer::default();let subscriber = tracing_subscriber::fmt().without_time()@@ -468,7 +534,32 @@ mod tests {fn preflight_bytes(bytes: &[u8]) -> PreparedClusterState {let dir = tempfile::tempdir().unwrap();std::fs::write(dir.path().join("cluster_state.json"), bytes).unwrap();- preflight(dir.path())+ preflight_serialised(dir.path())+ }++ /// `preflight`, but not while a `capture_preflight_log` is in flight.+ ///+ /// See `PREFLIGHT_CALLSITE`: this call has no subscriber installed, so it is+ /// the poisoner, not the victim.+ fn preflight_serialised(dir: &Path) -> PreparedClusterState {+ let _serialised = lock_preflight_callsite();+ preflight(dir)+ }++ /// Reproduce, on demand, what a subscriber-less sibling does to the+ /// process-global callsite interest cache.+ ///+ /// `preflight_bytes` only pins the boot diagnostic to `Interest::never()` on+ /// the process's *first* registration of that callsite, and a test cannot+ /// re-trigger a one-shot registration. Calling `rebuild_interest_cache`+ /// from a thread with no subscriber reaches the identical end state: the+ /// rebuild resolves to `NoSubscriber`, whose `register_callsite` is+ /// `Interest::never()`. Serialised on `PREFLIGHT_CALLSITE` because that is+ /// what makes it a faithful model of a sibling rather than an unfair+ /// hammer — a sibling holds the lock too.+ fn poison_boot_diagnostic_callsite() {+ let _serialised = lock_preflight_callsite();+ tracing::callsite::rebuild_interest_cache();}#[test]@@ -602,6 +693,55 @@ mod tests {}}+ /// Regression for the `cluster_state` half of #372.+ ///+ /// The reported failure was this module's boot-diagnostic assertions firing+ /// with an *empty* captured log at full test parallelism. The cause is a+ /// process-global cache, not the subscriber: a sibling test touching the+ /// same callsite from a thread with no subscriber pins it to+ /// `Interest::never()`, and if that lands inside a capture's window the+ /// event never reaches the writer. Model that sibling on a second thread+ /// and assert every capture still sees the diagnostic.+ #[test]+ fn a_concurrent_subscriber_less_preflight_cannot_blank_the_boot_diagnostic_capture() {+ use std::sync::atomic::{AtomicBool, Ordering};+ use std::sync::Arc;++ let dir = tempfile::tempdir().unwrap();+ let path = dir.path().join("cluster_state.json");+ std::fs::write(&path, br#"{"format_version":2}"#).unwrap();++ let stop = Arc::new(AtomicBool::new(false));+ let poisoner = {+ let stop = Arc::clone(&stop);+ std::thread::spawn(move || {+ while !stop.load(Ordering::Relaxed) {+ poison_boot_diagnostic_callsite();+ std::thread::yield_now();+ }+ })+ };++ let mut blank = 0usize;+ for _ in 0..200 {+ let (prepared, log) = capture_preflight_log(dir.path());+ assert_eq!(+ prepared.status.block_kind(),+ Some(&ClusterStateBlockKind::UnsupportedFormat)+ );+ if !log.contains("cluster state was rejected at boot") {+ blank += 1;+ }+ }+ stop.store(true, Ordering::Relaxed);+ poisoner.join().unwrap();++ assert_eq!(+ blank, 0,+ "a concurrent subscriber-less preflight blanked {blank}/200 boot-diagnostic captures"+ );+ }+#[cfg(unix)]#[test]fn read_failure_logs_the_same_boot_diagnostic_contract() {@@ -676,7 +816,7 @@ mod tests {let staging_bytes = b"owned-by-another-or-interrupted-writer";std::fs::write(&staging, staging_bytes).unwrap();- let prepared = preflight(dir.path());+ let prepared = preflight_serialised(dir.path());assert!(!prepared.status.is_writable() || live.is_none(),"fixture must not classify as loaded format 1"@@ -693,7 +833,7 @@ mod tests {let staging = dir.path().join("cluster_state.tmp");std::fs::write(&staging, b"interrupted format-1 stage").unwrap();- let prepared = preflight(dir.path());+ let prepared = preflight_serialised(dir.path());assert_eq!(prepared.status, ClusterStateBootStatus::LoadedV1);assert!(!staging.exists());}diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 00c2c4629..172914db0 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -2042,6 +2042,60 @@ mod merge_publication_transaction_tests {config}+ /// Open an engine plus one index with the background merge loop already+ /// stopped, for a test that drives every merge itself.+ ///+ /// `Index::create` (index.rs:6191) and `Index::open` (index.rs:6554) both+ /// start a 5-second merge pass, and with this module's+ /// `min_merge_count = max_merge_count = 2` a single tick rewrites the+ /// segment set two segments at a time. Every test here calls+ /// `run_merge_once()` by hand, so that tick is pure interference with two+ /// distinct failure shapes: it can win the `merge_in_progress` CAS and turn+ /// the test's own `run_merge_once()` into `Ok(0)` (the flake `fixture`+ /// already guarded against), and it can collapse fixture segments before the+ /// test has even looked at them — issue #372, where+ /// `partial_multi_batch_failure_is_reported_without_losing_either_batch`+ /// built its index inline, never stopped the loop, and failed its+ /// *precondition* with `left: 3, right: 4` once a full-parallelism run+ /// stretched the body past one tick. Constructing every index in this+ /// module here — including the one fixture that needs a custom `Config`,+ /// through `hand_driven_index_with_config` — means the omission cannot be+ /// reintroduced one test at a time.+ fn hand_driven_index(dir: &TempDir, name: &str, schema: Schema) -> (Engine, Arc<Index>) {+ hand_driven_index_with_config(config(dir), name, schema)+ }++ /// `hand_driven_index` for the fixture that needs a knob `config()` does+ /// not set.+ ///+ /// Only `artifact_bytes_at_level` uses this: it varies+ /// `compression.level` and pins `ingest_shards = 1`. It exists so that+ /// site does not have to remember `abort_background_tasks()` on its own —+ /// it is the one index in this module that was still being built by hand.+ fn hand_driven_index_with_config(+ cfg: xerj_common::config::Config,+ name: &str,+ schema: Schema,+ ) -> (Engine, Arc<Index>) {+ let engine = Engine::new(cfg).unwrap();+ engine.create_index(name, schema).unwrap();+ let index = engine.get_index(name).unwrap();+ index.abort_background_tasks();+ (engine, index)+ }++ /// Reopen `name` from `dir` with the background merge loop already stopped.+ ///+ /// Same contract as `hand_driven_index`: the restart half of these tests+ /// asserts on a recovered segment set, and `Index::open` starts its own+ /// merge pass too (`spawn_merge_task(5)`, index.rs:6554).+ fn reopen_hand_driven(dir: &TempDir, name: &str) -> (Engine, Arc<Index>) {+ let engine = Engine::new(config(dir)).unwrap();+ let index = engine.get_index(name).unwrap();+ index.abort_background_tasks();+ (engine, index)+ }+fn dot_field_schema() -> Schema {let mut schema = Schema::empty();schema@@ -2063,10 +2117,7 @@ mod merge_publication_transaction_tests {) -> (Engine, Arc<Index>, String, PathBuf) {use sha2::{Digest, Sha256};- let engine = Engine::new(config(dir)).unwrap();- engine.create_index(index_name, dot_field_schema()).unwrap();- let index = engine.get_index(index_name).unwrap();- index.abort_background_tasks();+ let (engine, index) = hand_driven_index(dir, index_name, dot_field_schema());for (id, value) in [("one", "alpha"), ("two", "beta")] {index.index_document(Some(id.into()), serde_json::json!({".": value}))@@ -2164,10 +2215,7 @@ mod merge_publication_transaction_tests {schema.add_field(FieldConfig::new("body", FieldType::Text)).unwrap();- let engine = Engine::new(cfg).unwrap();- engine.create_index("compression", schema).unwrap();- let index = engine.get_index("compression").unwrap();- index.abort_background_tasks();+ let (_engine, index) = hand_driven_index_with_config(cfg, "compression", schema);// Two flushes → two segments to merge. Each is comfortably over// `V2_MIN_DOCS` (128), so the stored section takes the columnar v2@@ -2338,11 +2386,13 @@ mod merge_publication_transaction_tests {}async fn fixture(dir: &TempDir) -> (Engine, Arc<Index>) {- let engine = Engine::new(config(dir)).unwrap();- engine- .create_index("post-save-panic", Schema::empty())- .unwrap();- let index = engine.get_index("post-save-panic").unwrap();+ // `hand_driven_index` stops the background merge loop *before* the two+ // flushes below, not after them. Stopping it afterwards still left the+ // `segments.len() == 2` assertion exposed to a tick that landed while+ // the fixture was still filling — the same window that broke+ // `partial_multi_batch_failure_is_reported_without_losing_either_batch`+ // in #372.+ let (engine, index) = hand_driven_index(dir, "post-save-panic", Schema::empty());index.index_document(Some("a".into()), serde_json::json!({"value": "old-a"})).await@@ -2354,14 +2404,6 @@ mod merge_publication_transaction_tests {.unwrap();index.flush().await.unwrap();assert_eq!(index.store.snapshot().segments.len(), 2);- // Every test in this module drives the merge itself with an explicit- // `run_merge_once()`. The 5-second background merge loop is pure- // interference here: a tick that wins the `merge_in_progress` CAS makes- // the test's own `run_merge_once()` return `Ok(0)`, after which- // `merge.await.unwrap_err()` fails with "called `unwrap_err()` on an- // `Ok` value" — a second, distinct flake mode. Same precedent as- // `cancelling_waiter_while_blocking_worker_is_queued_still_restores`.- index.abort_background_tasks();(engine, index)}@@ -2416,8 +2458,7 @@ mod merge_publication_transaction_tests {drop(index);drop(engine);- let reopened = Engine::new(config(&dir)).unwrap();- let reopened_index = reopened.get_index("post-save-panic").unwrap();+ let (_reopened, reopened_index) = reopen_hand_driven(&dir, "post-save-panic");for (id, expected) in [("a", "old-a"), ("b", "old-b")] {assert_eq!(reopened_index.get_document(id).await.unwrap().unwrap()["value"],@@ -2498,8 +2539,7 @@ mod merge_publication_transaction_tests {);drop(index);drop(engine);- let reopened = Engine::new(config(&dir)).unwrap();- let reopened_index = reopened.get_index("post-save-panic").unwrap();+ let (_reopened, reopened_index) = reopen_hand_driven(&dir, "post-save-panic");assert_eq!(reopened_index.get_document("a").await.unwrap().unwrap()["value"],"new-a"@@ -2615,8 +2655,7 @@ mod merge_publication_transaction_tests {index.set_publication_test_hook(None);drop(index);drop(engine);- let reopened = Engine::new(config(&dir)).unwrap();- let recovered = reopened.get_index("post-save-panic").unwrap();+ let (_reopened, recovered) = reopen_hand_driven(&dir, "post-save-panic");for id in ["a", "b"] {assert!(recovered.get_document(id).await.unwrap().is_some());}@@ -2631,12 +2670,51 @@ mod merge_publication_transaction_tests {);}+ /// Regression for #372: a background merge tick must not be able to move a+ /// hand-driven fixture's segment set.+ ///+ /// `start_paused` is what makes this deterministic instead of a 5-second+ /// sleep: with tokio's clock paused the runtime auto-advances to the next+ /// timer deadline whenever it goes idle, so the `sleep` below fires exactly+ /// the merge tick that `spawn_merge_task` scheduled — the tick that, on a+ /// real clock, only lands when a full-parallelism run stretches a test body+ /// past five seconds. Without `hand_driven_index`'s+ /// `abort_background_tasks()` this collapses two of the four segments and+ /// the assertion reports `left: 3, right: 4`, which is verbatim the failure+ /// reported against `index.rs:2650` on clean main.+ #[tokio::test(start_paused = true)]+ async fn a_background_merge_tick_cannot_move_a_hand_driven_fixture_segment_set() {+ let dir = TempDir::new().unwrap();+ let (_engine, index) = hand_driven_index(&dir, "tick-guard", Schema::empty());+ for id in ["a", "b", "c", "d"] {+ index+ .index_document(+ Some(id.into()),+ serde_json::json!({"value": format!("source-{id}")}),+ )+ .await+ .unwrap();+ index.flush().await.unwrap();+ }+ assert_eq!(index.store.snapshot().segments.len(), 4);+ // Jump the paused clock well past the 5 s merge cadence, several times+ // over, and hand the runtime enough turns for a tick to run to+ // completion if one were scheduled.+ for _ in 0..4 {+ tokio::time::sleep(std::time::Duration::from_secs(30)).await;+ tokio::task::yield_now().await;+ }+ assert_eq!(+ index.store.snapshot().segments.len(),+ 4,+ "a background merge tick rewrote a fixture that drives its own merges"+ );+ }+#[tokio::test]async fn partial_multi_batch_failure_is_reported_without_losing_either_batch() {let dir = TempDir::new().unwrap();- let engine = Engine::new(config(&dir)).unwrap();- engine.create_index("multi-batch", Schema::empty()).unwrap();- let index = engine.get_index("multi-batch").unwrap();+ let (engine, index) = hand_driven_index(&dir, "multi-batch", Schema::empty());for id in ["a", "b", "c", "d"] {index.index_document(@@ -2671,8 +2749,7 @@ mod merge_publication_transaction_tests {drop(index);drop(engine);- let reopened = Engine::new(config(&dir)).unwrap();- let reopened_index = reopened.get_index("multi-batch").unwrap();+ let (_reopened, reopened_index) = reopen_hand_driven(&dir, "multi-batch");for id in ["a", "b", "c", "d"] {assert_eq!(reopened_index.get_document(id).await.unwrap().unwrap()["value"],@@ -2715,8 +2792,7 @@ mod merge_publication_transaction_tests {drop(index);drop(engine);- let reopened = Engine::new(config(&dir)).unwrap();- let reopened_index = reopened.get_index("post-save-panic").unwrap();+ let (_reopened, reopened_index) = reopen_hand_driven(&dir, "post-save-panic");assert_eq!(reopened_index.store.snapshot().segments.len(), 1);for (id, expected) in [("a", "old-a"), ("b", "old-b")] {assert_eq!(diff --git a/engine/crates/xerj-engine/src/lifecycle.rs b/engine/crates/xerj-engine/src/lifecycle.rsindex 128d67be4..c34eee555 100644--- a/engine/crates/xerj-engine/src/lifecycle.rs+++ b/engine/crates/xerj-engine/src/lifecycle.rs@@ -774,7 +774,22 @@ mod tests {include_bytes!("../tests/fixtures/cluster_state/v2-format-version-minimal.json");let state_path = dir.path().join("cluster_state.json");std::fs::write(&state_path, future).unwrap();- let engine = Engine::new(config).unwrap();+ // The only reacher of the boot diagnostic outside `cluster_state::tests`+ // in this `--lib` binary: preflight classifies this fixture as+ // `UnsupportedFormat` and `PreparedClusterState::blocked` emits its+ // `error!` from this thread, which has no subscriber. Whichever thread+ // first registers that callsite decides its process-global `Interest`,+ // and a subscriber-less registration pins it to `never`, blanking any+ // concurrent `cluster_state::tests` capture — issue #372. Today this+ // cannot lose that race: libtest runs a byte-sorted name list, and+ // `cluster_state::tests::*` ends at #106 of 601 while this test is #436+ // (measured with `--list` at this commit), so registration is long+ // settled. The lock is what stops the guarantee from resting on that.+ // See `cluster_state::PREFLIGHT_CALLSITE`.+ let engine = {+ let _serialised = crate::cluster_state::lock_preflight_callsite();+ Engine::new(config).unwrap()+ };assert!(matches!(engine.get_index("logs-1"),Err(EngineError::Common(