Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): issue #372 — both parallel-run failures are test isolation, not durability

xerj-orgb7ffcfdb6167
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 1345da72e..31ddd21c7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -230,6 +230,40 @@ jobs:
working-directory: engine
run: cargo test --profile ci-test --workspace -- --test-threads=2
+ # Issue #372. `--test-threads=2` above is a deliberate configuration, and
+ # it is not the one a contributor gets: plain `cargo test` runs one thread
+ # per core. Two xerj-engine lib tests failed on a clean checkout at that
+ # default and this workflow could not see either of them, because two
+ # threads serialise enough of the suite to hide the interleaving. The
+ # general rule the third such defect in one cycle earns: CI must run at
+ # least one configuration that matches what a contributor actually types.
+ #
+ # Honest about its reach, because "default parallelism" is not a fixed
+ # number: `ubuntu-latest` is a 2-4 vCPU runner, so the default here is
+ # 2-4 threads -- barely above the `--test-threads=2` that hid #372, and
+ # far from the 20 that surfaced it (11 of 16 clean-main runs red on a
+ # 20-core box). This step would most likely NOT have caught #372 on this
+ # runner. It is cheap insurance and the place a wider, better-sized gate
+ # (#384) can land -- not a claim that the class is now covered.
+ #
+ # Scoped to the one crate whose lib tests have been *measured* green at
+ # default parallelism (10 dev-profile and 5 ci-test-profile clean runs at
+ # this commit, 20 cores). `--workspace --lib` is the destination, not this
+ # commit: at default parallelism it is currently red in xerj-autoindex
+ # (5 failures in 10 runs), where the `incremental_reconcile_http_tests`
+ # module reaches the global `SNAPSHOT_FAILPOINT` without holding the
+ # `SNAPSHOT_TEST_LOCK` that guards it and steals another test's injected
+ # failure -- the same defect class as #372, in a different crate. That is
+ # issue #385; widening this step past xerj-engine is #384. A gate is
+ # worth nothing if it is merged red, so it goes in at the width it
+ # passes at.
+ #
+ # Reuses the binaries the step above already built: the added cost is test
+ # runtime, not another compile.
+ - name: Engine lib tests at default parallelism (contributor's `cargo test`)
+ working-directory: engine
+ run: cargo test --profile ci-test -p xerj-engine --lib
+
# Fat LTO + codegen-units=1 + panic=abort over every workspace member is a
# failure mode of its own (LTO ICEs, symbol collisions, panic-strategy
# mismatches) and is not covered by `cargo clippy --all-targets` in `lint`
diff --git a/engine/crates/xerj-engine/Cargo.toml b/engine/crates/xerj-engine/Cargo.toml
index 826245fe7..e710362df 100644
--- a/engine/crates/xerj-engine/Cargo.toml
+++ b/engine/crates/xerj-engine/Cargo.toml
@@ -80,7 +80,10 @@ stackprobe = []
[dev-dependencies]
xerj-storage = { path = "../xerj-storage", features = ["test-hooks"] }
tempfile = { workspace = true }
-tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
+# `test-util` is what lets a test pause tokio's clock (`start_paused = true`)
+# and step a background timer deterministically instead of sleeping through it.
+# Dev-only: with resolver = "2" it never reaches the release binary's graph.
+tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util"] }
criterion = { workspace = true }
tracing-subscriber = { workspace = true }
diff --git a/engine/crates/xerj-engine/src/cluster_state.rs b/engine/crates/xerj-engine/src/cluster_state.rs
index f6bf27565..715e38307 100644
--- a/engine/crates/xerj-engine/src/cluster_state.rs
+++ b/engine/crates/xerj-engine/src/cluster_state.rs
@@ -85,6 +85,71 @@ impl PreparedClusterState {
}
}
+/// Serialises every test in this crate's `--lib` binary that can reach the
+/// `error!` in `PreparedClusterState::blocked` above.
+///
+/// `tracing-core` keeps one process-global `Interest` per callsite, and while a
+/// single dispatcher is registered it recomputes that slot from the *calling
+/// thread's* default subscriber — `rebuild_callsite_interest` ->
+/// `Rebuilder::JustOne` -> `dispatcher::get_default`, tracing-core-0.1.36
+/// `src/callsite.rs:490` and `:564`-`:565`. A thread with no subscriber
+/// resolves to `NoSubscriber`, whose `register_callsite` returns
+/// `Interest::never()` (`src/subscriber.rs:676`). So a test that reaches
+/// `blocked` with no subscriber installed pins the boot diagnostic to "never"
+/// for the whole process if it is the thread that first registers that
+/// callsite — and a `capture_preflight_log` running concurrently comes back
+/// with an empty string. That is the `cluster_state` half of issue #372:
+/// `every_representative_block_class_logs_kind_detail_and_operator_path_at_boot`
+/// failed with an empty log, and only at full test parallelism, because
+/// `--test-threads=2` rarely produces the losing interleaving.
+///
+/// A poison that lands *before* a capture starts is harmless — installing a
+/// subscriber is `Dispatch::new`, which unconditionally rebuilds every
+/// registered callsite against the live dispatchers (`src/dispatcher.rs:479`
+/// -> `src/callsite.rs:484`), and our subscriber is one of them. Only a poison
+/// that lands *inside* the install-subscriber-then-log window can blank the
+/// capture, so mutual exclusion is the whole fix.
+///
+/// Scope, exactly. This is `pub(crate)` rather than private to
+/// `cluster_state::tests` because `blocked` is reachable from a *second* test
+/// module in the same binary, so "no interleaving remains" is only true if that
+/// one takes the lock too. `blocked` has exactly one production caller —
+/// `preflight`, below — and `preflight` has exactly one production caller,
+/// `Engine::new` (engine.rs:763), so the reachers are enumerable by grep. All
+/// of them, as of this commit:
+///
+/// - `tests::capture_preflight_log` — holds it across
+/// install-subscriber-then-log. The victim.
+/// - `tests::preflight_serialised`, and `tests::preflight_bytes` through it —
+/// no subscriber installed, so a poisoner.
+/// - `tests::poison_boot_diagnostic_callsite` — the regression test's
+/// deliberate model of a subscriber-less sibling.
+/// - `lifecycle::tests::cluster_state_format1_lifecycle_delete_and_detach_guard_before_side_effects`
+/// — writes a `format_version: 2` state and calls `Engine::new`, which
+/// reaches `preflight`, classifies `UnsupportedFormat` and lands here.
+/// Also subscriber-less, so also a poisoner.
+///
+/// That last one is latent rather than live, and the reason is test-name
+/// ordering, not design: libtest runs a byte-sorted list, in which
+/// `cluster_state::tests::*` ends at #106 of 601 and the `lifecycle` case is
+/// #436 (measured with `--list` at this commit), and a callsite's registration
+/// is one-shot per process — the first capture wins it and nothing `lifecycle`
+/// does afterwards re-registers it. Nothing *enforces* that ordering, so the
+/// lock is taken there rather than argued away. Any new test that boots an
+/// `Engine` over a cluster state which classifies as blocked owes the same
+/// call.
+#[cfg(test)]
+pub(crate) static PREFLIGHT_CALLSITE: std::sync::Mutex<()> = std::sync::Mutex::new(());
+
+/// Lock `PREFLIGHT_CALLSITE`, ignoring poisoning.
+///
+/// A test that panics while holding it has already failed; propagating the
+/// poison would convert one real failure into a cascade of unrelated ones.
+#[cfg(test)]
+pub(crate) fn lock_preflight_callsite() -> std::sync::MutexGuard<'static, ()> {
+ PREFLIGHT_CALLSITE.lock().unwrap_or_else(|e| e.into_inner())
+}
+
/// Exact seven-field envelope emitted by the only shipped format-1 writer.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
@@ -436,6 +501,7 @@ mod tests {
}
fn capture_preflight_log(dir: &Path) -> (PreparedClusterState, String) {
+ let _serialised = lock_preflight_callsite();
let buffer = LogBuffer::default();
let subscriber = tracing_subscriber::fmt()
.without_time()
@@ -468,7 +534,32 @@ mod tests {
fn preflight_bytes(bytes: &[u8]) -> PreparedClusterState {
let dir = tempfile::tempdir().unwrap();
std::fs::write(dir.path().join("cluster_state.json"), bytes).unwrap();
- preflight(dir.path())
+ preflight_serialised(dir.path())
+ }
+
+ /// `preflight`, but not while a `capture_preflight_log` is in flight.
+ ///
+ /// See `PREFLIGHT_CALLSITE`: this call has no subscriber installed, so it is
+ /// the poisoner, not the victim.
+ fn preflight_serialised(dir: &Path) -> PreparedClusterState {
+ let _serialised = lock_preflight_callsite();
+ preflight(dir)
+ }
+
+ /// Reproduce, on demand, what a subscriber-less sibling does to the
+ /// process-global callsite interest cache.
+ ///
+ /// `preflight_bytes` only pins the boot diagnostic to `Interest::never()` on
+ /// the process's *first* registration of that callsite, and a test cannot
+ /// re-trigger a one-shot registration. Calling `rebuild_interest_cache`
+ /// from a thread with no subscriber reaches the identical end state: the
+ /// rebuild resolves to `NoSubscriber`, whose `register_callsite` is
+ /// `Interest::never()`. Serialised on `PREFLIGHT_CALLSITE` because that is
+ /// what makes it a faithful model of a sibling rather than an unfair
+ /// hammer — a sibling holds the lock too.
+ fn poison_boot_diagnostic_callsite() {
+ let _serialised = lock_preflight_callsite();
+ tracing::callsite::rebuild_interest_cache();
}
#[test]
@@ -602,6 +693,55 @@ mod tests {
}
}
+ /// Regression for the `cluster_state` half of #372.
+ ///
+ /// The reported failure was this module's boot-diagnostic assertions firing
+ /// with an *empty* captured log at full test parallelism. The cause is a
+ /// process-global cache, not the subscriber: a sibling test touching the
+ /// same callsite from a thread with no subscriber pins it to
+ /// `Interest::never()`, and if that lands inside a capture's window the
+ /// event never reaches the writer. Model that sibling on a second thread
+ /// and assert every capture still sees the diagnostic.
+ #[test]
+ fn a_concurrent_subscriber_less_preflight_cannot_blank_the_boot_diagnostic_capture() {
+ use std::sync::atomic::{AtomicBool, Ordering};
+ use std::sync::Arc;
+
+ let dir = tempfile::tempdir().unwrap();
+ let path = dir.path().join("cluster_state.json");
+ std::fs::write(&path, br#"{"format_version":2}"#).unwrap();
+
+ let stop = Arc::new(AtomicBool::new(false));
+ let poisoner = {
+ let stop = Arc::clone(&stop);
+ std::thread::spawn(move || {
+ while !stop.load(Ordering::Relaxed) {
+ poison_boot_diagnostic_callsite();
+ std::thread::yield_now();
+ }
+ })
+ };
+
+ let mut blank = 0usize;
+ for _ in 0..200 {
+ let (prepared, log) = capture_preflight_log(dir.path());
+ assert_eq!(
+ prepared.status.block_kind(),
+ Some(&ClusterStateBlockKind::UnsupportedFormat)
+ );
+ if !log.contains("cluster state was rejected at boot") {
+ blank += 1;
+ }
+ }
+ stop.store(true, Ordering::Relaxed);
+ poisoner.join().unwrap();
+
+ assert_eq!(
+ blank, 0,
+ "a concurrent subscriber-less preflight blanked {blank}/200 boot-diagnostic captures"
+ );
+ }
+
#[cfg(unix)]
#[test]
fn read_failure_logs_the_same_boot_diagnostic_contract() {
@@ -676,7 +816,7 @@ mod tests {
let staging_bytes = b"owned-by-another-or-interrupted-writer";
std::fs::write(&staging, staging_bytes).unwrap();
- let prepared = preflight(dir.path());
+ let prepared = preflight_serialised(dir.path());
assert!(
!prepared.status.is_writable() || live.is_none(),
"fixture must not classify as loaded format 1"
@@ -693,7 +833,7 @@ mod tests {
let staging = dir.path().join("cluster_state.tmp");
std::fs::write(&staging, b"interrupted format-1 stage").unwrap();
- let prepared = preflight(dir.path());
+ let prepared = preflight_serialised(dir.path());
assert_eq!(prepared.status, ClusterStateBootStatus::LoadedV1);
assert!(!staging.exists());
}
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 00c2c4629..172914db0 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -2042,6 +2042,60 @@ mod merge_publication_transaction_tests {
config
}
+ /// Open an engine plus one index with the background merge loop already
+ /// stopped, for a test that drives every merge itself.
+ ///
+ /// `Index::create` (index.rs:6191) and `Index::open` (index.rs:6554) both
+ /// start a 5-second merge pass, and with this module's
+ /// `min_merge_count = max_merge_count = 2` a single tick rewrites the
+ /// segment set two segments at a time. Every test here calls
+ /// `run_merge_once()` by hand, so that tick is pure interference with two
+ /// distinct failure shapes: it can win the `merge_in_progress` CAS and turn
+ /// the test's own `run_merge_once()` into `Ok(0)` (the flake `fixture`
+ /// already guarded against), and it can collapse fixture segments before the
+ /// test has even looked at them — issue #372, where
+ /// `partial_multi_batch_failure_is_reported_without_losing_either_batch`
+ /// built its index inline, never stopped the loop, and failed its
+ /// *precondition* with `left: 3, right: 4` once a full-parallelism run
+ /// stretched the body past one tick. Constructing every index in this
+ /// module here — including the one fixture that needs a custom `Config`,
+ /// through `hand_driven_index_with_config` — means the omission cannot be
+ /// reintroduced one test at a time.
+ fn hand_driven_index(dir: &TempDir, name: &str, schema: Schema) -> (Engine, Arc<Index>) {
+ hand_driven_index_with_config(config(dir), name, schema)
+ }
+
+ /// `hand_driven_index` for the fixture that needs a knob `config()` does
+ /// not set.
+ ///
+ /// Only `artifact_bytes_at_level` uses this: it varies
+ /// `compression.level` and pins `ingest_shards = 1`. It exists so that
+ /// site does not have to remember `abort_background_tasks()` on its own —
+ /// it is the one index in this module that was still being built by hand.
+ fn hand_driven_index_with_config(
+ cfg: xerj_common::config::Config,
+ name: &str,
+ schema: Schema,
+ ) -> (Engine, Arc<Index>) {
+ let engine = Engine::new(cfg).unwrap();
+ engine.create_index(name, schema).unwrap();
+ let index = engine.get_index(name).unwrap();
+ index.abort_background_tasks();
+ (engine, index)
+ }
+
+ /// Reopen `name` from `dir` with the background merge loop already stopped.
+ ///
+ /// Same contract as `hand_driven_index`: the restart half of these tests
+ /// asserts on a recovered segment set, and `Index::open` starts its own
+ /// merge pass too (`spawn_merge_task(5)`, index.rs:6554).
+ fn reopen_hand_driven(dir: &TempDir, name: &str) -> (Engine, Arc<Index>) {
+ let engine = Engine::new(config(dir)).unwrap();
+ let index = engine.get_index(name).unwrap();
+ index.abort_background_tasks();
+ (engine, index)
+ }
+
fn dot_field_schema() -> Schema {
let mut schema = Schema::empty();
schema
@@ -2063,10 +2117,7 @@ mod merge_publication_transaction_tests {
) -> (Engine, Arc<Index>, String, PathBuf) {
use sha2::{Digest, Sha256};
- let engine = Engine::new(config(dir)).unwrap();
- engine.create_index(index_name, dot_field_schema()).unwrap();
- let index = engine.get_index(index_name).unwrap();
- index.abort_background_tasks();
+ let (engine, index) = hand_driven_index(dir, index_name, dot_field_schema());
for (id, value) in [("one", "alpha"), ("two", "beta")] {
index
.index_document(Some(id.into()), serde_json::json!({".": value}))
@@ -2164,10 +2215,7 @@ mod merge_publication_transaction_tests {
schema
.add_field(FieldConfig::new("body", FieldType::Text))
.unwrap();
- let engine = Engine::new(cfg).unwrap();
- engine.create_index("compression", schema).unwrap();
- let index = engine.get_index("compression").unwrap();
- index.abort_background_tasks();
+ let (_engine, index) = hand_driven_index_with_config(cfg, "compression", schema);
// Two flushes → two segments to merge. Each is comfortably over
// `V2_MIN_DOCS` (128), so the stored section takes the columnar v2
@@ -2338,11 +2386,13 @@ mod merge_publication_transaction_tests {
}
async fn fixture(dir: &TempDir) -> (Engine, Arc<Index>) {
- let engine = Engine::new(config(dir)).unwrap();
- engine
- .create_index("post-save-panic", Schema::empty())
- .unwrap();
- let index = engine.get_index("post-save-panic").unwrap();
+ // `hand_driven_index` stops the background merge loop *before* the two
+ // flushes below, not after them. Stopping it afterwards still left the
+ // `segments.len() == 2` assertion exposed to a tick that landed while
+ // the fixture was still filling — the same window that broke
+ // `partial_multi_batch_failure_is_reported_without_losing_either_batch`
+ // in #372.
+ let (engine, index) = hand_driven_index(dir, "post-save-panic", Schema::empty());
index
.index_document(Some("a".into()), serde_json::json!({"value": "old-a"}))
.await
@@ -2354,14 +2404,6 @@ mod merge_publication_transaction_tests {
.unwrap();
index.flush().await.unwrap();
assert_eq!(index.store.snapshot().segments.len(), 2);
- // Every test in this module drives the merge itself with an explicit
- // `run_merge_once()`. The 5-second background merge loop is pure
- // interference here: a tick that wins the `merge_in_progress` CAS makes
- // the test's own `run_merge_once()` return `Ok(0)`, after which
- // `merge.await.unwrap_err()` fails with "called `unwrap_err()` on an
- // `Ok` value" — a second, distinct flake mode. Same precedent as
- // `cancelling_waiter_while_blocking_worker_is_queued_still_restores`.
- index.abort_background_tasks();
(engine, index)
}
@@ -2416,8 +2458,7 @@ mod merge_publication_transaction_tests {
drop(index);
drop(engine);
- let reopened = Engine::new(config(&dir)).unwrap();
- let reopened_index = reopened.get_index("post-save-panic").unwrap();
+ let (_reopened, reopened_index) = reopen_hand_driven(&dir, "post-save-panic");
for (id, expected) in [("a", "old-a"), ("b", "old-b")] {
assert_eq!(
reopened_index.get_document(id).await.unwrap().unwrap()["value"],
@@ -2498,8 +2539,7 @@ mod merge_publication_transaction_tests {
);
drop(index);
drop(engine);
- let reopened = Engine::new(config(&dir)).unwrap();
- let reopened_index = reopened.get_index("post-save-panic").unwrap();
+ let (_reopened, reopened_index) = reopen_hand_driven(&dir, "post-save-panic");
assert_eq!(
reopened_index.get_document("a").await.unwrap().unwrap()["value"],
"new-a"
@@ -2615,8 +2655,7 @@ mod merge_publication_transaction_tests {
index.set_publication_test_hook(None);
drop(index);
drop(engine);
- let reopened = Engine::new(config(&dir)).unwrap();
- let recovered = reopened.get_index("post-save-panic").unwrap();
+ let (_reopened, recovered) = reopen_hand_driven(&dir, "post-save-panic");
for id in ["a", "b"] {
assert!(recovered.get_document(id).await.unwrap().is_some());
}
@@ -2631,12 +2670,51 @@ mod merge_publication_transaction_tests {
);
}
+ /// Regression for #372: a background merge tick must not be able to move a
+ /// hand-driven fixture's segment set.
+ ///
+ /// `start_paused` is what makes this deterministic instead of a 5-second
+ /// sleep: with tokio's clock paused the runtime auto-advances to the next
+ /// timer deadline whenever it goes idle, so the `sleep` below fires exactly
+ /// the merge tick that `spawn_merge_task` scheduled — the tick that, on a
+ /// real clock, only lands when a full-parallelism run stretches a test body
+ /// past five seconds. Without `hand_driven_index`'s
+ /// `abort_background_tasks()` this collapses two of the four segments and
+ /// the assertion reports `left: 3, right: 4`, which is verbatim the failure
+ /// reported against `index.rs:2650` on clean main.
+ #[tokio::test(start_paused = true)]
+ async fn a_background_merge_tick_cannot_move_a_hand_driven_fixture_segment_set() {
+ let dir = TempDir::new().unwrap();
+ let (_engine, index) = hand_driven_index(&dir, "tick-guard", Schema::empty());
+ for id in ["a", "b", "c", "d"] {
+ index
+ .index_document(
+ Some(id.into()),
+ serde_json::json!({"value": format!("source-{id}")}),
+ )
+ .await
+ .unwrap();
+ index.flush().await.unwrap();
+ }
+ assert_eq!(index.store.snapshot().segments.len(), 4);
+ // Jump the paused clock well past the 5 s merge cadence, several times
+ // over, and hand the runtime enough turns for a tick to run to
+ // completion if one were scheduled.
+ for _ in 0..4 {
+ tokio::time::sleep(std::time::Duration::from_secs(30)).await;
+ tokio::task::yield_now().await;
+ }
+ assert_eq!(
+ index.store.snapshot().segments.len(),
+ 4,
+ "a background merge tick rewrote a fixture that drives its own merges"
+ );
+ }
+
#[tokio::test]
async fn partial_multi_batch_failure_is_reported_without_losing_either_batch() {
let dir = TempDir::new().unwrap();
- let engine = Engine::new(config(&dir)).unwrap();
- engine.create_index("multi-batch", Schema::empty()).unwrap();
- let index = engine.get_index("multi-batch").unwrap();
+ let (engine, index) = hand_driven_index(&dir, "multi-batch", Schema::empty());
for id in ["a", "b", "c", "d"] {
index
.index_document(
@@ -2671,8 +2749,7 @@ mod merge_publication_transaction_tests {
drop(index);
drop(engine);
- let reopened = Engine::new(config(&dir)).unwrap();
- let reopened_index = reopened.get_index("multi-batch").unwrap();
+ let (_reopened, reopened_index) = reopen_hand_driven(&dir, "multi-batch");
for id in ["a", "b", "c", "d"] {
assert_eq!(
reopened_index.get_document(id).await.unwrap().unwrap()["value"],
@@ -2715,8 +2792,7 @@ mod merge_publication_transaction_tests {
drop(index);
drop(engine);
- let reopened = Engine::new(config(&dir)).unwrap();
- let reopened_index = reopened.get_index("post-save-panic").unwrap();
+ let (_reopened, reopened_index) = reopen_hand_driven(&dir, "post-save-panic");
assert_eq!(reopened_index.store.snapshot().segments.len(), 1);
for (id, expected) in [("a", "old-a"), ("b", "old-b")] {
assert_eq!(
diff --git a/engine/crates/xerj-engine/src/lifecycle.rs b/engine/crates/xerj-engine/src/lifecycle.rs
index 128d67be4..c34eee555 100644
--- a/engine/crates/xerj-engine/src/lifecycle.rs
+++ b/engine/crates/xerj-engine/src/lifecycle.rs
@@ -774,7 +774,22 @@ mod tests {
include_bytes!("../tests/fixtures/cluster_state/v2-format-version-minimal.json");
let state_path = dir.path().join("cluster_state.json");
std::fs::write(&state_path, future).unwrap();
- let engine = Engine::new(config).unwrap();
+ // The only reacher of the boot diagnostic outside `cluster_state::tests`
+ // in this `--lib` binary: preflight classifies this fixture as
+ // `UnsupportedFormat` and `PreparedClusterState::blocked` emits its
+ // `error!` from this thread, which has no subscriber. Whichever thread
+ // first registers that callsite decides its process-global `Interest`,
+ // and a subscriber-less registration pins it to `never`, blanking any
+ // concurrent `cluster_state::tests` capture — issue #372. Today this
+ // cannot lose that race: libtest runs a byte-sorted name list, and
+ // `cluster_state::tests::*` ends at #106 of 601 while this test is #436
+ // (measured with `--list` at this commit), so registration is long
+ // settled. The lock is what stops the guarantee from resting on that.
+ // See `cluster_state::PREFLIGHT_CALLSITE`.
+ let engine = {
+ let _serialised = crate::cluster_state::lock_preflight_callsite();
+ Engine::new(config).unwrap()
+ };
assert!(matches!(
engine.get_index("logs-1"),
Err(EngineError::Common(