AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
test(autoindex): fix flaky corpus-lock tests in xc::tests (#2739)
xerj-org5ed9cb09ce1e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3020fa116..ce9ce2f21 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -97,6 +97,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0longer take a node from below the watermark into the parent breaker band andturn ingest into HTTP 429s. The relief for a search node resting between themerge-pool ceiling and the watermark is kept (#2427).+- The `xerj-autoindex` corpus-lock tests no longer fail at random in CI+ (#2520, Refs #2427). One test put `-c core.fsmonitor=false` after the git+ subcommand, so git exited 129 at once and the test only passed by winning a+ race against that exit; and under the test harness a lock another test just+ released can stay held for 10-30 ms by a sibling test's forked child until it+ execs. Test builds now wait out a busy lock for up to 500 ms; a real+ `corpus add` still refuses at once.- A search that decodes a segment's doc-values sidecar the segment cache hasno room to keep now reserves the decoded size in the read transient budgetbefore decoding, and holds it until the query drops the columns. Before,diff --git a/engine/crates/xerj-autoindex/src/xc.rs b/engine/crates/xerj-autoindex/src/xc.rsindex a7b1b9ef8..4b5449076 100644--- a/engine/crates/xerj-autoindex/src/xc.rs+++ b/engine/crates/xerj-autoindex/src/xc.rs@@ -971,14 +971,45 @@ fn lock_corpus_add(root: &Path, name: &str) -> Result<std::fs::File> {.write(true).open(&path).with_context(|| format!("cannot open {}", path.display()))?;- match fs2::FileExt::try_lock_exclusive(&file) {- Ok(()) => Ok(file),- Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => anyhow::bail!(+ let deadline = std::time::Instant::now() + lock_busy_grace();+ loop {+ match fs2::FileExt::try_lock_exclusive(&file) {+ Ok(()) => return Ok(file),+ Err(e)+ if e.kind() == std::io::ErrorKind::WouldBlock+ && std::time::Instant::now() < deadline =>+ {+ std::thread::sleep(std::time::Duration::from_millis(5));+ }+ Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => anyhow::bail!("corpus '{name}' is in use by another `xerj corpus add`, or by a git it started that \is still running (lock: {}). Run this again when it exits.",path.display()),- Err(e) => Err(anyhow::Error::new(e).context(format!("cannot lock {}", path.display()))),+ Err(e) => {+ return Err(+ anyhow::Error::new(e).context(format!("cannot lock {}", path.display()))+ );+ }+ }+ }+}++/// How long [`lock_corpus_add`] waits out a busy lock before refusing: none+/// in a real run, which refuses at once.+///+/// The unit tests run as threads of one process, and every `Command::spawn`+/// there forks a child that holds a copy of each open descriptor, a lock some+/// other test just dropped included, until that child execs (`O_CLOEXEC` only+/// closes it at `exec`). On a busy runner that keeps a released lock held for+/// 10-30 ms, and a test that locks the same corpus again in that window was+/// refused (#2520 test flake). A real `corpus add` locks once per process, so+/// it never races its own forks this way.+fn lock_busy_grace() -> std::time::Duration {+ if cfg!(test) {+ std::time::Duration::from_millis(500)+ } else {+ std::time::Duration::ZERO}}@@ -6073,14 +6104,17 @@ precedence = ["a", "b"]let lock = lock_corpus_add(home.path(), "held").unwrap();let handoff = GitLockHandoff::new(&lock);// `hash-object --stdin` blocks until its stdin closes, which this test- // never does, so a real git holds the lock until it is killed.+ // never does, so a real git holds the lock until it is killed. The+ // handoff adds `-c core.fsmonitor=false`, a global option, so it goes+ // before the subcommand as in `git_long`: after it, git exits 129 at+ // once and the refusal below only won a race against that exit.let mut child = {let mut c = Command::new("git");+ inherit_corpus_lock(&mut c);c.args(["hash-object", "--stdin"]).stdin(std::process::Stdio::piped()).stdout(std::process::Stdio::piped()).stderr(std::process::Stdio::piped());- inherit_corpus_lock(&mut c);c.spawn().unwrap()};drop(handoff);@@ -6097,6 +6131,10 @@ precedence = ["a", "b"]}}}+ assert!(+ child.try_wait().unwrap().is_none(),+ "git is still running when the lock is probed"+ );assert!(err.contains("still running"),"a running git still holds the corpus lock: {err:?}"