AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): scope the catalog canonical/alias barrier to its own run (#360)
xerj-org79e4b1a2bf6d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c2591c4ef..10a0ce012 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -39,6 +39,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0equality rather than failing it — the state directory left behind by theaborted repro run resumes and commits on the fixed binary.+- **`xerj autoindex` aborted a run because *another* corpus on the same node had+ already indexed a byte-identical file** — the second of the three aborts+ [#360](https://github.com/xerj-org/xerj/issues/360) reports, and the one that+ killed the reporter's second corpus. A file's identity is derived from its+ CONTENT alone (`ids::file_key`) and the catalog is one global index+ (`autoindex-catalog`) that no `--prefix` namespaces, so two unrelated+ checkouts sharing one file — an Apache-2.0 `LICENSE` is the everyday case —+ share that file's catalog document IDs. A corpus holding the content five+ times publishes a canonical document plus four alias documents; a corpus+ holding it once republishes only the canonical, and the other corpus's four+ alias documents survive under their own `run_id`. The generation-wide barrier+ counted every catalog document carrying the group's `file_key` — across every+ run on the node — against `1 + aliases.len()` for the group in front of it, so+ the second run exited `1` with `catalog canonical/alias count disagrees with+ desired group axg1-…` on a corpus whose data was complete. The count is now+ scoped to this generation's `run_id`, the same way the run-summary read-back+ already is, so the barrier asks what it exists to ask: did *this* generation+ publish one canonical document and its aliases.++ Measured on the reporter's own corpora, one node, release binary,+ `aarch64-unknown-linux-gnu`. `apache/lucene` and `unum-cloud/usearch` share+ exactly one indexable content identity — the 11,357-byte Apache-2.0 `LICENSE`,+ present 5× in lucene and 1× in usearch. Indexing lucene and then usearch on+ one node, each with its own `--prefix` and `--state-dir`: **before**, usearch+ exited `1` after 41.5s with `catalog canonical/alias count disagrees with+ desired group axg1-f5c714bcb8c945dd3687e83218352c66`. **After**, the same+ usearch run prints `generation 1 committed — 4 datasets, 308 records live` and+ exits `3` (junk recorded, never fatal — `cli.rs` EXIT CODES) in 30.2s, and the+ 43 alias documents lucene published are still in the catalog: the barrier+ stopped counting them, it does not delete them.++ This does **not** close #360. The third abort the issue reports — `catalog+ read-back for ds:… disagrees with the sealed generation projection`, which+ ended the full `apache/lucene` run after 1,073s before this change and 663s+ after it — reproduces on this branch and is not an autoindex defect:+ `POST /_bulk` drops explicit `null` fields from `_source` once the request is+ large enough (measured directly with `curl` against this binary: 1,500+ documents / 129,786 B keeps them, 4,000 documents / 349,786 B loses them,+ 2,000 documents is where it starts to be mixed), and autoindex's dataset+ documents carry `"time_field": null`, `"time_min": null`, `"time_max": null`+ and `"semantic_field": null`. The exact read-back is the tripwire that caught+ the engine rewriting a document, so it is deliberately left armed rather than+ taught to ignore the difference.+- **A lexical query on a `semantic_text` field now says so** (#363). `match`,`match_phrase`, `multi_match` and `query_string` against a `semantic_text`field score with BM25 over the analysed text and never consult the embeddingdiff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rsindex 2eee88018..36b85a168 100644--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs@@ -1474,6 +1474,83 @@ fn a_readme_documenting_two_tables_is_not_a_fatal_condition() {);}+/// The second abort #360 reported — the one that killed both of the corpora+/// the reporter pointed autoindex at, on one node, one after the other.+///+/// `ids::file_key` derives a file's identity from its CONTENT alone, and+/// `catalog::CATALOG_INDEX` is a single global index that no `--prefix`+/// namespaces. Two unrelated checkouts that happen to share one byte-identical+/// file — an Apache-2.0 `LICENSE`, a `.gitignore`, an empty `__init__.py` —+/// therefore share that file's catalog document IDs. A corpus holding the+/// content twice publishes a canonical document plus one alias document; a+/// corpus holding it once republishes only the canonical, so the *other*+/// corpus's alias document survives under its own `run_id`. The generation-wide+/// barrier counted every catalog document carrying that `file_key`, across every+/// run on the node, against `1 + aliases.len()` for the group in front of it —+/// so the second run aborted with `exit=1` while its data was complete.+#[test]+fn a_file_shared_with_another_corpus_on_the_same_node_is_not_a_fatal_condition() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap_or_else(|p| p.into_inner());+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK+ .lock()+ .unwrap_or_else(|p| p.into_inner());+ let shared = "id,value\n1,apache-2.0\n2,copied-verbatim\n";+ let first = tempfile::tempdir().unwrap();+ let first_state = tempfile::tempdir().unwrap();+ fs::create_dir(first.path().join("docs")).unwrap();+ fs::write(first.path().join("LICENSE.csv"), shared).unwrap();+ fs::write(first.path().join("docs").join("LICENSE.csv"), shared).unwrap();+ let endpoint = HttpEndpoint::start();++ let (code, summary) =+ run_index_report(cfg(first.path(), first_state.path(), &endpoint.url, false))+ .expect("the first corpus indexes");+ let summary = summary.expect("the first corpus commits its generation");+ assert_eq!(code, 0);+ assert_eq!(+ summary["duplicate_files"], 1,+ "the first corpus must publish one alias document, or nothing is left \+ behind for the second run to trip over: {summary}"+ );++ // A second, unrelated corpus on the same node, isolated exactly the way the+ // CLI's refusals recommend: its own --state-dir and its own --prefix. Its+ // one shared file is a group with no aliases at all.+ let second = tempfile::tempdir().unwrap();+ let second_state = tempfile::tempdir().unwrap();+ fs::write(second.path().join("LICENSE.csv"), shared).unwrap();+ fs::write(second.path().join("own.csv"), "id,value\n7,second\n").unwrap();+ let mut config = cfg(second.path(), second_state.path(), &endpoint.url, false);+ config.prefix = "incremental-http-second".into();++ let (code, summary) = run_index_report(config)+ .expect("a file shared with another corpus on the same node is not a fatal condition");+ let summary = summary.expect("the second corpus commits its generation");+ assert_eq!(code, 0, "the second corpus is a clean run: {summary}");+ assert_eq!(summary["generation"], 1);+ assert_eq!(summary["records_total"], 3);+ assert_eq!(summary["duplicate_files"], 0);+ assert_eq!(journal_events(second_state.path(), "sync_commit"), 1);++ // And the first corpus's alias document is still there — the barrier stopped+ // counting it, it was not swept away from under the other corpus's map.+ let aliases = endpoint+ .state+ .lock()+ .unwrap()+ .docs+ .iter()+ .filter(|((index, _), doc)| {+ index == catalog::CATALOG_INDEX+ && doc.get("status").and_then(Value::as_str) == Some("duplicate")+ })+ .count();+ assert_eq!(+ aliases, 1,+ "the other corpus's alias document must survive this run untouched"+ );+}+/// Where this branch and #241 meet.////// The generated `--no-graph` route returns from inside `run_index_report`,diff --git a/engine/crates/xerj-autoindex/src/sync_executor.rs b/engine/crates/xerj-autoindex/src/sync_executor.rsindex 4d46bb8d4..0ae4df236 100644--- a/engine/crates/xerj-autoindex/src/sync_executor.rs+++ b/engine/crates/xerj-autoindex/src/sync_executor.rs@@ -646,7 +646,7 @@ impl SyncOperationBackend for EsSyncBackend<'_> {&mut self,base: &CommittedManifest,desired: &GenerationManifest,- _snapshot: &SourceSnapshot,+ snapshot: &SourceSnapshot,) -> Result<()> {for dataset in &desired.plan.datasets {self.es.refresh(&dataset.index)?;@@ -664,12 +664,26 @@ impl SyncOperationBackend for EsSyncBackend<'_> {"live semantic count disagrees with desired group {}",group.group_id);+ // Scoped to this generation's `run_id`, the same way the run-summary+ // read-back is (`lib.rs`). `file_key` is derived from CONTENT alone+ // (`ids::file_key`) and the catalog is one global index that no+ // `--prefix` namespaces, so an unscoped count also sees the+ // canonical and alias documents that ANOTHER corpus on this node+ // published for byte-identical content — two Apache-2.0 checkouts+ // sharing a LICENSE is enough. Those documents are that run's, not+ // this one's: counting them aborted a generation whose own+ // publication was exactly right (#360). What this barrier is for is+ // "this generation published one canonical document and its aliases",+ // and that is what it now asks.let catalog = self.es.search(crate::catalog::CATALOG_INDEX,&serde_json::json!({"size": 0,"track_total_hits": true,- "query": {"term": {"file_key": &group.content_id}}+ "query": {"bool": {"filter": [+ {"term": {"file_key": &group.content_id}},+ {"term": {"run_id": &snapshot.tx_id}}+ ]}}}),)?;anyhow::ensure!(