Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(autoindex): scope the catalog canonical/alias barrier to its own run (#360)

xerj-org79e4b1a2bf6d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c2591c4ef..10a0ce012 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -39,6 +39,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
equality rather than failing it — the state directory left behind by the
aborted repro run resumes and commits on the fixed binary.
+- **`xerj autoindex` aborted a run because *another* corpus on the same node had
+ already indexed a byte-identical file** — the second of the three aborts
+ [#360](https://github.com/xerj-org/xerj/issues/360) reports, and the one that
+ killed the reporter's second corpus. A file's identity is derived from its
+ CONTENT alone (`ids::file_key`) and the catalog is one global index
+ (`autoindex-catalog`) that no `--prefix` namespaces, so two unrelated
+ checkouts sharing one file — an Apache-2.0 `LICENSE` is the everyday case —
+ share that file's catalog document IDs. A corpus holding the content five
+ times publishes a canonical document plus four alias documents; a corpus
+ holding it once republishes only the canonical, and the other corpus's four
+ alias documents survive under their own `run_id`. The generation-wide barrier
+ counted every catalog document carrying the group's `file_key` — across every
+ run on the node — against `1 + aliases.len()` for the group in front of it, so
+ the second run exited `1` with `catalog canonical/alias count disagrees with
+ desired group axg1-…` on a corpus whose data was complete. The count is now
+ scoped to this generation's `run_id`, the same way the run-summary read-back
+ already is, so the barrier asks what it exists to ask: did *this* generation
+ publish one canonical document and its aliases.
+
+ Measured on the reporter's own corpora, one node, release binary,
+ `aarch64-unknown-linux-gnu`. `apache/lucene` and `unum-cloud/usearch` share
+ exactly one indexable content identity — the 11,357-byte Apache-2.0 `LICENSE`,
+ present 5× in lucene and 1× in usearch. Indexing lucene and then usearch on
+ one node, each with its own `--prefix` and `--state-dir`: **before**, usearch
+ exited `1` after 41.5s with `catalog canonical/alias count disagrees with
+ desired group axg1-f5c714bcb8c945dd3687e83218352c66`. **After**, the same
+ usearch run prints `generation 1 committed — 4 datasets, 308 records live` and
+ exits `3` (junk recorded, never fatal — `cli.rs` EXIT CODES) in 30.2s, and the
+ 43 alias documents lucene published are still in the catalog: the barrier
+ stopped counting them, it does not delete them.
+
+ This does **not** close #360. The third abort the issue reports — `catalog
+ read-back for ds:… disagrees with the sealed generation projection`, which
+ ended the full `apache/lucene` run after 1,073s before this change and 663s
+ after it — reproduces on this branch and is not an autoindex defect:
+ `POST /_bulk` drops explicit `null` fields from `_source` once the request is
+ large enough (measured directly with `curl` against this binary: 1,500
+ documents / 129,786 B keeps them, 4,000 documents / 349,786 B loses them,
+ 2,000 documents is where it starts to be mixed), and autoindex's dataset
+ documents carry `"time_field": null`, `"time_min": null`, `"time_max": null`
+ and `"semantic_field": null`. The exact read-back is the tripwire that caught
+ the engine rewriting a document, so it is deliberately left armed rather than
+ taught to ignore the difference.
+
- **A lexical query on a `semantic_text` field now says so** (#363). `match`,
`match_phrase`, `multi_match` and `query_string` against a `semantic_text`
field score with BM25 over the analysed text and never consult the embedding
diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
index 2eee88018..36b85a168 100644
--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
@@ -1474,6 +1474,83 @@ fn a_readme_documenting_two_tables_is_not_a_fatal_condition() {
);
}
+/// The second abort #360 reported — the one that killed both of the corpora
+/// the reporter pointed autoindex at, on one node, one after the other.
+///
+/// `ids::file_key` derives a file's identity from its CONTENT alone, and
+/// `catalog::CATALOG_INDEX` is a single global index that no `--prefix`
+/// namespaces. Two unrelated checkouts that happen to share one byte-identical
+/// file — an Apache-2.0 `LICENSE`, a `.gitignore`, an empty `__init__.py` —
+/// therefore share that file's catalog document IDs. A corpus holding the
+/// content twice publishes a canonical document plus one alias document; a
+/// corpus holding it once republishes only the canonical, so the *other*
+/// corpus's alias document survives under its own `run_id`. The generation-wide
+/// barrier counted every catalog document carrying that `file_key`, across every
+/// run on the node, against `1 + aliases.len()` for the group in front of it —
+/// so the second run aborted with `exit=1` while its data was complete.
+#[test]
+fn a_file_shared_with_another_corpus_on_the_same_node_is_not_a_fatal_condition() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap_or_else(|p| p.into_inner());
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK
+ .lock()
+ .unwrap_or_else(|p| p.into_inner());
+ let shared = "id,value\n1,apache-2.0\n2,copied-verbatim\n";
+ let first = tempfile::tempdir().unwrap();
+ let first_state = tempfile::tempdir().unwrap();
+ fs::create_dir(first.path().join("docs")).unwrap();
+ fs::write(first.path().join("LICENSE.csv"), shared).unwrap();
+ fs::write(first.path().join("docs").join("LICENSE.csv"), shared).unwrap();
+ let endpoint = HttpEndpoint::start();
+
+ let (code, summary) =
+ run_index_report(cfg(first.path(), first_state.path(), &endpoint.url, false))
+ .expect("the first corpus indexes");
+ let summary = summary.expect("the first corpus commits its generation");
+ assert_eq!(code, 0);
+ assert_eq!(
+ summary["duplicate_files"], 1,
+ "the first corpus must publish one alias document, or nothing is left \
+ behind for the second run to trip over: {summary}"
+ );
+
+ // A second, unrelated corpus on the same node, isolated exactly the way the
+ // CLI's refusals recommend: its own --state-dir and its own --prefix. Its
+ // one shared file is a group with no aliases at all.
+ let second = tempfile::tempdir().unwrap();
+ let second_state = tempfile::tempdir().unwrap();
+ fs::write(second.path().join("LICENSE.csv"), shared).unwrap();
+ fs::write(second.path().join("own.csv"), "id,value\n7,second\n").unwrap();
+ let mut config = cfg(second.path(), second_state.path(), &endpoint.url, false);
+ config.prefix = "incremental-http-second".into();
+
+ let (code, summary) = run_index_report(config)
+ .expect("a file shared with another corpus on the same node is not a fatal condition");
+ let summary = summary.expect("the second corpus commits its generation");
+ assert_eq!(code, 0, "the second corpus is a clean run: {summary}");
+ assert_eq!(summary["generation"], 1);
+ assert_eq!(summary["records_total"], 3);
+ assert_eq!(summary["duplicate_files"], 0);
+ assert_eq!(journal_events(second_state.path(), "sync_commit"), 1);
+
+ // And the first corpus's alias document is still there — the barrier stopped
+ // counting it, it was not swept away from under the other corpus's map.
+ let aliases = endpoint
+ .state
+ .lock()
+ .unwrap()
+ .docs
+ .iter()
+ .filter(|((index, _), doc)| {
+ index == catalog::CATALOG_INDEX
+ && doc.get("status").and_then(Value::as_str) == Some("duplicate")
+ })
+ .count();
+ assert_eq!(
+ aliases, 1,
+ "the other corpus's alias document must survive this run untouched"
+ );
+}
+
/// Where this branch and #241 meet.
///
/// The generated `--no-graph` route returns from inside `run_index_report`,
diff --git a/engine/crates/xerj-autoindex/src/sync_executor.rs b/engine/crates/xerj-autoindex/src/sync_executor.rs
index 4d46bb8d4..0ae4df236 100644
--- a/engine/crates/xerj-autoindex/src/sync_executor.rs
+++ b/engine/crates/xerj-autoindex/src/sync_executor.rs
@@ -646,7 +646,7 @@ impl SyncOperationBackend for EsSyncBackend<'_> {
&mut self,
base: &CommittedManifest,
desired: &GenerationManifest,
- _snapshot: &SourceSnapshot,
+ snapshot: &SourceSnapshot,
) -> Result<()> {
for dataset in &desired.plan.datasets {
self.es.refresh(&dataset.index)?;
@@ -664,12 +664,26 @@ impl SyncOperationBackend for EsSyncBackend<'_> {
"live semantic count disagrees with desired group {}",
group.group_id
);
+ // Scoped to this generation's `run_id`, the same way the run-summary
+ // read-back is (`lib.rs`). `file_key` is derived from CONTENT alone
+ // (`ids::file_key`) and the catalog is one global index that no
+ // `--prefix` namespaces, so an unscoped count also sees the
+ // canonical and alias documents that ANOTHER corpus on this node
+ // published for byte-identical content — two Apache-2.0 checkouts
+ // sharing a LICENSE is enough. Those documents are that run's, not
+ // this one's: counting them aborted a generation whose own
+ // publication was exactly right (#360). What this barrier is for is
+ // "this generation published one canonical document and its aliases",
+ // and that is what it now asks.
let catalog = self.es.search(
crate::catalog::CATALOG_INDEX,
&serde_json::json!({
"size": 0,
"track_total_hits": true,
- "query": {"term": {"file_key": &group.content_id}}
+ "query": {"bool": {"filter": [
+ {"term": {"file_key": &group.content_id}},
+ {"term": {"run_id": &snapshot.tx_id}}
+ ]}}
}),
)?;
anyhow::ensure!(