Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

test(autoindex): pin #360 on the shape it was reported against, and log it

xerj-org11e66745b74b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f81c3c688..8dcf0875e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,7 +7,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
-_Nothing yet._
+### Fixed
+
+- **`xerj autoindex` aborted a whole run when one file declared two or more SQL
+ tables** ([#360](https://github.com/xerj-org/xerj/issues/360)). One file can
+ feed several datasets — a SQL dump is one file and N tables — but the sealed
+ record count was a single whole-file total. Finalisation charged that total to
+ *each* of the file's datasets in turn and compared it against that one
+ dataset's read-back, so any multi-table file disagreed with itself by
+ construction and exited `1` with `dataset t1 exact read-back count 2 disagrees
+ with sealed count 4`. The rows were always indexed correctly; only the
+ reconciliation arithmetic was wrong, so `exit=1` discarded a complete corpus.
+ Reported against a real `unum-cloud/usearch` file — `sqlite/README.md`,
+ ordinary prose with three ` ```sql ` blocks in it, which the sniffer routes to
+ the `sqldump` family.
+
+ The sealed ledger is now keyed by the dataset identity each record was written
+ under (`records_by_dataset` on the prepared artifact,
+ `expected_records_by_dataset` on the manifest group), the read-back is scoped
+ by `ax_dataset` as well as `ax_file`, and `validate_groups` rejects a ledger
+ that names a dataset the group does not feed or that does not sum to the group
+ total. Measured against a local node with the fixed binary: the two-table and
+ three-table repros from the issue exit `0`, every table's rows are present
+ (`_count` 2 per table), and each dataset's catalog document reports
+ `record_count: 2` instead of the whole file's total.
+
+ Existing state directories survive the upgrade: both new fields are omitted
+ from serialization when empty, so a snapshot or manifest sealed by an older
+ version re-serializes to the same bytes and keeps its digest. A pre-upgrade
+ group that fanned out has no recoverable split, so it withdraws from that one
+ equality rather than failing it — the state directory left behind by the
+ aborted repro run resumes and commits on the fixed binary.
## [1.0.0-rc.17] - 2026-08-15
diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
index 660cafde4..2eee88018 100644
--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
@@ -1409,6 +1409,71 @@ fn a_two_table_sqlite_file_commits_its_generation() {
assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);
}
+/// The shape #360 was actually reported against, and the one that reaches the
+/// fan-out through the *sniffer* rather than through an extension: ordinary
+/// markdown prose with SQL blocks in it, as in `unum-cloud/usearch`'s
+/// `sqlite/README.md`. `sniff` routes any text containing `CREATE TABLE` and a
+/// `;` to the `sqldump` family, so a README documenting two tables is one
+/// content group feeding two datasets — no dump and no database file involved.
+/// Worth pinning separately: the reporter's corpus contained neither.
+#[test]
+fn a_readme_documenting_two_tables_is_not_a_fatal_condition() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap_or_else(|p| p.into_inner());
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK
+ .lock()
+ .unwrap_or_else(|p| p.into_inner());
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(
+ corpus.path().join("README.md"),
+ "# vectors\n\nStore them like this:\n\n```sql\n\
+ CREATE TABLE t1 (id INTEGER PRIMARY KEY, v JSON NOT NULL);\n\
+ INSERT INTO t1 (id, v) VALUES (10, '[1.0]'), (11, '[2.0]');\n\
+ CREATE TABLE t2 (id INTEGER PRIMARY KEY, v JSON NOT NULL);\n\
+ INSERT INTO t2 (id, v) VALUES (20, '[1.0]'), (21, '[2.0]');\n\
+ ```\n",
+ )
+ .unwrap();
+ let endpoint = HttpEndpoint::start();
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);
+
+ let (code, summary) = run_index_report(config).unwrap();
+ let summary = summary.expect("a README describing two tables still commits");
+ assert_eq!(code, 0, "a valid README is not a failed run");
+ assert_eq!(summary["records_total"], 4);
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);
+
+ let mut counts: Vec<(String, u64)> = endpoint
+ .state
+ .lock()
+ .unwrap()
+ .docs
+ .iter()
+ .filter(|((index, _), doc)| {
+ index == catalog::CATALOG_INDEX
+ && doc.get("doc_kind").and_then(Value::as_str) == Some("dataset")
+ })
+ .map(|(_, doc)| {
+ (
+ doc["slug"].as_str().unwrap_or_default().to_owned(),
+ doc["record_count"]
+ .as_u64()
+ .expect("a dataset catalog document reports its record count"),
+ )
+ })
+ .collect();
+ counts.sort();
+ assert_eq!(
+ counts.len(),
+ 2,
+ "one dataset per declared table: {counts:?}"
+ );
+ assert!(
+ counts.iter().all(|(_, records)| *records == 2),
+ "each dataset carries only its own rows: {counts:?}"
+ );
+}
+
/// Where this branch and #241 meet.
///
/// The generated `--no-graph` route returns from inside `run_index_report`,