Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(autoindex): enable incremental non-graph reconciliation

buger61b31ef3d8e7
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ef625a381..98cb1e89f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -386,6 +386,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
prints the funnel on demand, including which repo-level numbers are
contaminated and why.
+### Changed — fail-closed autoindex restart semantics
+
+- **`autoindex --fresh` no longer discards a durable resume plan.** A durable
+ plan contains the alias, path, graph, and stale-record knowledge needed for
+ safe reconciliation. `--fresh` is accepted only when the selected state
+ directory has no durable plan and never cleans the destination. An
+ independent rebuild must use a new state directory, new prefix, and new brain
+ namespace when graph detection is enabled (or disable graph writes), be
+ validated, and only then replace the old reader target. The shared catalog
+ and old target require explicit, validated cleanup. Generated journals with
+ `--no-graph` reconcile additions, changes, deletions, renames, and no-op
+ reruns. Legacy journals and graph-enabled generations refuse membership
+ changes before remote mutation.
+
## [1.0.0-rc.13] - 2026-08-08
### Security
@@ -655,7 +669,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
caveats. They are tracked for the following release rather than rushed into
this one.
-
### Changed — runtime-field types (can break existing mappings)
- **`mappings.runtime` is now validated against ES's runtime-field type
diff --git a/demo/usecases/autoindex/RECIPE_DRAFT.md b/demo/usecases/autoindex/RECIPE_DRAFT.md
index 023080bac..e24b4819b 100644
--- a/demo/usecases/autoindex/RECIPE_DRAFT.md
+++ b/demo/usecases/autoindex/RECIPE_DRAFT.md
@@ -203,11 +203,15 @@ done in 0.1s — 3 datasets, 5801 records live, 0 junk records, 1 junk/skipped f
just polite re-runs: in the robustness evaluation, a `kill -9` at 18 s into a
200 MB file followed by a re-run converged to **byte-identical counts across
all 9 datasets** — no duplicates. `xerj autoindex status` shows the journal
-and live index counts; `--fresh` ignores the journal and rebuilds the plan in
-place (ids stay idempotent), which is how a rerun picks up files added since
-the last one. Deleting an indexed file is refused before any destination
-change — nothing here removes the documents it already published — and the
-error names what is gone plus the recovery routes.
+and live index counts. `--fresh` starts without resume state only when the
+selected state directory has no durable plan; it never removes stale
+destination records. For an independent rebuild, use a new `--state-dir`, new
+`--prefix`, and new `--brain` when graph detection is enabled (or add
+`--no-graph`). Validate before switching readers. The shared
+`autoindex-catalog` and old target require explicit, validated cleanup.
+Generated journals with `--no-graph` reconcile additions, changes, deletions,
+renames, and no-op reruns. Legacy journals and graph-enabled generations refuse
+membership changes before remote mutation.
## Reproduce it yourself
diff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.md
index b75f062df..88b27917f 100644
--- a/docs/recipes/zero-config-indexing.md
+++ b/docs/recipes/zero-config-indexing.md
@@ -228,17 +228,14 @@ live indices at http://localhost:9280:
autoindex-catalog 9 docs
```
-`--fresh` ignores the journal and rebuilds the plan in place (ids stay
-idempotent), which is how you pick up files added since the last run.
-
-A rerun that resumes an existing plan indexes changed files and reports added
-ones as skipped. Deleting a file is refused: nothing here removes the
-documents it already published, so the rerun stops before touching the
-destination and names what is gone. Restore the file and rerun, or rebuild —
-in place by deleting the named indices and the state directory, or isolated
-under a new `--state-dir`, `--prefix` and `--brain` (or `--no-graph`),
-validated before you switch readers. The shared `autoindex-catalog` and the
-old target require explicit cleanup.
+`--fresh` starts without resume state only when the selected state directory
+has no durable plan; it never removes stale destination records. For an
+independent rebuild, use a new `--state-dir`, new `--prefix`, and new
+`--brain` when graph detection is enabled (or add `--no-graph`). Validate
+before switching readers. The shared `autoindex-catalog` and old target require
+explicit, validated cleanup. Generated journals with `--no-graph` reconcile
+additions, changes, deletions, renames, and no-op reruns. Legacy journals and
+graph-enabled generations refuse membership changes before remote mutation.
## Reproduce it yourself
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index e225b4732..6499a7f80 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -25,6 +25,7 @@ pub struct IndexCfg {
pub pdf_timeout_secs: u64,
pub bulk_mb: usize,
pub bulk_timeout_secs: u64,
+ pub snapshot_max_bytes: u64,
pub prefix: String,
pub state_dir: Option<PathBuf>,
pub fresh: bool,
@@ -75,6 +76,9 @@ pub enum Cmd {
const FRESH_HELP: &str =
"start without resume state only when the selected state directory has no \
durable plan; an existing plan is refused and destination records are never reset";
+const RESUME_POLICY_HELP: &str =
+ "generated --no-graph journals reconcile add, change, delete, rename, and no-op runs; \
+legacy journals and graph-enabled generations refuse membership changes before remote mutation";
pub fn print_help() {
println!(
@@ -103,6 +107,8 @@ pub fn print_help() {
valid range 1..=3600)\n\
--prefix <P> index prefix (default ax)\n\
--state-dir <PATH> resume journal location (default ~/.xerj/autoindex/<hash>/)\n\
+ --snapshot-max-gb <N> logical payload cap for sealed source+prepared records\n\
+ bytes (default 64); excludes filesystem/manifest overhead\n\
--fresh {fresh_help}\n\
--follow-symlinks follow symlinks (loop-safe); off by default\n\
--max-file-gb <N> skip+record oversized non-streamable files (default 2)\n\
@@ -143,6 +149,14 @@ pub fn print_help() {
--pdf-workers bounds both phase-A parser processes and phase-B replay\n\
materialization; --workers does not widen that PDF memory gate.\n\
\n\
+ INCREMENTAL RECONCILIATION:\n\
+ Generated journals with --no-graph reconcile added, removed, moved, and changed\n\
+ files. Legacy journals and graph-enabled generations remain fail-closed. Each\n\
+ changed generation currently copies and prepares the full corpus (O(N)); the\n\
+ latest full snapshot remains retained, and cleanup re-reads protected artifacts\n\
+ to verify them. --snapshot-max-gb limits logical staged payload bytes before each\n\
+ write; it is not a physical disk-space or peak-allocation guarantee.\n\
+ \n\
EMBEDDINGS:\n\
autoindex sends semantic_text to the running server; it does not choose the\n\
server's embedding backend. The default is lexical (not neural). For the\n\
@@ -170,16 +184,17 @@ pub fn print_help() {
cannot be computed honestly, never a filler number.\n\
\n\
RESUME POLICY:\n\
- A durable plan supports no-op resume and same-path content replacement.\n\
- Added or removed content groups are refused before remote mutation.\n\
- An independent rebuild needs a new --state-dir, new --prefix, and, when\n\
- graph detection is enabled, new --brain (or --no-graph). Validate before\n\
- switching readers; the shared autoindex-catalog and old target require\n\
- explicit, validated cleanup.\n\
+ {resume_policy_help}.\n\
+ If a legacy or graph-enabled journal refuses a change, do not use --fresh as\n\
+ cleanup. Restore its original destination, or rebuild with a new --state-dir and\n\
+ new --prefix plus a new --brain when graph is enabled (or use --no-graph).\n\
+ Validate before switching readers; explicitly clean the shared\n\
+ autoindex-catalog and old target only after validation.\n\
\n\
EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\
2 usage; 1 endpoint/journal failure or unsupported corpus delta\n",
- fresh_help = FRESH_HELP
+ fresh_help = FRESH_HELP,
+ resume_policy_help = RESUME_POLICY_HELP
);
}
@@ -198,6 +213,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
let mut pdf_timeout_secs = 120u64;
let mut bulk_mb = 8usize;
let mut bulk_timeout_secs = 300u64;
+ let mut snapshot_max_bytes = 64u64 << 30;
let mut bulk_timeout_explicit = false;
let mut prefix = "ax".to_string();
let mut state_dir: Option<PathBuf> = None;
@@ -272,6 +288,17 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
return Err("--bulk-timeout-secs must be in the range 1..=3600 seconds".into());
}
}
+ "--snapshot-max-gb" => {
+ let gib: u64 = it
+ .next()
+ .ok_or("--snapshot-max-gb needs a positive integer")?
+ .parse()
+ .map_err(|_| "--snapshot-max-gb needs a positive integer")?;
+ snapshot_max_bytes = gib
+ .checked_mul(1u64 << 30)
+ .filter(|bytes| *bytes > 0)
+ .ok_or("--snapshot-max-gb is too large")?;
+ }
"--in-flight" => {
let _ = it.next(); // reserved (bulks are worker-synchronous in v1)
}
@@ -407,6 +434,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
pdf_timeout_secs,
bulk_mb,
bulk_timeout_secs,
+ snapshot_max_bytes,
prefix,
state_dir,
fresh,
@@ -475,6 +503,24 @@ mod tests {
assert_eq!(index(&["data"]).bulk_timeout_secs, 300);
}
+ #[test]
+ fn snapshot_budget_defaults_and_accepts_gibibytes() {
+ assert_eq!(index(&["data"]).snapshot_max_bytes, 64u64 << 30);
+ assert_eq!(
+ index(&["data", "--snapshot-max-gb", "7"]).snapshot_max_bytes,
+ 7u64 << 30
+ );
+ for value in ["0", "nope", "18446744073709551615"] {
+ assert!(super::parse(
+ ["data", "--snapshot-max-gb", value]
+ .into_iter()
+ .map(str::to_owned)
+ .collect()
+ )
+ .is_err());
+ }
+ }
+
#[test]
fn fresh_help_warns_that_it_is_not_destination_reconciliation() {
assert!(super::FRESH_HELP.contains("no durable plan"));
@@ -482,6 +528,20 @@ mod tests {
assert!(super::FRESH_HELP.contains("destination records are never reset"));
}
+ #[test]
+ fn resume_policy_help_distinguishes_supported_and_refused_state() {
+ let help = super::RESUME_POLICY_HELP;
+ for claim in [
+ "generated --no-graph journals",
+ "add, change, delete, rename, and no-op",
+ "legacy journals",
+ "graph-enabled generations",
+ "before remote mutation",
+ ] {
+ assert!(help.contains(claim), "missing resume-policy claim: {claim}");
+ }
+ }
+
#[test]
fn graph_is_on_by_default_with_derived_brain() {
let cfg = index(&["data"]);
diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rs
index 8a228f0de..59174c85a 100644
--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs
+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs
@@ -263,6 +263,7 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
pdf_timeout_secs: 30,
bulk_mb: 8,
bulk_timeout_secs: 300,
+ snapshot_max_bytes: 64 << 30,
prefix: "ax".into(),
state_dir: Some(state_dir.to_owned()),
fresh: false,
diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
index e3e750aa8..fb2b1285a 100644
--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
@@ -133,7 +133,11 @@ impl MockEndpoint {
match listener.accept() {
Ok((stream, _)) => handle(stream, &server_state),
Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {
- if server_state.lock().unwrap().stop {
+ if server_state
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ .stop
+ {
break;
}
thread::yield_now();
@@ -157,10 +161,13 @@ impl MockEndpoint {
impl Drop for MockEndpoint {
fn drop(&mut self) {
- self.state.lock().unwrap().stop = true;
+ self.state
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ .stop = true;
// Wake the nonblocking listener even on heavily loaded test hosts.
let _ = TcpStream::connect(self.url.trim_start_matches("http://"));
- self.join.take().unwrap().join().unwrap();
+ let _ = self.join.take().unwrap().join();
}
}
@@ -281,6 +288,10 @@ fn handle(mut stream: TcpStream, state: &Arc<Mutex<MockState>>) {
.pointer("/query/term/ax_file")
.and_then(Value::as_str)
.map(str::to_owned);
+ let file_key = query
+ .pointer("/query/term/file_key")
+ .and_then(Value::as_str)
+ .map(str::to_owned);
let mut locked = state.lock().unwrap();
locked.delete_calls += 1;
if let Some(key) = ax_file {
@@ -288,17 +299,45 @@ fn handle(mut stream: TcpStream, state: &Arc<Mutex<MockState>>) {
.docs
.retain(|_, doc| doc.get("ax_file").and_then(Value::as_str) != Some(key.as_str()));
}
+ if let Some(key) = file_key {
+ locked
+ .catalog_docs
+ .retain(|_, doc| doc.get("file_key").and_then(Value::as_str) != Some(key.as_str()));
+ }
(200, json!({"deleted": 0, "failures": []}))
} else if method == "GET" && path.ends_with("/_count") {
(200, json!({"count": state.lock().unwrap().docs.len()}))
} else if method == "POST" && path.ends_with("/_search") {
- // Report the REAL number of stored docs: `run_index` now verifies
- // live counts against the journal (#195), so a mock that always
- // answered 0 hits would (rightly) fail every successful run.
- let total = state.lock().unwrap().docs.len();
+ // Report the REAL number of stored docs: `run_index` verifies live
+ // counts against the journal (#195), so a mock that always answered 0
+ // hits would (rightly) fail every successful run. The generated path
+ // additionally reads back per-file and per-catalog projections, so the
+ // count is filtered by whichever identity the query names.
+ let query: Value = serde_json::from_slice(&body).unwrap_or(Value::Null);
+ let ax_file = query
+ .pointer("/query/term/ax_file")
+ .or_else(|| query.pointer("/query/bool/must/0/term/ax_file"))
+ .and_then(Value::as_str);
+ let file_key = query
+ .pointer("/query/term/file_key")
+ .and_then(Value::as_str);
+ let locked = state.lock().unwrap();
+ let documents = if file_key.is_some() {
+ &locked.catalog_docs
+ } else {
+ &locked.docs
+ };
+ let count = documents
+ .values()
+ .filter(|doc| {
+ ax_file.is_none_or(|key| doc.get("ax_file").and_then(Value::as_str) == Some(key))
+ && file_key
+ .is_none_or(|key| doc.get("file_key").and_then(Value::as_str) == Some(key))
+ })
+ .count();
(
200,
- json!({"hits":{"total":{"value":total},"hits":[]},"aggregations":{}}),
+ json!({"hits":{"total":{"value":count},"hits":[]},"aggregations":{}}),
)
} else {
// ping, index creation/mapping, refresh, and catalog operations
@@ -326,6 +365,18 @@ fn bulk_response(body: &[u8], state: &Arc<Mutex<MockState>>) -> Value {
.and_then(|action| action.pointer("/index/_index").cloned())
.and_then(|index| index.as_str().map(str::to_owned))
.is_some_and(|index| index != catalog::CATALOG_INDEX);
+ // Graph edge bulks are accepted and discarded: this module counts *data*
+ // bulks to place injected failures, so edge bulks must not shift the
+ // numbering (the graph pipeline has its own e2e suite in detect::e2e).
+ let is_graph = lines
+ .first()
+ .and_then(|line| serde_json::from_slice::<Value>(line).ok())
+ .and_then(|action| action.pointer("/index/_index").cloned())
+ .and_then(|index| index.as_str().map(str::to_owned))
+ .is_some_and(|index| index.ends_with("-edges"));
+ if is_graph {
+ return json!({"errors": false, "items": []});
+ }
if !is_data {
// Catalog bulks mix `index` and `delete` actions (stale aliases, and
// the junk sweep of #238), so walk the NDJSON rather than assuming
@@ -448,6 +499,7 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
pdf_timeout_secs: 30,
bulk_mb: 64,
bulk_timeout_secs: 3_600,
+ snapshot_max_bytes: 64 << 30,
prefix: "failure-test".into(),
state_dir: Some(state_dir.to_owned()),
fresh: false,
@@ -455,11 +507,10 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
max_file_gb: 1,
sample: 50,
no_semantic: true,
- // These tests count data bulks to place injected failures; graph
- // edge bulks would shift that numbering, so keep detection off here
- // (the graph pipeline has its own e2e suite in detect::e2e).
+ // This module preserves legacy graph-enabled failure behavior;
+ // incremental non-graph behavior has its own HTTP suite.
brain: None,
- no_graph: true,
+ no_graph: false,
dry_run: false,
json: false,
quiet: true,
@@ -1294,25 +1345,44 @@ fn existing_completion_is_invalidated_and_partial_visibility_is_deleted_on_resum
{
let locked = endpoint.state.lock().unwrap();
assert!(locked.failed_once);
- assert!(locked.docs.len() < rows);
+ assert!(
+ locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .count()
+ < rows
+ );
}
assert_eq!(run_index(config).unwrap(), 0);
assert_eq!(file_done_count(state_dir.path()), 2);
let locked = endpoint.state.lock().unwrap();
- assert_eq!(locked.docs.len(), rows);
+ assert_eq!(
+ locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .count(),
+ rows
+ );
assert_eq!(
locked.delete_calls, 2,
"fresh publication skips delete; replacement and repair each clean once"
);
- assert!(locked.docs.values().all(|doc| {
- doc["value"]
- .as_str()
- .is_some_and(|value| value.starts_with("replacement-"))
- }));
+ assert!(locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .all(|doc| {
+ doc["value"]
+ .as_str()
+ .is_some_and(|value| value.starts_with("replacement-"))
+ }));
let mut locators: Vec<usize> = locked
.docs
.values()
+ .filter(|doc| doc.get("value").is_some())
.map(|doc| {
doc["ax_locator"]
.as_str()
@@ -1371,12 +1441,24 @@ fn resume_repairs_kills_after_plan_delete_and_final_bulk_before_file_done() {
assert_eq!(run_index(config).unwrap(), 0);
let locked = endpoint.state.lock().unwrap();
- assert_eq!(locked.docs.len(), rows, "boundary {boundary}");
- assert!(locked.docs.values().all(|doc| {
- doc["value"]
- .as_str()
- .is_some_and(|value| value.starts_with("new-"))
- }));
+ assert_eq!(
+ locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .count(),
+ rows,
+ "boundary {boundary}"
+ );
+ assert!(locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .all(|doc| {
+ doc["value"]
+ .as_str()
+ .is_some_and(|value| value.starts_with("new-"))
+ }));
}
}
@@ -1394,7 +1476,7 @@ fn file_done_append_and_fsync_failures_are_fatal_and_resume_repairs() {
assert_eq!(run_index(config.clone()).unwrap(), 0);
fs::write(&path, "id,value\n0,new\n1,new\n").unwrap();
- state::fail_next_file_done_io(io_boundary);
+ state::fail_next_file_done_io(io_boundary, &state_dir.path().join("journal.ndjson"));
let error = run_index(config.clone()).unwrap_err();
let message = format!("{error:#}");
assert!(
@@ -1426,10 +1508,18 @@ fn file_done_append_and_fsync_failures_are_fatal_and_resume_repairs() {
assert_eq!(run_index(config).unwrap(), 0);
let locked = endpoint.state.lock().unwrap();
- assert_eq!(locked.docs.len(), 2);
+ assert_eq!(
+ locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .count(),
+ 2
+ );
assert!(locked
.docs
.values()
+ .filter(|doc| doc.get("value").is_some())
.all(|doc| doc["value"].as_str() == Some("new")));
}
}
@@ -1497,10 +1587,18 @@ fn pending_generation_b_is_superseded_when_source_changes_to_c() {
assert_eq!(starts.last().copied(), Some(committed_generation));
drop(replay_c);
let locked = endpoint.state.lock().unwrap();
- assert_eq!(locked.docs.len(), 2);
+ assert_eq!(
+ locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .count(),
+ 2
+ );
assert!(locked
.docs
.values()
+ .filter(|doc| doc.get("value").is_some())
.all(|doc| doc["value"].as_str() == Some("generation-c")));
}
@@ -1533,9 +1631,12 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {
assert_unsupported_delta_without_remote_mutation(&endpoint, config.clone(), &["b.csv"], &[]);
{
let locked = endpoint.state.lock().unwrap();
- assert_eq!(locked.docs.len(), 2);
+ assert_eq!(locked.docs.len(), 3);
assert!(locked.docs.values().all(|doc| {
- doc["ax_path"].as_str() == Some("a.csv") && doc["value"].as_str() == Some("original")
+ doc["ax_path"].as_str() == Some("a.csv")
+ && doc
+ .get("value")
+ .is_none_or(|value| value.as_str() == Some("original"))
}));
}
let replay = state::Journal::open(
@@ -1559,11 +1660,10 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {
assert_unsupported_delta_without_remote_mutation(&endpoint, config, &["b.csv"], &[]);
assert_eq!(event_count(state_dir.path(), "plan"), plans_before);
let locked = endpoint.state.lock().unwrap();
- assert_eq!(locked.docs.len(), 2);
- assert!(locked
- .docs
- .values()
- .all(|doc| doc["value"].as_str() == Some("original")));
+ assert_eq!(locked.docs.len(), 3);
+ assert!(locked.docs.values().all(|doc| doc
+ .get("value")
+ .is_none_or(|value| value.as_str() == Some("original"))));
}
/// #238: a file added after the plan was frozen is skipped and reported in the
@@ -1723,7 +1823,7 @@ fn partial_file_done_is_rolled_back_before_another_worker_commits() {
fs::write(corpus.path().join("a.csv"), "id,value\n0,a-new\n1,a-new\n").unwrap();
fs::write(corpus.path().join("b.csv"), "id,value\n2,b-new\n3,b-new\n").unwrap();
- state::fail_next_file_done_io(3);
+ state::fail_next_file_done_io(3, &state_dir.path().join("journal.ndjson"));
let error = run_index(config.clone()).unwrap_err();
assert!(format!("{error:#}").contains("partial file_done write"));
@@ -1745,12 +1845,23 @@ fn partial_file_done_is_rolled_back_before_another_worker_commits() {
assert_eq!(run_index(config).unwrap(), 0);
let locked = endpoint.state.lock().unwrap();
- assert_eq!(locked.docs.len(), 4);
- assert!(locked.docs.values().all(|doc| {
- doc["value"]
- .as_str()
- .is_some_and(|value| value.ends_with("-new"))
- }));
+ assert_eq!(
+ locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .count(),
+ 4
+ );
+ assert!(locked
+ .docs
+ .values()
+ .filter(|doc| doc.get("value").is_some())
+ .all(|doc| {
+ doc["value"]
+ .as_str()
+ .is_some_and(|value| value.ends_with("-new"))
+ }));
}
/// #195: an index write block rejects every bulk item with status 403 (the
diff --git a/engine/crates/xerj-autoindex/src/generation_catalog.rs b/engine/crates/xerj-autoindex/src/generation_catalog.rs
new file mode 100644
index 000000000..c78df338c
--- /dev/null
+++ b/engine/crates/xerj-autoindex/src/generation_catalog.rs
@@ -0,0 +1,608 @@
+//! Exact catalog projection for one committed corpus generation.
+//!
+//! Data-group operations are intentionally not catalog authority. A generation
+//! which only adds junk, removes an alias, or changes one of many files still
+//! has to publish a complete agent-facing data map. This module builds that
+//! complete, retry-stable projection without performing HTTP I/O.
+
+use crate::catalog;
+use crate::sync::{CommittedManifest, GenerationManifest, SyncOperationKind};
+use anyhow::{Context, Result};
+use serde_json::{json, Value};
+use std::collections::{BTreeMap, BTreeSet};
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct DatasetCatalogStats {
+ pub record_count: u64,
+ pub junk_records: u64,
+ pub bytes: u64,
+ pub formats: Vec<String>,
+ pub time_min: Option<String>,
+ pub time_max: Option<String>,
+ pub sample_queries: Vec<Value>,
+ pub notes: Vec<String>,
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct GenerationCatalogMetadata {
+ /// Stable ID sealed into the pending generation. This is the `run_id`
+ /// attached to every catalog document in this projection.
+ pub generation_id: String,
+ /// Stable timestamp sealed before publication. Retries must reuse it.
+ pub started: String,
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct CatalogProjection {
+ pub generation_id: String,
+ /// Complete desired managed document set, keyed by catalog `_id`.
+ pub documents: BTreeMap<String, Value>,
+ /// IDs managed by the previous generation which no longer exist.
+ pub stale_ids: BTreeSet<String>,
+}
+
+impl CatalogProjection {
+ /// Validate an exact read-back of all desired managed documents.
+ ///
+ /// The HTTP backend should fetch these IDs after refresh and pass the
+ /// resulting `_source` values here. Extra unrelated catalog documents are
+ /// outside this corpus projection and therefore are not accepted through
+ /// this API.
+ pub fn validate_observed(&self, observed: &BTreeMap<String, Value>) -> Result<()> {
+ anyhow::ensure!(
+ observed.len() == self.documents.len(),
+ "catalog read-back contains {} documents; expected {}",
+ observed.len(),
+ self.documents.len()
+ );
+ for (id, expected) in &self.documents {
+ let actual = observed
+ .get(id)
+ .with_context(|| format!("catalog read-back is missing {id}"))?;
+ anyhow::ensure!(
+ actual == expected,
+ "catalog read-back for {id} disagrees with the sealed generation projection"
+ );
+ }
+ Ok(())
+ }
+}
+
+/// Build the complete catalog projection for `desired`.
+///
+/// `dataset_stats` must be exact, generation-bound values obtained from the
+/// sealed prepared artifacts or from a refreshed exact read-back. Supplying
+/// sampled estimates here would make the data map lie.
+///
+/// `prior_managed_ids` is the exact set observed through the authenticated
+/// catalog endpoint for the prior committed catalog generation. The manifest
+/// cannot derive correlation IDs, so callers must query or durably persist
+/// this set.
+pub fn project_generation(
+ _base: &CommittedManifest,
+ desired: &GenerationManifest,
+ metadata: &GenerationCatalogMetadata,
+ dataset_stats: &BTreeMap<String, DatasetCatalogStats>,
+ correlations: &BTreeMap<String, Value>,
+ prior_managed_ids: &BTreeSet<String>,
+) -> Result<CatalogProjection> {
+ anyhow::ensure!(
+ !metadata.generation_id.is_empty() && !metadata.started.is_empty(),
+ "catalog generation metadata is incomplete"
+ );
+ let execution = desired
+ .execution
+ .as_ref()
+ .context("catalog generation has no execution identity")?;
+ let mut documents = BTreeMap::new();
+
+ for group in &desired.groups {
+ let assignment = desired.plan.files.get(&group.content_id).with_context(|| {
+ format!(
+ "catalog group {} has no desired file assignment",
+ group.group_id
+ )
+ })?;
+ let format = assignment_format(assignment.family.as_str(), assignment.gzip);
+ let (id, doc) = catalog::file_doc(
+ &group.content_id,
+ &group.canonical.rel,
+ &format,
+ "indexed",
+ None,
+ group.expected_records,
+ 0,
+ group.content_size,
+ &metadata.generation_id,
+ );
+ insert_unique(&mut documents, id, doc)?;
+ for alias in &group.aliases {
+ let (id, doc) = catalog::duplicate_file_doc(
+ &group.content_id,
+ &alias.rel,
+ &alias.path_id,
+ &group.canonical.rel,
+ group.content_size,
+ &metadata.generation_id,
+ );
+ insert_unique(&mut documents, id, doc)?;
+ }
+ }
+
+ for junk in &desired.plan.junk_files {
+ let (id, doc) = catalog::file_doc(
+ &junk.file_key,
+ &junk.rel,
+ &junk.format,
+ &junk.status,
+ Some(&junk.reason),
+ 0,
+ 0,
+ junk.bytes,
+ &metadata.generation_id,
+ );
+ insert_unique(&mut documents, id, doc)?;
+ }
+
+ let mut total_records = 0u64;
+ let mut total_junk_records = 0u64;
+ for dataset in &desired.plan.datasets {
+ let stats = dataset_stats.get(&dataset.slug).with_context(|| {
+ format!(
+ "catalog generation has no exact statistics for dataset {}",
+ dataset.slug
+ )
+ })?;
+ total_records = total_records
+ .checked_add(stats.record_count)
+ .context("catalog total record count overflow")?;
+ total_junk_records = total_junk_records
+ .checked_add(stats.junk_records)
+ .context("catalog total junk-record count overflow")?;
+ let mut formats = stats.formats.clone();
+ formats.sort();
+ formats.dedup();
+ let (id, doc) = catalog::dataset_doc(&catalog::DatasetDocInput {
+ pd: dataset,
+ record_count: stats.record_count,
+ junk_records: stats.junk_records,
+ bytes: stats.bytes,
+ file_count: dataset.file_count,
+ formats,
+ time_min: stats.time_min.clone(),
+ time_max: stats.time_max.clone(),
+ sample_queries: stats.sample_queries.clone(),
+ notes: stats.notes.clone(),
+ run_id: &metadata.generation_id,
+ });
+ insert_unique(&mut documents, id, doc)?;
+ }
+
+ for (id, correlation) in correlations {
+ let mut correlation = correlation.clone();
+ anyhow::ensure!(
+ correlation.get("doc_kind").and_then(Value::as_str) == Some("correlation"),
+ "catalog correlation {id} has the wrong doc_kind"
+ );
+ correlation["run_id"] = Value::String(metadata.generation_id.clone());
+ insert_unique(&mut documents, id.clone(), correlation)?;
+ }
+
+ let files_total = desired
+ .groups
+ .iter()
+ .map(|group| 1usize + group.aliases.len())
+ .sum::<usize>()
+ .checked_add(desired.plan.junk_files.len())
+ .context("catalog file count overflow")?;
+ let run_doc = json!({
+ "doc_kind": "run",
+ "run_id": metadata.generation_id,
+ "root": execution.root_identity,
+ "url": execution.url,
+ "prefix": execution.prefix,
+ "started": metadata.started,
+ "generation": desired.generation,
+ "files_total": files_total,
+ "unique_content_files": desired.groups.len(),
+ "files_indexed": desired.groups.len(),
+ "duplicate_files": desired.plan.duplicate_files.len(),
+ "files_junk": desired.plan.junk_files.len(),
+ "records_total": total_records,
+ "junk_records_total": total_junk_records,
+ "semantic": desired
+ .plan
+ .datasets
+ .iter()
+ .any(|dataset| dataset.semantic_field.is_some()),
+ });
+ insert_unique(
+ &mut documents,
+ format!("run:{}", metadata.generation_id),
+ run_doc,
+ )?;
+
+ anyhow::ensure!(
+ documents.values().all(|doc| {
+ doc.get("run_id").and_then(Value::as_str) == Some(metadata.generation_id.as_str())
+ }),
+ "every desired catalog document must carry the current generation run_id"
+ );
+
+ let desired_ids: BTreeSet<String> = documents.keys().cloned().collect();
+ let stale_ids = prior_managed_ids
+ .difference(&desired_ids)
+ .filter(|id| !desired_ids.contains(*id))
+ .cloned()
+ .collect();
+
+ Ok(CatalogProjection {
+ generation_id: metadata.generation_id.clone(),
+ documents,
+ stale_ids,
+ })
+}
+
+/// Provenance rule used by the publication backend.
+///
+/// Upserts publish newly prepared bytes and therefore must carry the desired
+/// generation ID. Metadata-only operations must preserve the existing
+/// document `ax_run`; deletes leave no live provenance value.
+#[cfg_attr(not(test), allow(dead_code))]
+pub fn expected_ax_run<'a>(
+ kind: &SyncOperationKind,
+ desired_generation_id: &'a str,
+ existing_ax_run: Option<&'a str>,
+) -> Result<Option<&'a str>> {
+ match kind {
+ SyncOperationKind::Upsert => Ok(Some(desired_generation_id)),
+ SyncOperationKind::Metadata => existing_ax_run
+ .map(Some)
+ .context("metadata-only publication has no existing ax_run to preserve"),
+ SyncOperationKind::Delete => Ok(None),
+ }
+}
+
+#[cfg(test)]
+fn managed_non_run_ids(plan: &crate::state::Plan) -> BTreeSet<String> {
+ let mut ids = BTreeSet::new();
+ ids.extend(plan.files.keys().map(|key| format!("file:{key}")));
+ ids.extend(
+ plan.junk_files
+ .iter()
+ .map(|junk| format!("file:{}", junk.file_key)),
+ );
+ ids.extend(
+ plan.duplicate_files
+ .iter()
+ .map(|alias| catalog::duplicate_file_id(&alias.file_key, &alias.rel, &alias.path_id)),
+ );
+ ids.extend(
+ plan.datasets
+ .iter()
+ .map(|dataset| format!("ds:{}", dataset.slug)),
+ );
+ ids
+}
+
+fn assignment_format(family: &str, gzip: bool) -> String {
+ if gzip {
+ format!("{family}(gzip)")
+ } else {
+ family.to_owned()
+ }
+}
+
+fn insert_unique(documents: &mut BTreeMap<String, Value>, id: String, doc: Value) -> Result<()> {
+ anyhow::ensure!(
+ documents.insert(id.clone(), doc).is_none(),
+ "catalog projection contains duplicate document ID {id}"
+ );
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::state::{DuplicateFile, FileAssignment, JunkFile, Plan, PlanDataset};
+ use crate::sync::{
+ ExecutionIdentity, ManifestGroup, ManifestPath, SourceExecutionPolicy,
+ EXECUTION_IDENTITY_VERSION,
+ };
+ fn unix_path_id(path: &str) -> String {
+ let encoded = path
+ .as_bytes()
+ .iter()
+ .map(|byte| format!("{byte:02x}"))
+ .collect::<String>();
+ format!("unix:{encoded}")
+ }
+
+ fn dataset() -> PlanDataset {
+ PlanDataset {
+ slug: "reports".into(),
+ index: "ax-reports".into(),
+ family: "csv".into(),
+ group: None,
+ specs: Vec::new(),
+ time_field: None,
+ semantic_field: None,
+ sampled_records: 2,
+ file_count: 1,
+ }
+ }
+
+ fn assignment(path: &str) -> FileAssignment {
+ FileAssignment {
+ rel: path.into(),
+ path_id: unix_path_id(path),
+ is_symlink: Some(false),
+ family: "csv".into(),
+ gzip: false,
+ content_digest: Some("digest".into()),
+ assignments: vec![(None, "reports".into())],
+ }
+ }
+
+ fn group(key: &str, path: &str, aliases: Vec<ManifestPath>) -> ManifestGroup {
+ ManifestGroup {
+ group_id: format!("group-{key}"),
+ content_id: key.into(),
+ content_digest: "digest".into(),
+ content_size: 10,
+ canonical: ManifestPath {
+ path_id: unix_path_id(path),
+ rel: path.into(),
+ is_symlink: false,
+ },
+ aliases,
+ dataset_slugs: vec!["reports".into()],
+ expected_records: 2,
+ expected_passages: 0,
+ expected_vectors: 0,
+ }
+ }
+
+ fn execution(generation_id: &str) -> ExecutionIdentity {
+ ExecutionIdentity {
+ version: EXECUTION_IDENTITY_VERSION,
+ root_identity: "/corpus".into(),
+ url: "http://engine".into(),
+ prefix: "ax".into(),
+ follow_symlinks: false,
+ chunker_identity: "prepared-records-v1".into(),
+ embedding_identity_sha256: "a".repeat(64),
+ embedding_backend: "lexical".into(),
+ embedding_dimension: 384,
+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),
+ embedding_resumable: true,
+ graph_enabled: false,
+ brain: "none".into(),
+ detector_identity: "disabled".into(),
+ schema_identity: "schema".into(),
+ index_identity: "index".into(),
+ source_policy: SourceExecutionPolicy::DurableSnapshot {
+ reference: format!("sync-snapshots/{generation_id}"),
+ snapshot_digest: "snapshot".into(),
+ },
+ }
+ }
+
+ fn manifest(
+ generation: u64,
+ id: &str,
+ plan: Plan,
+ groups: Vec<ManifestGroup>,
+ ) -> GenerationManifest {
+ GenerationManifest {
+ generation,
+ execution: Some(execution(id)),
+ plan,
+ groups,
+ }
+ }
+
+ fn committed(generation: GenerationManifest) -> CommittedManifest {
+ CommittedManifest {
+ generation: generation.generation,
+ manifest_digest: "base".into(),
+ plan: generation.plan,
+ groups: generation.groups,
+ execution: generation.execution,
+ }
+ }
+
+ fn stats() -> BTreeMap<String, DatasetCatalogStats> {
+ BTreeMap::from([(
+ "reports".into(),
+ DatasetCatalogStats {
+ record_count: 2,
+ junk_records: 0,
+ bytes: 10,
+ formats: vec!["csv".into()],
+ time_min: None,
+ time_max: None,
+ sample_queries: Vec::new(),
+ notes: Vec::new(),
+ },
+ )])
+ }
+
+ fn metadata(id: &str) -> GenerationCatalogMetadata {
+ GenerationCatalogMetadata {
+ generation_id: id.into(),
+ started: "2026-08-03T00:00:00Z".into(),
+ }
+ }
+
+ #[test]
+ fn every_current_document_moves_to_latest_catalog_generation() {
+ let mut base_plan = Plan {
+ datasets: vec![dataset()],
+ ..Plan::default()
+ };
+ base_plan
+ .files
+ .insert("keep".into(), assignment("keep.csv"));
+ base_plan
+ .files
+ .insert("change".into(), assignment("change.csv"));
+ let base_generation = manifest(
+ 1,
+ "generation-1",
+ base_plan.clone(),
+ vec![
+ group("keep", "keep.csv", vec![]),
+ group("change", "change.csv", vec![]),
+ ],
+ );
+ let base = committed(base_generation);
+
+ let desired = manifest(
+ 2,
+ "generation-2",
+ base_plan,
+ vec![
+ group("keep", "keep.csv", vec![]),
+ group("change", "change.csv", vec![]),
+ ],
+ );
+ let projection = project_generation(
+ &base,
+ &desired,
+ &metadata("generation-2"),
+ &stats(),
+ &BTreeMap::new(),
+ &managed_non_run_ids(&base.plan),
+ )
+ .unwrap();
+
+ assert!(projection.documents.len() >= 4);
+ assert!(projection
+ .documents
+ .values()
+ .all(|doc| { doc.get("run_id").and_then(Value::as_str) == Some("generation-2") }));
+ assert_eq!(
+ projection.documents["file:keep"]["run_id"], "generation-2",
+ "an unchanged peer remains visible in the latest data map"
+ );
+ }
+
+ #[test]
+ fn junk_transitions_and_removed_aliases_have_exact_stale_ids() {
+ let alias = ManifestPath {
+ path_id: unix_path_id("alias.csv"),
+ rel: "alias.csv".into(),
+ is_symlink: false,
+ };
+ let duplicate = DuplicateFile {
+ file_key: "indexed".into(),
+ rel: alias.rel.clone(),
+ path_id: alias.path_id.clone(),
+ is_symlink: Some(false),
+ duplicate_of: "indexed.csv".into(),
+ bytes: 10,
+ };
+ let mut base_plan = Plan {
+ datasets: vec![dataset()],
+ duplicate_files: vec![duplicate.clone()],
+ ..Plan::default()
+ };
+ base_plan
+ .files
+ .insert("indexed".into(), assignment("indexed.csv"));
+ base_plan.junk_files.push(JunkFile {
+ file_key: "old-junk".into(),
+ rel: "old.bin".into(),
+ format: "binary".into(),
+ status: "junk".into(),
+ reason: "binary".into(),
+ bytes: 10,
+ });
+ let base = committed(manifest(
+ 1,
+ "generation-1",
+ base_plan,
+ vec![group("indexed", "indexed.csv", vec![alias])],
+ ));
+
+ let mut desired_plan = Plan {
+ datasets: vec![dataset()],
+ ..Plan::default()
+ };
+ desired_plan.junk_files.push(JunkFile {
+ file_key: "indexed".into(),
+ rel: "indexed.csv".into(),
+ format: "binary".into(),
+ status: "junk".into(),
+ reason: "became binary".into(),
+ bytes: 10,
+ });
+ let desired = manifest(2, "generation-2", desired_plan, Vec::new());
+ let mut no_dataset_stats = stats();
+ no_dataset_stats.get_mut("reports").unwrap().record_count = 0;
+ let projection = project_generation(
+ &base,
+ &desired,
+ &metadata("generation-2"),
+ &no_dataset_stats,
+ &BTreeMap::new(),
+ &managed_non_run_ids(&base.plan),
+ )
+ .unwrap();
+
+ assert_eq!(projection.documents["file:indexed"]["status"], "junk");
+ assert!(projection.stale_ids.contains("file:old-junk"));
+ assert!(projection.stale_ids.contains(&catalog::duplicate_file_id(
+ &duplicate.file_key,
+ &duplicate.rel,
+ &duplicate.path_id
+ )));
+ assert!(!projection.stale_ids.contains("file:indexed"));
+ }
+
+ #[test]
+ fn provenance_contract_changes_only_upsert_ax_run() {
+ assert_eq!(
+ expected_ax_run(&SyncOperationKind::Upsert, "g2", Some("g1")).unwrap(),
+ Some("g2")
+ );
+ assert_eq!(
+ expected_ax_run(&SyncOperationKind::Metadata, "g2", Some("g1")).unwrap(),
+ Some("g1")
+ );
+ assert_eq!(
+ expected_ax_run(&SyncOperationKind::Delete, "g2", Some("g1")).unwrap(),
+ None
+ );
+ assert!(expected_ax_run(&SyncOperationKind::Metadata, "g2", None).is_err());
+ }
+
+ #[test]
+ fn exact_readback_rejects_missing_or_different_documents() {
+ let plan = Plan {
+ datasets: vec![dataset()],
+ ..Plan::default()
+ };
+ let base = committed(manifest(1, "g1", plan.clone(), Vec::new()));
+ let desired = manifest(2, "g2", plan, Vec::new());
+ let projection = project_generation(
+ &base,
+ &desired,
+ &metadata("g2"),
+ &stats(),
+ &BTreeMap::new(),
+ &managed_non_run_ids(&base.plan),
+ )
+ .unwrap();
+ projection.validate_observed(&projection.documents).unwrap();
+
+ let mut missing = projection.documents.clone();
+ missing.pop_first();
+ assert!(projection.validate_observed(&missing).is_err());
+
+ let mut changed = projection.documents.clone();
+ changed.get_mut("ds:reports").unwrap()["record_count"] = json!(999);
+ assert!(projection.validate_observed(&changed).is_err());
+ }
+}
diff --git a/engine/crates/xerj-autoindex/src/generation_catalog_http_tests.rs b/engine/crates/xerj-autoindex/src/generation_catalog_http_tests.rs
new file mode 100644
index 000000000..82ae54453
--- /dev/null
+++ b/engine/crates/xerj-autoindex/src/generation_catalog_http_tests.rs
@@ -0,0 +1,519 @@
+//! Backend-contract tests for whole-generation catalog publication.
+//!
+//! The in-memory endpoint models the observable HTTP contract: a bulk response
+//! may be accepted remotely and lost locally, so applying one exact projection
+//! twice must converge. Correlation IDs are supplied as independently observed
+//! prior-generation IDs because the current committed manifest does not persist
+//! them; a real backend must query them by the prior `run_id`.
+
+use crate::catalog;
+use crate::generation_catalog::{
+ expected_ax_run, project_generation, CatalogProjection, DatasetCatalogStats,
+ GenerationCatalogMetadata,
+};
+use crate::state::{DuplicateFile, FileAssignment, JunkFile, Plan, PlanDataset};
+use crate::sync::{
+ CommittedManifest, ExecutionIdentity, GenerationManifest, ManifestGroup, ManifestPath,
+ SourceExecutionPolicy, SyncOperationKind, EXECUTION_IDENTITY_VERSION,
+};
+use anyhow::Result;
+use serde_json::{json, Value};
+use std::collections::{BTreeMap, BTreeSet};
+
+#[derive(Default)]
+struct CatalogEndpoint {
+ documents: BTreeMap<String, Value>,
+ fail_after_accept_once: bool,
+}
+
+impl CatalogEndpoint {
+ fn publish(
+ &mut self,
+ projection: &CatalogProjection,
+ independently_observed_stale: &BTreeSet<String>,
+ ) -> Result<()> {
+ for id in projection
+ .stale_ids
+ .iter()
+ .chain(independently_observed_stale)
+ {
+ self.documents.remove(id);
+ }
+ self.documents.extend(projection.documents.clone());
+ if std::mem::take(&mut self.fail_after_accept_once) {
+ anyhow::bail!("response lost after the endpoint accepted the publication");
+ }
+ Ok(())
+ }
+
+ fn desired_readback(&self, projection: &CatalogProjection) -> BTreeMap<String, Value> {
+ projection
+ .documents
+ .keys()
+ .filter_map(|id| self.documents.get(id).cloned().map(|doc| (id.clone(), doc)))
+ .collect()
+ }
+}
+
+fn path_id(path: &str) -> String {
+ let encoded = path
+ .as_bytes()
+ .iter()
+ .map(|byte| format!("{byte:02x}"))
+ .collect::<String>();
+ format!("unix:{encoded}")
+}
+
+fn dataset() -> PlanDataset {
+ PlanDataset {
+ slug: "reports".into(),
+ index: "ax-reports".into(),
+ family: "csv".into(),
+ group: None,
+ specs: Vec::new(),
+ time_field: None,
+ semantic_field: None,
+ sampled_records: 2,
+ file_count: 0,
+ }
+}
+
+fn assignment(path: &str) -> FileAssignment {
+ FileAssignment {
+ rel: path.into(),
+ path_id: path_id(path),
+ is_symlink: Some(false),
+ family: "csv".into(),
+ gzip: false,
+ content_digest: Some(format!("digest-{path}")),
+ assignments: vec![(None, "reports".into())],
+ }
+}
+
+fn group(key: &str, path: &str, aliases: &[&str]) -> ManifestGroup {
+ ManifestGroup {
+ group_id: format!("group-{key}"),
+ content_id: key.into(),
+ content_digest: format!("digest-{path}"),
+ content_size: 20,
+ canonical: ManifestPath {
+ path_id: path_id(path),
+ rel: path.into(),
+ is_symlink: false,
+ },
+ aliases: aliases
+ .iter()
+ .map(|alias| ManifestPath {
+ path_id: path_id(alias),
+ rel: (*alias).into(),
+ is_symlink: false,
+ })
+ .collect(),
+ dataset_slugs: vec!["reports".into()],
+ expected_records: 2,
+ expected_passages: 0,
+ expected_vectors: 0,
+ }
+}
+
+fn junk(key: &str, path: &str, reason: &str) -> JunkFile {
+ JunkFile {
+ file_key: key.into(),
+ rel: path.into(),
+ format: "binary".into(),
+ status: "junk".into(),
+ reason: reason.into(),
+ bytes: 9,
+ }
+}
+
+fn execution(generation_id: &str) -> ExecutionIdentity {
+ ExecutionIdentity {
+ version: EXECUTION_IDENTITY_VERSION,
+ root_identity: "/finance".into(),
+ url: "http://engine".into(),
+ prefix: "ax".into(),
+ follow_symlinks: false,
+ chunker_identity: "prepared-records-v1".into(),
+ embedding_identity_sha256: "a".repeat(64),
+ embedding_backend: "lexical".into(),
+ embedding_dimension: 384,
+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),
+ embedding_resumable: true,
+ graph_enabled: false,
+ brain: "none".into(),
+ detector_identity: "disabled".into(),
+ schema_identity: "schema".into(),
+ index_identity: "index".into(),
+ source_policy: SourceExecutionPolicy::DurableSnapshot {
+ reference: format!("sync-snapshots/{generation_id}"),
+ snapshot_digest: format!("snapshot-{generation_id}"),
+ },
+ }
+}
+
+fn generation(
+ number: u64,
+ generation_id: &str,
+ mut plan: Plan,
+ groups: Vec<ManifestGroup>,
+) -> GenerationManifest {
+ for dataset in &mut plan.datasets {
+ dataset.file_count = groups
+ .iter()
+ .filter(|group| group.dataset_slugs.contains(&dataset.slug))
+ .count();
+ }
+ GenerationManifest {
+ generation: number,
+ execution: Some(execution(generation_id)),
+ plan,
+ groups,
+ }
+}
+
+fn committed(generation: GenerationManifest) -> CommittedManifest {
+ CommittedManifest {
+ generation: generation.generation,
+ manifest_digest: format!("manifest-{}", generation.generation),
+ plan: generation.plan,
+ groups: generation.groups,
+ execution: generation.execution,
+ }
+}
+
+fn metadata(id: &str) -> GenerationCatalogMetadata {
+ GenerationCatalogMetadata {
+ generation_id: id.into(),
+ started: "2026-08-03T12:00:00Z".into(),
+ }
+}
+
+fn stats(records: u64, bytes: u64) -> BTreeMap<String, DatasetCatalogStats> {
+ BTreeMap::from([(
+ "reports".into(),
+ DatasetCatalogStats {
+ record_count: records,
+ junk_records: 0,
+ bytes,
+ formats: vec!["csv".into()],
+ time_min: None,
+ time_max: None,
+ sample_queries: vec![json!({"class": "full_text"})],
+ notes: Vec::new(),
+ },
+ )])
+}
+
+fn plan(indexed: &[(&str, &str)], junk_files: Vec<JunkFile>) -> Plan {
+ let mut plan = Plan {
+ datasets: vec![dataset()],
+ junk_files,
+ ..Plan::default()
+ };
+ for (key, path) in indexed {
+ plan.files.insert((*key).into(), assignment(path));
+ }
+ plan
+}
+
+fn prior_managed_ids(base: &CommittedManifest) -> BTreeSet<String> {
+ let mut ids = BTreeSet::new();
+ ids.extend(base.plan.files.keys().map(|key| format!("file:{key}")));
+ ids.extend(
+ base.plan
+ .junk_files
+ .iter()
+ .map(|junk| format!("file:{}", junk.file_key)),
+ );
+ ids.extend(
+ base.plan
+ .duplicate_files
+ .iter()
+ .map(|alias| catalog::duplicate_file_id(&alias.file_key, &alias.rel, &alias.path_id)),
+ );
+ ids.extend(
+ base.plan
+ .datasets
+ .iter()
+ .map(|dataset| format!("ds:{}", dataset.slug)),
+ );
+ ids
+}
+
+#[test]
+fn junk_only_add_change_and_delete_converge_without_hiding_unchanged_data() {
+ let base_generation = generation(
+ 1,
+ "g1",
+ plan(&[("keep", "keep.csv")], vec![]),
+ vec![group("keep", "keep.csv", &[])],
+ );
+ let base = committed(base_generation);
+ let mut endpoint = CatalogEndpoint::default();
+
+ let added = generation(
+ 2,
+ "g2",
+ plan(
+ &[("keep", "keep.csv")],
+ vec![junk("junk-a", "opaque.bin", "binary")],
+ ),
+ vec![group("keep", "keep.csv", &[])],
+ );
+ let projection = project_generation(
+ &base,
+ &added,
+ &metadata("g2"),
+ &stats(2, 20),
+ &BTreeMap::new(),
+ &prior_managed_ids(&base),
+ )
+ .unwrap();
+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();
+ assert_eq!(endpoint.documents["file:junk-a"]["status"], "junk");
+ assert_eq!(endpoint.documents["file:keep"]["run_id"], "g2");
+
+ let base = committed(added);
+ let changed = generation(
+ 3,
+ "g3",
+ plan(
+ &[("keep", "keep.csv")],
+ vec![junk("junk-a", "opaque.bin", "unsupported archive")],
+ ),
+ vec![group("keep", "keep.csv", &[])],
+ );
+ let projection = project_generation(
+ &base,
+ &changed,
+ &metadata("g3"),
+ &stats(2, 20),
+ &BTreeMap::new(),
+ &prior_managed_ids(&base),
+ )
+ .unwrap();
+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();
+ assert_eq!(
+ endpoint.documents["file:junk-a"]["reason"],
+ "unsupported archive"
+ );
+
+ let base = committed(changed);
+ let deleted = generation(
+ 4,
+ "g4",
+ plan(&[("keep", "keep.csv")], vec![]),
+ vec![group("keep", "keep.csv", &[])],
+ );
+ let projection = project_generation(
+ &base,
+ &deleted,
+ &metadata("g4"),
+ &stats(2, 20),
+ &BTreeMap::new(),
+ &prior_managed_ids(&base),
+ )
+ .unwrap();
+ assert!(projection.stale_ids.contains("file:junk-a"));
+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();
+ assert!(!endpoint.documents.contains_key("file:junk-a"));
+ assert_eq!(endpoint.documents["file:keep"]["run_id"], "g4");
+}
+
+#[test]
+fn indexed_and_junk_transitions_replace_the_same_catalog_identity() {
+ let indexed = generation(
+ 1,
+ "g1",
+ plan(&[("same", "report.csv")], vec![]),
+ vec![group("same", "report.csv", &[])],
+ );
+ let base = committed(indexed);
+ let became_junk = generation(
+ 2,
+ "g2",
+ plan(
+ &[],
+ vec![junk("same", "report.csv", "no records extracted")],
+ ),
+ Vec::new(),
+ );
+ let mut empty_stats = stats(0, 0);
+ empty_stats.get_mut("reports").unwrap().formats.clear();
+ let junk_projection = project_generation(
+ &base,
+ &became_junk,
+ &metadata("g2"),
+ &empty_stats,
+ &BTreeMap::new(),
+ &prior_managed_ids(&base),
+ )
+ .unwrap();
+ assert_eq!(junk_projection.documents["file:same"]["status"], "junk");
+ assert!(!junk_projection.stale_ids.contains("file:same"));
+
+ let base = committed(became_junk);
+ let indexed_again = generation(
+ 3,
+ "g3",
+ plan(&[("same", "report.csv")], vec![]),
+ vec![group("same", "report.csv", &[])],
+ );
+ let projection = project_generation(
+ &base,
+ &indexed_again,
+ &metadata("g3"),
+ &stats(2, 20),
+ &BTreeMap::new(),
+ &prior_managed_ids(&base),
+ )
+ .unwrap();
+ assert_eq!(projection.documents["file:same"]["status"], "indexed");
+ assert!(!projection.stale_ids.contains("file:same"));
+}
+
+#[test]
+fn accepted_response_retry_is_idempotent_and_exactly_validated() {
+ let base = committed(generation(
+ 1,
+ "g1",
+ plan(&[("a", "a.csv")], vec![]),
+ vec![group("a", "a.csv", &[])],
+ ));
+ let desired = generation(
+ 2,
+ "g2",
+ plan(&[("a", "a.csv"), ("b", "b.csv")], vec![]),
+ vec![group("a", "a.csv", &[]), group("b", "b.csv", &[])],
+ );
+ let projection = project_generation(
+ &base,
+ &desired,
+ &metadata("g2"),
+ &stats(4, 40),
+ &BTreeMap::new(),
+ &prior_managed_ids(&base),
+ )
+ .unwrap();
+ let mut endpoint = CatalogEndpoint {
+ fail_after_accept_once: true,
+ ..CatalogEndpoint::default()
+ };
+ assert!(endpoint.publish(&projection, &BTreeSet::new()).is_err());
+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();
+ projection
+ .validate_observed(&endpoint.desired_readback(&projection))
+ .unwrap();
+}
+
+#[test]
+fn exact_run_dataset_file_ids_and_payloads_are_generation_bound() {
+ let base = committed(generation(1, "g1", plan(&[], vec![]), Vec::new()));
+ let desired = generation(
+ 2,
+ "g2",
+ plan(&[("report", "report.csv")], vec![]),
+ vec![group("report", "report.csv", &[])],
+ );
+ let projection = project_generation(
+ &base,
+ &desired,
+ &metadata("g2"),
+ &stats(2, 20),
+ &BTreeMap::new(),
+ &prior_managed_ids(&base),
+ )
+ .unwrap();
+
+ assert_eq!(
+ projection
+ .documents
+ .keys()
+ .cloned()
+ .collect::<BTreeSet<_>>(),
+ BTreeSet::from(["ds:reports".into(), "file:report".into(), "run:g2".into()])
+ );
+ assert_eq!(projection.documents["run:g2"]["records_total"], 2);
+ assert_eq!(projection.documents["run:g2"]["files_total"], 1);
+ assert_eq!(projection.documents["ds:reports"]["record_count"], 2);
+ assert_eq!(projection.documents["file:report"]["records"], 2);
+ assert!(projection
+ .documents
+ .values()
+ .all(|doc| doc["run_id"] == "g2"));
+}
+
+#[test]
+fn stale_alias_and_independently_observed_correlation_are_deleted() {
+ let alias = DuplicateFile {
+ file_key: "report".into(),
+ rel: "copy.csv".into(),
+ path_id: path_id("copy.csv"),
+ is_symlink: Some(false),
+ duplicate_of: "report.csv".into(),
+ bytes: 20,
+ };
+ let mut base_plan = plan(&[("report", "report.csv")], vec![]);
+ base_plan.duplicate_files.push(alias.clone());
+ let base = committed(generation(
+ 1,
+ "g1",
+ base_plan,
+ vec![group("report", "report.csv", &["copy.csv"])],
+ ));
+ let desired = generation(
+ 2,
+ "g2",
+ plan(&[("report", "report.csv")], vec![]),
+ vec![group("report", "report.csv", &[])],
+ );
+ let stale_correlation = "corr:reports:old".to_string();
+ let mut observed_prior = prior_managed_ids(&base);
+ observed_prior.insert(stale_correlation.clone());
+ let projection = project_generation(
+ &base,
+ &desired,
+ &metadata("g2"),
+ &stats(2, 20),
+ &BTreeMap::new(),
+ &observed_prior,
+ )
+ .unwrap();
+ let alias_id = catalog::duplicate_file_id(&alias.file_key, &alias.rel, &alias.path_id);
+ assert!(projection.stale_ids.contains(&alias_id));
+
+ let mut endpoint = CatalogEndpoint::default();
+ endpoint
+ .documents
+ .insert(alias_id.clone(), json!({"doc_kind": "file"}));
+ endpoint.documents.insert(
+ stale_correlation.clone(),
+ json!({"doc_kind": "correlation", "run_id": "g1"}),
+ );
+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();
+ assert!(!endpoint.documents.contains_key(&alias_id));
+ assert!(!endpoint.documents.contains_key(&stale_correlation));
+}
+
+#[test]
+fn metadata_publication_preserves_ax_run_while_upsert_changes_it() {
+ let mut live_document = json!({"ax_path": "old.csv", "ax_run": "g1"});
+ let preserved = expected_ax_run(
+ &SyncOperationKind::Metadata,
+ "g2",
+ live_document.get("ax_run").and_then(Value::as_str),
+ )
+ .unwrap()
+ .unwrap()
+ .to_owned();
+ live_document["ax_path"] = Value::String("renamed.csv".into());
+ live_document["ax_run"] = Value::String(preserved);
+ assert_eq!(live_document["ax_run"], "g1");
+
+ let replacement = expected_ax_run(&SyncOperationKind::Upsert, "g3", Some("g1"))
+ .unwrap()
+ .unwrap();
+ live_document["ax_run"] = Value::String(replacement.into());
+ assert_eq!(live_document["ax_run"], "g3");
+}
diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
new file mode 100644
index 000000000..231d7d288
--- /dev/null
+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
@@ -0,0 +1,612 @@
+//! Real-HTTP end-to-end coverage for incremental corpus generations.
+//!
+//! The endpoint below is deliberately stateful. It applies bulk index/update
+//! actions, delete-by-query, and exact count searches so `run_index` traverses
+//! the same HTTP client and validation barriers as a real server.
+
+use super::*;
+use std::collections::HashMap;
+use std::fs;
+use std::io::{BufRead, BufReader, Read, Write};
+use std::net::{TcpListener, TcpStream};
+use std::path::Path;
+use std::sync::{Arc, Mutex};
+use std::thread;
+
+static HTTP_E2E_LOCK: Mutex<()> = Mutex::new(());
+
+#[derive(Default)]
+struct HttpState {
+ docs: HashMap<(String, String), Value>,
+ requests: Vec<(String, String)>,
+ data_bulk_requests: usize,
+ fail_next_data_bulk: bool,
+ fail_embedding_identity: bool,
+ stop: bool,
+ embedding_identity_sha256: String,
+ saw_dataset_mapping_update: bool,
+ saw_catalog_mapping_update: bool,
+}
+
+struct HttpEndpoint {
+ url: String,
+ state: Arc<Mutex<HttpState>>,
+ join: Option<thread::JoinHandle<()>>,
+}
+
+impl HttpEndpoint {
+ fn start() -> Self {
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ listener.set_nonblocking(true).unwrap();
+ let url = format!("http://{}", listener.local_addr().unwrap());
+ let state = Arc::new(Mutex::new(HttpState {
+ embedding_identity_sha256: "a".repeat(64),
+ ..HttpState::default()
+ }));
+ let server_state = Arc::clone(&state);
+ let join = thread::spawn(move || loop {
+ match listener.accept() {
+ Ok((stream, _)) => handle_http(stream, &server_state),
+ Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {
+ if server_state.lock().unwrap().stop {
+ break;
+ }
+ thread::yield_now();
+ }
+ Err(error) => panic!("HTTP test endpoint accept failed: {error}"),
+ }
+ });
+ Self {
+ url,
+ state,
+ join: Some(join),
+ }
+ }
+
+ fn data_docs(&self) -> Vec<Value> {
+ self.state
+ .lock()
+ .unwrap()
+ .docs
+ .iter()
+ .filter(|((index, _), _)| index != catalog::CATALOG_INDEX)
+ .map(|(_, doc)| doc.clone())
+ .collect()
+ }
+
+ fn data_bulk_requests(&self) -> usize {
+ self.state.lock().unwrap().data_bulk_requests
+ }
+}
+
+impl Drop for HttpEndpoint {
+ fn drop(&mut self) {
+ self.state.lock().unwrap().stop = true;
+ let _ = TcpStream::connect(self.url.trim_start_matches("http://"));
+ self.join.take().unwrap().join().unwrap();
+ }
+}
+
+fn handle_http(mut stream: TcpStream, state: &Arc<Mutex<HttpState>>) {
+ let mut reader = BufReader::new(stream.try_clone().unwrap());
+ let mut request_line = String::new();
+ if reader.read_line(&mut request_line).unwrap() == 0 {
+ return;
+ }
+ let mut content_length = 0usize;
+ loop {
+ let mut line = String::new();
+ reader.read_line(&mut line).unwrap();
+ if line == "\r\n" || line.is_empty() {
+ break;
+ }
+ if let Some(value) = line
+ .to_ascii_lowercase()
+ .strip_prefix("content-length:")
+ .map(str::trim)
+ {
+ content_length = value.parse().unwrap();
+ }
+ }
+ let mut body = vec![0; content_length];
+ reader.read_exact(&mut body).unwrap();
+ let mut parts = request_line.split_whitespace();
+ let method = parts.next().unwrap_or("");
+ let path = parts.next().unwrap_or("");
+ state
+ .lock()
+ .unwrap()
+ .requests
+ .push((method.to_owned(), path.to_owned()));
+
+ let (status, response) = if method == "GET" && path == "/v1/embedding/identity" {
+ let locked = state.lock().unwrap();
+ if locked.fail_embedding_identity {
+ (
+ 500,
+ json!({"error": {
+ "type": "injected_identity_failure",
+ "reason": "genesis recovery boundary"
+ }}),
+ )
+ } else {
+ (
+ 200,
+ json!({"data": {
+ "version": 1,
+ "backend": "lexical",
+ "identity_sha256": locked.embedding_identity_sha256.clone(),
+ "dimensions": 384,
+ "semantic_contract": "semantic_text-derived-vector.v1",
+ "resumable": true
+ }, "took_ms": 0, "request_id": "incremental-http-test"}),
+ )
+ }
+ } else if method == "POST" && path == "/_bulk" {
+ let locked = state.lock().unwrap();
+ assert!(
+ locked.saw_dataset_mapping_update && locked.saw_catalog_mapping_update,
+ "bulk publication preceded exact generation mapping updates"
+ );
+ drop(locked);
+ bulk_http(&body, state)
+ } else if method == "POST" && path.contains("/_delete_by_query") {
+ let deleted = delete_http(path, &body, state);
+ (200, json!({"deleted": deleted, "failures": []}))
+ } else if method == "POST" && path.ends_with("/_search") {
+ (200, search_http(path, &body, state))
+ } else if method == "GET" && path.ends_with("/_count") {
+ let index = path.trim_start_matches('/').trim_end_matches("/_count");
+ let count = state
+ .lock()
+ .unwrap()
+ .docs
+ .keys()
+ .filter(|(candidate, _)| candidate == index)
+ .count();
+ (200, json!({"count": count}))
+ } else if method == "PUT" {
+ let value: Value = serde_json::from_slice(&body).unwrap_or(Value::Null);
+ let mut locked = state.lock().unwrap();
+ if path.ends_with("/_mapping") {
+ assert!(value.get("properties").is_some());
+ assert!(value.get("mappings").is_none());
+ if path.starts_with(&format!("/{}/", catalog::CATALOG_INDEX)) {
+ locked.saw_catalog_mapping_update = true;
+ } else {
+ locked.saw_dataset_mapping_update = true;
+ }
+ } else {
+ assert!(value.pointer("/mappings/properties").is_some());
+ assert!(value.get("properties").is_none());
+ }
+ (200, json!({"acknowledged": true}))
+ } else {
+ // Readiness, index creation/mapping, and refresh.
+ (200, json!({"acknowledged": true}))
+ };
+ let bytes = response.to_string();
+ write!(
+ stream,
+ "HTTP/1.1 {status} {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",
+ if status == 200 {
+ "OK"
+ } else {
+ "Internal Server Error"
+ },
+ bytes.len(),
+ bytes
+ )
+ .unwrap();
+}
+
+fn bulk_http(body: &[u8], state: &Arc<Mutex<HttpState>>) -> (u16, Value) {
+ let lines: Vec<&[u8]> = body
+ .split(|byte| *byte == b'\n')
+ .filter(|line| !line.is_empty())
+ .collect();
+ let is_data = lines.iter().step_by(2).any(|line| {
+ serde_json::from_slice::<Value>(line)
+ .ok()
+ .and_then(|action| {
+ action
+ .pointer("/index/_index")
+ .or_else(|| action.pointer("/update/_index"))
+ .and_then(Value::as_str)
+ .map(|index| index != catalog::CATALOG_INDEX)
+ })
+ .unwrap_or(false)
+ });
+ let mut locked = state.lock().unwrap();
+ if is_data {
+ locked.data_bulk_requests += 1;
+ if std::mem::take(&mut locked.fail_next_data_bulk) {
+ return (
+ 200,
+ json!({
+ "errors": true,
+ "items": [{"index": {
+ "status": 500,
+ "error": {
+ "type": "injected_failure",
+ "reason": "leave durable generation pending"
+ }
+ }}]
+ }),
+ );
+ }
+ }
+ let mut cursor = 0;
+ while cursor < lines.len() {
+ let action: Value = serde_json::from_slice(lines[cursor]).unwrap();
+ cursor += 1;
+ if let Some(meta) = action.get("delete") {
+ let index = meta["_index"].as_str().unwrap().to_owned();
+ let id = meta["_id"].as_str().unwrap().to_owned();
+ locked.docs.remove(&(index, id));
+ continue;
+ }
+ let payload: Value = serde_json::from_slice(lines[cursor]).unwrap();
+ cursor += 1;
+ if let Some(meta) = action.get("index") {
+ let index = meta["_index"].as_str().unwrap().to_owned();
+ let id = meta["_id"].as_str().unwrap().to_owned();
+ locked.docs.insert((index, id), payload);
+ } else if let Some(meta) = action.get("update") {
+ let index = meta["_index"].as_str().unwrap().to_owned();
+ let id = meta["_id"].as_str().unwrap().to_owned();
+ let patch = payload["doc"].as_object().unwrap();
+ let target = locked.docs.get_mut(&(index, id)).unwrap();
+ let target = target.as_object_mut().unwrap();
+ target.extend(patch.clone());
+ }
+ }
+ (200, json!({"errors": false, "items": []}))
+}
+
+fn delete_http(path: &str, body: &[u8], state: &Arc<Mutex<HttpState>>) -> usize {
+ let index = path
+ .trim_start_matches('/')
+ .split("/_delete_by_query")
+ .next()
+ .unwrap();
+ let query: Value = serde_json::from_slice(body).unwrap();
+ let term = query
+ .pointer("/query/term")
+ .and_then(Value::as_object)
+ .and_then(|term| term.iter().next())
+ .map(|(field, value)| (field.clone(), value.clone()));
+ let mut locked = state.lock().unwrap();
+ let before = locked.docs.len();
+ locked.docs.retain(|(candidate, _), doc| {
+ if candidate != index {
+ return true;
+ }
+ let Some((field, expected)) = &term else {
+ return false;
+ };
+ doc.get(field) != Some(expected)
+ });
+ before - locked.docs.len()
+}
+
+fn search_http(path: &str, body: &[u8], state: &Arc<Mutex<HttpState>>) -> Value {
+ let indices = path
+ .trim_start_matches('/')
+ .trim_end_matches("/_search")
+ .split(',')
+ .collect::<Vec<_>>();
+ let query: Value = serde_json::from_slice(body).unwrap();
+ let term = query
+ .pointer("/query/term")
+ .or_else(|| query.pointer("/query/bool/must/0/term"))
+ .and_then(Value::as_object)
+ .and_then(|term| term.iter().next());
+ let terms = query
+ .pointer("/query/terms")
+ .and_then(Value::as_object)
+ .and_then(|terms| terms.iter().next());
+ let filters = query
+ .pointer("/query/bool/filter")
+ .and_then(Value::as_array)
+ .cloned()
+ .unwrap_or_default();
+ let exists = query
+ .pointer("/query/bool/must/1/exists/field")
+ .and_then(Value::as_str);
+ let locked = state.lock().unwrap();
+ let matching: Vec<_> = locked
+ .docs
+ .iter()
+ .filter(|((index, _), doc)| {
+ indices.contains(&index.as_str())
+ && term
+ .map(|(field, value)| doc.get(field) == Some(value))
+ .unwrap_or(true)
+ && terms
+ .map(|(field, values)| {
+ values.as_array().is_some_and(|values| {
+ values.iter().any(|value| doc.get(field) == Some(value))
+ })
+ })
+ .unwrap_or(true)
+ && filters.iter().all(|filter| {
+ filter
+ .get("term")
+ .and_then(Value::as_object)
+ .and_then(|term| term.iter().next())
+ .is_none_or(|(field, value)| doc.get(field) == Some(value))
+ })
+ && exists.map(|field| doc.get(field).is_some()).unwrap_or(true)
+ })
+ .collect();
+ let hits = matching
+ .iter()
+ .map(|((_, id), source)| json!({"_id": id, "_source": source}))
+ .collect::<Vec<_>>();
+ json!({
+ "hits": {
+ "total": {"value": matching.len(), "relation": "eq"},
+ "hits": hits
+ },
+ "aggregations": {}
+ })
+}
+
+fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {
+ IndexCfg {
+ root: root.to_owned(),
+ url: url.to_owned(),
+ api_key: None,
+ workers: 1,
+ pdf_workers: 1,
+ pdf_timeout_secs: 30,
+ bulk_mb: 1,
+ bulk_timeout_secs: 30,
+ snapshot_max_bytes: 64 << 30,
+ prefix: "incremental-http".into(),
+ state_dir: Some(state_dir.to_owned()),
+ fresh: false,
+ follow_symlinks: false,
+ max_file_gb: 1,
+ sample: 50,
+ no_semantic: !semantic,
+ brain: None,
+ no_graph: true,
+ dry_run: false,
+ json: false,
+ quiet: true,
+ }
+}
+
+fn journal_events(state_dir: &Path, kind: &str) -> usize {
+ fs::read_to_string(state_dir.join("journal.ndjson"))
+ .unwrap()
+ .lines()
+ .filter_map(|line| serde_json::from_str::<Value>(line).ok())
+ .filter(|event| event.get("kind").and_then(Value::as_str) == Some(kind))
+ .count()
+}
+
+fn paths(docs: &[Value]) -> Vec<String> {
+ let mut paths: Vec<_> = docs
+ .iter()
+ .filter_map(|doc| doc.get("ax_path").and_then(Value::as_str))
+ .map(str::to_owned)
+ .collect();
+ paths.sort();
+ paths.dedup();
+ paths
+}
+
+fn final_snapshot_count(state_dir: &Path) -> usize {
+ fs::read_dir(state_dir.join("sync-snapshots"))
+ .map(|entries| {
+ entries
+ .filter_map(Result::ok)
+ .filter(|entry| {
+ entry.file_type().is_ok_and(|kind| kind.is_dir())
+ && !entry.file_name().to_string_lossy().starts_with('.')
+ })
+ .count()
+ })
+ .unwrap_or(0)
+}
+
+#[test]
+fn genesis_recovers_after_snapshot_budget_failure_before_sync_begin() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap();
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("a.csv"), "id,value\n1,alpha\n").unwrap();
+ let endpoint = HttpEndpoint::start();
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);
+ config.snapshot_max_bytes = 1;
+
+ let error = run_index(config.clone()).unwrap_err();
+ assert!(format!("{error:#}").contains("snapshot source footprint"));
+ assert_eq!(journal_events(state_dir.path(), "sync_bootstrap"), 1);
+ assert_eq!(journal_events(state_dir.path(), "sync_begin"), 0);
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 0);
+ assert_eq!(endpoint.data_bulk_requests(), 0);
+
+ config.snapshot_max_bytes = 64 << 30;
+ assert_eq!(run_index(config).unwrap(), 0);
+ assert_eq!(journal_events(state_dir.path(), "sync_bootstrap"), 1);
+ assert_eq!(journal_events(state_dir.path(), "sync_begin"), 1);
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);
+ assert_eq!(endpoint.data_docs().len(), 1);
+}
+
+#[test]
+fn no_semantic_generation_does_not_require_embedding_identity_endpoint() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap();
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("a.csv"), "id,value\n1,alpha\n").unwrap();
+ let endpoint = HttpEndpoint::start();
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);
+ endpoint.state.lock().unwrap().fail_embedding_identity = true;
+
+ assert_eq!(run_index(config).unwrap(), 0);
+ assert_eq!(journal_events(state_dir.path(), "sync_bootstrap"), 1);
+ assert_eq!(journal_events(state_dir.path(), "sync_begin"), 1);
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);
+ assert_eq!(journal_events(state_dir.path(), "finish"), 1);
+ assert_eq!(final_snapshot_count(state_dir.path()), 1);
+ assert_eq!(endpoint.data_docs().len(), 1);
+}
+
+#[test]
+fn initial_generation_and_noop_are_end_to_end_idempotent() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap();
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("a.csv"), "id,value\n1,alpha\n2,beta\n").unwrap();
+ let endpoint = HttpEndpoint::start();
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);
+
+ let (code, summary) = run_index_report(config.clone()).unwrap();
+ assert_eq!(code, 0);
+ let summary = summary.expect("generated run must return its committed run projection");
+ assert_eq!(summary["generation"], 1);
+ assert_eq!(summary["records_total"], 2);
+ let first_docs = endpoint.data_docs();
+ assert_eq!(first_docs.len(), 2);
+ assert_eq!(paths(&first_docs), ["a.csv"]);
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);
+ let root = corpus.path().canonicalize().unwrap();
+ let preflight = state::Journal::preflight(
+ state_dir.path(),
+ &root.to_string_lossy(),
+ &endpoint.url,
+ "incremental-http",
+ )
+ .unwrap();
+ assert!(
+ preflight.committed_manifest.is_some(),
+ "sync_commit must be visible to the next preflight"
+ );
+ drop(preflight);
+
+ let bulks = endpoint.data_bulk_requests();
+ let (_, noop_summary) = run_index_report(config).unwrap();
+ assert_eq!(noop_summary.unwrap()["generation"], 1);
+ assert_eq!(endpoint.data_docs(), first_docs);
+ assert_eq!(endpoint.data_bulk_requests(), bulks);
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);
+ assert_eq!(journal_events(state_dir.path(), "finish"), 2);
+}
+
+#[test]
+fn add_change_delete_and_rename_converge_over_real_http() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap();
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("a.csv"), "id,value\n1,alpha\n").unwrap();
+ let endpoint = HttpEndpoint::start();
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::write(corpus.path().join("b.csv"), "id,value\n2,bravo\n").unwrap();
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+ assert_eq!(paths(&endpoint.data_docs()), ["a.csv", "b.csv"]);
+
+ fs::write(
+ corpus.path().join("a.csv"),
+ "id,value\n1,alpha-new\n3,charlie\n",
+ )
+ .unwrap();
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+ assert_eq!(endpoint.data_docs().len(), 3);
+
+ fs::remove_file(corpus.path().join("b.csv")).unwrap();
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+ assert_eq!(paths(&endpoint.data_docs()), ["a.csv"]);
+
+ fs::rename(
+ corpus.path().join("a.csv"),
+ corpus.path().join("renamed.csv"),
+ )
+ .unwrap();
+ assert_eq!(run_index(config).unwrap(), 0);
+ assert_eq!(paths(&endpoint.data_docs()), ["renamed.csv"]);
+ assert_eq!(endpoint.data_docs().len(), 2);
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 5);
+}
+
+#[test]
+fn pending_generation_replays_sealed_source_after_live_source_mutates() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap();
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ let source = corpus.path().join("a.csv");
+ fs::write(&source, "id,value\n1,committed\n").unwrap();
+ let endpoint = HttpEndpoint::start();
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::write(&source, "id,value\n1,sealed-pending\n2,sealed-second\n").unwrap();
+ endpoint.state.lock().unwrap().fail_next_data_bulk = true;
+ assert!(run_index(config.clone()).is_err());
+ assert_eq!(journal_events(state_dir.path(), "sync_begin"), 2);
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);
+
+ fs::write(&source, "id,value\n9,mutated-after-sync-begin\n").unwrap();
+ assert_eq!(run_index(config).unwrap(), 0);
+ let docs = endpoint.data_docs();
+ assert_eq!(docs.len(), 2);
+ let rendered = serde_json::to_string(&docs).unwrap();
+ assert!(rendered.contains("sealed-pending"), "{rendered}");
+ assert!(rendered.contains("sealed-second"), "{rendered}");
+ assert!(!rendered.contains("mutated-after-sync-begin"), "{rendered}");
+}
+
+#[test]
+fn pending_semantic_identity_drift_rejects_before_any_further_bulk() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap();
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ let source = corpus.path().join("report.txt");
+ fs::write(
+ &source,
+ "Initial quarterly report discusses durable subscription revenue and operating income.",
+ )
+ .unwrap();
+ let endpoint = HttpEndpoint::start();
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, true);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::write(
+ &source,
+ "Updated quarterly report discusses materially stronger subscription revenue and margin.",
+ )
+ .unwrap();
+ endpoint.state.lock().unwrap().fail_next_data_bulk = true;
+ assert!(run_index(config.clone()).is_err());
+ let bulks_before = endpoint.data_bulk_requests();
+ endpoint.state.lock().unwrap().embedding_identity_sha256 = "b".repeat(64);
+
+ let error = run_index(config).unwrap_err();
+ let message = format!("{error:#}");
+ assert!(
+ message.contains("different embedding execution identity"),
+ "{message}"
+ );
+ assert!(
+ message.contains("no remote mutation was attempted"),
+ "{message}"
+ );
+ assert_eq!(
+ endpoint.data_bulk_requests(),
+ bulks_before,
+ "identity drift must be rejected before another data bulk"
+ );
+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);
+}
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index ee89b8cf7..47cf70f80 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -12,10 +12,14 @@ pub mod dataset;
pub mod detect;
pub mod esclient;
pub mod extract;
+mod generation_catalog;
+#[cfg(test)]
+mod generation_catalog_http_tests;
pub mod ids;
pub mod infer;
pub mod pool;
pub mod progress;
+mod reconcile_plan;
pub mod resources;
pub mod sniff;
pub mod state;
@@ -42,6 +46,267 @@ use progress::Progress;
use sniff::{Family, Sniffed};
use state::{DuplicateFile, FileAssignment, FileDone, JunkFile, Plan, PlanDataset};
+const PREPARED_RECORDS_IDENTITY: &str = "prepared-records-v1";
+const DOCUMENT_IDS_IDENTITY: &str = "document-ids-v1";
+const DETECTOR_DISABLED_IDENTITY: &str = "disabled";
+
+fn prepared_records_identity(cfg: &IndexCfg) -> Result<String> {
+ let value = json!({
+ "contract": PREPARED_RECORDS_IDENTITY,
+ "sample": cfg.sample,
+ "max_file_gb": cfg.max_file_gb,
+ "no_semantic": cfg.no_semantic,
+ // Worker counts are operational only. The timeout can change whether
+ // a PDF yields records, so it is part of the semantic contract.
+ "pdf_timeout_secs": cfg.pdf_timeout_secs,
+ });
+ Ok(format!(
+ "{}-{:032x}",
+ PREPARED_RECORDS_IDENTITY,
+ xxhash_rust::xxh3::xxh3_128(&serde_json::to_vec(&value)?)
+ ))
+}
+
+fn preparation_contract_digest(cfg: &IndexCfg, plan: &Plan) -> Result<String> {
+ let (schema_identity, index_identity) = generation_contract_identities(plan)?;
+ let encoded = serde_json::to_vec(&json!({
+ "prepared_records": prepared_records_identity(cfg)?,
+ "document_ids": DOCUMENT_IDS_IDENTITY,
+ "schema_identity": schema_identity,
+ "index_identity": index_identity,
+ "plan": plan,
+ }))?;
+ Ok(format!(
+ "axpc1-{:032x}",
+ xxhash_rust::xxh3::xxh3_128(&encoded)
+ ))
+}
+
+pub(crate) fn generation_contract_identities(plan: &Plan) -> Result<(String, String)> {
+ let mut datasets = plan.datasets.iter().collect::<Vec<_>>();
+ datasets.sort_by(|left, right| {
+ left.slug
+ .cmp(&right.slug)
+ .then_with(|| left.index.cmp(&right.index))
+ });
+ let schema = datasets
+ .iter()
+ .map(|dataset| {
+ json!({
+ "slug": dataset.slug,
+ "family": dataset.family,
+ "group": dataset.group,
+ "specs": dataset.specs,
+ "time_field": dataset.time_field,
+ "semantic_field": dataset.semantic_field,
+ })
+ })
+ .collect::<Vec<_>>();
+ let indices = datasets
+ .iter()
+ .map(|dataset| {
+ json!({
+ "index": dataset.index,
+ "mapping": build_mapping(&dataset.specs),
+ })
+ })
+ .collect::<Vec<_>>();
+ let digest = |label: &str, value: Value| -> Result<String> {
+ let bytes = serde_json::to_vec(&json!({"contract": label, "value": value}))?;
+ Ok(format!("{:032x}", xxhash_rust::xxh3::xxh3_128(&bytes)))
+ };
+ Ok((
+ digest(PREPARED_RECORDS_IDENTITY, Value::Array(schema))?,
+ digest(
+ DOCUMENT_IDS_IDENTITY,
+ json!({
+ "datasets": indices,
+ "catalog_index": catalog::CATALOG_INDEX,
+ "catalog_mapping": catalog::catalog_mapping(),
+ "document_ids": DOCUMENT_IDS_IDENTITY,
+ }),
+ )?,
+ ))
+}
+
+pub(crate) fn ensure_generation_mappings(es: &Es, plan: &Plan) -> Result<()> {
+ for dataset in &plan.datasets {
+ let mut create_body = build_mapping(&dataset.specs);
+ create_body["mappings"]["properties"]["ax_paths"] = json!({"type": "keyword"});
+ let update_body = json!({
+ "properties": create_body["mappings"]["properties"].clone()
+ });
+ es.ensure_index(&dataset.index, &create_body)
+ .with_context(|| format!("create generation index {}", dataset.index))?;
+ es.update_mapping(&dataset.index, &update_body)
+ .with_context(|| format!("install generation mapping for {}", dataset.index))?;
+ }
+ let mut catalog_create_body = catalog::catalog_mapping();
+ catalog_create_body["mappings"]["properties"]["duplicate_of"] = json!({"type": "keyword"});
+ let catalog_update_body = json!({
+ "properties": catalog_create_body["mappings"]["properties"].clone()
+ });
+ es.ensure_index(catalog::CATALOG_INDEX, &catalog_create_body)?;
+ es.update_mapping(catalog::CATALOG_INDEX, &catalog_update_body)
+ .context("install generation catalog mapping")
+}
+
+fn begin_non_graph_generation(
+ es: &Es,
+ journal: &mut state::Journal,
+ state_dir: &Path,
+ cfg: &IndexCfg,
+ root_identity: &str,
+ inventory: &content::Inventory,
+ plan: Plan,
+) -> Result<()> {
+ anyhow::ensure!(
+ cfg.no_graph,
+ "non-graph generation cutover requires --no-graph"
+ );
+ anyhow::ensure!(
+ journal.pending_sync.is_none(),
+ "cannot prepare over an existing pending generation"
+ );
+ if journal.committed_manifest.is_none() {
+ journal.sync_bootstrap_genesis()?;
+ }
+ let base = journal
+ .committed_manifest
+ .as_ref()
+ .context("generation cutover has no committed base")?
+ .clone();
+ let tx_id = format!("{}-g{}", journal.run_id, base.generation + 1);
+ let preparation_contract = preparation_contract_digest(cfg, &plan)?;
+ let snapshot = sync_executor::create_prepared_snapshot(
+ state_dir,
+ &tx_id,
+ inventory,
+ &plan,
+ &preparation_contract,
+ cfg.snapshot_max_bytes,
+ )?;
+ let chunker_identity = prepared_records_identity(cfg)?;
+ let semantic = plan
+ .datasets
+ .iter()
+ .any(|dataset| dataset.semantic_field.is_some());
+ let identity = if semantic {
+ let identity = es
+ .embedding_execution_identity()
+ .context("generation cutover could not pin the server embedding execution identity")?;
+ anyhow::ensure!(
+ identity.resumable,
+ "generation cutover requires a resumable embedding execution identity: {}",
+ identity
+ .non_resumable_reason
+ .as_deref()
+ .unwrap_or("the server did not provide an immutable identity")
+ );
+ journal.pin_embedding_identity(
+ &identity.identity_sha256,
+ identity.resumable,
+ identity.non_resumable_reason.as_deref(),
+ )?;
+ identity
+ } else {
+ crate::esclient::EmbeddingExecutionIdentity {
+ version: 1,
+ backend: "disabled".into(),
+ identity_sha256: "0".repeat(64),
+ dimensions: 1,
+ semantic_contract: "disabled-no-semantic-fields-v1".into(),
+ resumable: true,
+ non_resumable_reason: None,
+ }
+ };
+
+ let aliases_by_content = plan.duplicate_files.iter().fold(
+ HashMap::<&str, Vec<sync::ManifestPath>>::new(),
+ |mut aliases, alias| {
+ aliases
+ .entry(alias.file_key.as_str())
+ .or_default()
+ .push(sync::ManifestPath {
+ path_id: alias.path_id.clone(),
+ rel: alias.rel.clone(),
+ is_symlink: alias.is_symlink.unwrap_or(false),
+ });
+ aliases
+ },
+ );
+ let file_by_content = inventory
+ .keys
+ .iter()
+ .zip(&inventory.files)
+ .zip(&inventory.digests)
+ .map(|((key, file), digest)| (key.as_str(), (file, digest)))
+ .collect::<HashMap<_, _>>();
+ let mut candidates = Vec::with_capacity(plan.files.len());
+ for (content_id, assignment) in &plan.files {
+ let (file, digest) = file_by_content
+ .get(content_id.as_str())
+ .with_context(|| format!("planned content {content_id} is absent from inventory"))?;
+ let mut paths = vec![sync::ManifestPath {
+ path_id: assignment.path_id.clone(),
+ rel: assignment.rel.clone(),
+ is_symlink: assignment.is_symlink.unwrap_or(file.is_symlink),
+ }];
+ paths.extend(
+ aliases_by_content
+ .get(content_id.as_str())
+ .cloned()
+ .unwrap_or_default(),
+ );
+ candidates.push(sync::DesiredContentGroup {
+ content_id: content_id.clone(),
+ content_digest: (*digest).clone(),
+ content_size: file.size,
+ dataset_slugs: assignment
+ .assignments
+ .iter()
+ .map(|(_, slug)| slug.clone())
+ .collect(),
+ paths,
+ expected_records: 0,
+ expected_passages: 0,
+ expected_vectors: 0,
+ });
+ }
+ let mut groups = sync::reconcile_groups(&base.groups, candidates)?;
+ sync_executor::bind_prepared_counts(&mut groups, &snapshot, &inventory.keys)?;
+ let (schema_identity, index_identity) = generation_contract_identities(&plan)?;
+ let desired = sync::GenerationManifest {
+ generation: base.generation + 1,
+ execution: Some(sync::ExecutionIdentity {
+ version: sync::EXECUTION_IDENTITY_VERSION,
+ root_identity: root_identity.to_owned(),
+ url: cfg.url.clone(),
+ prefix: cfg.prefix.clone(),
+ follow_symlinks: cfg.follow_symlinks,
+ chunker_identity,
+ embedding_identity_sha256: identity.identity_sha256,
+ embedding_backend: identity.backend,
+ embedding_dimension: identity.dimensions,
+ embedding_semantic_contract: identity.semantic_contract,
+ embedding_resumable: identity.resumable,
+ graph_enabled: false,
+ brain: "disabled".into(),
+ detector_identity: DETECTOR_DISABLED_IDENTITY.into(),
+ schema_identity,
+ index_identity,
+ source_policy: sync::SourceExecutionPolicy::DurableSnapshot {
+ reference: format!("sync-snapshots/{tx_id}"),
+ snapshot_digest: snapshot.snapshot_digest,
+ },
+ }),
+ plan,
+ groups,
+ };
+ let pending = sync::PendingSync::new(tx_id, &base, desired)?;
+ journal.sync_begin(&pending)
+}
+
#[derive(Debug, PartialEq, Eq)]
struct CliErrorRoute {
exit_code: i32,
@@ -1392,6 +1657,117 @@ fn invocation_report_timestamps(
(started.to_rfc3339(), summary_generated_at.to_rfc3339())
}
+fn pin_pending_embedding_identity(
+ es: &Es,
+ journal: &mut state::Journal,
+ pending: &sync::PendingSync,
+) -> Result<()> {
+ if !pending
+ .desired
+ .plan
+ .datasets
+ .iter()
+ .any(|dataset| dataset.semantic_field.is_some())
+ {
+ return Ok(());
+ }
+ let expected = pending
+ .desired
+ .execution
+ .as_ref()
+ .context("pending semantic generation has no execution identity")?;
+ let current = es.embedding_execution_identity().context(
+ "pending semantic generation cannot verify the server embedding execution identity",
+ )?;
+ anyhow::ensure!(
+ current.resumable,
+ "pending semantic generation cannot resume because the current embedding backend is not \
+ resumable: {}; restore the original backend or rebuild with --fresh and a new --prefix",
+ current
+ .non_resumable_reason
+ .as_deref()
+ .unwrap_or("the server did not provide a stable execution identity")
+ );
+ anyhow::ensure!(
+ current.identity_sha256 == expected.embedding_identity_sha256
+ && current.backend == expected.embedding_backend
+ && current.dimensions == expected.embedding_dimension
+ && current.semantic_contract == expected.embedding_semantic_contract
+ && current.resumable == expected.embedding_resumable,
+ "pending semantic generation was prepared for a different embedding execution identity; \
+ no remote mutation was attempted. Restore the original embedding backend, or discard the \
+ pending generation and rebuild with --fresh and a new --prefix"
+ );
+ journal.pin_embedding_identity(
+ &current.identity_sha256,
+ current.resumable,
+ current.non_resumable_reason.as_deref(),
+ )
+}
+
+fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -> Result<Value> {
+ let committed = journal
+ .committed_manifest
+ .as_ref()
+ .context("generated run finished without committed generation authority")?;
+ let execution = committed
+ .execution
+ .as_ref()
+ .context("generated run finished without execution identity")?;
+ let generation = committed.generation;
+ let dataset_count = committed.plan.datasets.len();
+ let sync::SourceExecutionPolicy::DurableSnapshot { reference, .. } = &execution.source_policy
+ else {
+ anyhow::bail!("generated run does not reference a durable snapshot");
+ };
+ let run_id = reference
+ .strip_prefix("sync-snapshots/")
+ .context("generated run snapshot reference is not state-relative")?
+ .to_owned();
+ let response = es.search(
+ catalog::CATALOG_INDEX,
+ &json!({
+ "size": 2,
+ "query": {"bool": {"filter": [
+ {"term": {"run_id": &run_id}},
+ {"term": {"doc_kind": "run"}}
+ ]}}
+ }),
+ )?;
+ let hits = response
+ .pointer("/hits/hits")
+ .and_then(Value::as_array)
+ .context("generated run summary query has no hits")?;
+ anyhow::ensure!(
+ hits.len() == 1,
+ "generated run summary query returned {} documents; expected exactly one",
+ hits.len()
+ );
+ let summary = hits[0]
+ .get("_source")
+ .cloned()
+ .context("generated run summary hit has no _source")?;
+ anyhow::ensure!(
+ summary.get("generation").and_then(Value::as_u64) == Some(generation),
+ "generated run summary generation disagrees with committed authority"
+ );
+ journal.finish(&summary)?;
+ if cfg.json {
+ println!("{summary}");
+ } else if !cfg.quiet {
+ println!(
+ "generation {} committed — {} datasets, {} records live",
+ generation,
+ dataset_count,
+ summary
+ .get("records_total")
+ .and_then(Value::as_u64)
+ .unwrap_or(0)
+ );
+ }
+ Ok(summary)
+}
+
fn run_index(cfg: IndexCfg) -> Result<i32> {
run_index_report(cfg).map(|(code, _)| code)
}
@@ -1473,6 +1849,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// must never classify a path snapshot taken while another owner was
// publishing or replacing the durable plan.
let preflight = state::Journal::preflight(&state_dir, &root_str, &cfg.url, &cfg.prefix)?;
+ let genesis_recovery = preflight
+ .committed_manifest
+ .as_ref()
+ .is_some_and(|manifest| manifest.generation == 0 && manifest.groups.is_empty());
// A durable sync_begin owns the desired generation. Never rediscover and
// replan from a mutable source tree while that transaction is pending.
// Operation handlers are deliberately not enabled by this foundation
@@ -1487,6 +1867,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
cfg.bulk_timeout_secs,
cfg.fresh,
)?;
+ sync_executor::gc_snapshots(&state_dir, &journal)?;
+ let pending = journal
+ .pending_sync
+ .as_ref()
+ .context("preflight reported a pending generation but authoritative replay did not")?
+ .clone();
+ pin_pending_embedding_identity(&es, &mut journal, &pending)?;
pr.phase("replay", 0, 0);
let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);
sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;
@@ -1494,8 +1881,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// belongs to that surface, so `--progress none` stays silent and
// `--progress json` stays one parseable stream (#241).
pr.note("autoindex: resumed and committed pending corpus generation from durable source");
- pr.finish(true, 0, "completed", &[]);
- return Ok((0, None));
+ let summary = finish_generated_run(&es, &mut journal, &cfg)?;
+ return Ok((0, Some(summary)));
}
// Totals are unknown until the walk returns, so this phase honestly
// reports `pct=unknown` and proves liveness with the clock alone.
@@ -1524,33 +1911,191 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// real work, and before #241 it was minutes with no output at all.
pr.phase("hash", discovered_files.len() as u64, discovered_bytes);
let mut inventory = content::resolve_reporting(discovered_files, &|bytes| pr.item_done(bytes))?;
- if let Some(prior_plan) = preflight.plan.as_ref() {
- let comparison_keys = if cfg.fresh {
- inventory.keys.clone()
+ if cfg.no_graph && preflight.committed_manifest.is_some() && !genesis_recovery {
+ let mut journal = state::Journal::open_after_preflight(
+ preflight,
+ &root_str,
+ &cfg.url,
+ &cfg.prefix,
+ cfg.bulk_timeout_secs,
+ cfg.fresh,
+ )?;
+ sync_executor::gc_snapshots(&state_dir, &journal)?;
+ let base = journal
+ .committed_manifest
+ .as_ref()
+ .context("generated journal lost its committed manifest")?
+ .clone();
+ // Phase A's scan runs in the run's own pool and reports every file it
+ // touches, exactly as the legacy route does: `--workers` has to bound
+ // the CPU-bound phase for the knob to mean anything (#240 §2), and no
+ // file may drop out of the progress denominator (#241). Run-local PDF
+ // artifact reuse is disabled here on purpose — this route publishes
+ // from its own durable snapshot and never reaches the legacy phase B,
+ // so a retained artifact could never be replayed (#248).
+ let reconcile_budget = extract::pdf::ExtractionSpoolBudget::new(0, 0);
+ let reconcile_ctx = PhaseAContext {
+ state_dir: &state_dir,
+ budget: &reconcile_budget,
+ capacity_warning: None,
+ progress: &pr,
+ };
+ pr.phase(
+ "scan",
+ inventory.files.len() as u64,
+ inventory.files.iter().map(|file| file.size).sum(),
+ );
+ let scans: Vec<FileScan> = crate::pool::install(|| {
+ use rayon::prelude::*;
+ inventory
+ .files
+ .par_iter()
+ .zip(inventory.digests.par_iter())
+ .map(|(file, digest)| {
+ let _in_flight = pr.file(&file.rel, file.size);
+ scan_file(
+ &file.path,
+ file.size,
+ digest,
+ &reconcile_ctx,
+ cfg.sample,
+ cfg.max_file_gb,
+ )
+ })
+ .collect()
+ });
+ let plan = reconcile_plan::reconcile_plan(&inventory, &base.plan, scans)?;
+ if serde_json::to_value(&plan)? == serde_json::to_value(&base.plan)? {
+ if let Some(expected) = &base.execution {
+ let (schema_identity, index_identity) = generation_contract_identities(&plan)?;
+ anyhow::ensure!(
+ expected.root_identity == root_str
+ && expected.url == cfg.url
+ && expected.prefix == cfg.prefix
+ && expected.follow_symlinks == cfg.follow_symlinks
+ && expected.chunker_identity == prepared_records_identity(&cfg)?
+ && !expected.graph_enabled
+ && expected.brain == "disabled"
+ && expected.detector_identity == DETECTOR_DISABLED_IDENTITY
+ && expected.schema_identity == schema_identity
+ && expected.index_identity == index_identity,
+ "autoindex execution configuration changed since the committed generation; \
+ rebuild with --fresh and a new --prefix"
+ );
+ if plan
+ .datasets
+ .iter()
+ .any(|dataset| dataset.semantic_field.is_some())
+ {
+ let current = es.embedding_execution_identity()?;
+ anyhow::ensure!(
+ current.resumable
+ && current.identity_sha256 == expected.embedding_identity_sha256
+ && current.backend == expected.embedding_backend
+ && current.dimensions == expected.embedding_dimension
+ && current.semantic_contract == expected.embedding_semantic_contract,
+ "embedding execution identity changed since this autoindex journal was \
+ created; refusing to mix vector spaces. No remote mutation was attempted. \
+ Restore the original identity or rebuild with --fresh and a new --prefix"
+ );
+ }
+ }
+ let summary = finish_generated_run(&es, &mut journal, &cfg)?;
+ return Ok((0, Some(summary)));
+ }
+ begin_non_graph_generation(
+ &es,
+ &mut journal,
+ &state_dir,
+ &cfg,
+ &root_str,
+ &inventory,
+ plan,
+ )?;
+ let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);
+ sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;
+ let summary = finish_generated_run(&es, &mut journal, &cfg)?;
+ return Ok((0, Some(summary)));
+ }
+ if cfg.no_graph && preflight.plan.is_some() && !genesis_recovery {
+ let replacement_state = state_dir.with_extension("generation-v1");
+ let replacement_prefix = format!("{}-generation-v1", cfg.prefix);
+ let reasons = if preflight.legacy_migration_reasons.is_empty() {
+ "legacy journal has no complete generated-manifest authority".to_owned()
} else {
- // Ordinary resume preserves planned-key identity for supported
- // same-path replacement and legacy plans.
- select_resume_plan_keys(
- &inventory.files,
- &inventory.keys,
- prior_plan,
- &state_dir.join("journal.ndjson"),
- )?
- .into_iter()
- .zip(inventory.keys.iter())
- .map(|(planned, current)| planned.unwrap_or_else(|| current.clone()))
- .collect()
+ preflight.legacy_migration_reasons.join("; ")
};
- let delta =
- UnsupportedInventoryDelta::between(&inventory.files, &comparison_keys, prior_plan);
- if cfg.fresh {
- // Even an apparently unchanged content-key set can have alias,
- // path, graph, catalog, or partial-publication history that only
- // the old plan can reconcile. Route 1 cannot safely discard it.
- return Err(delta.into_fresh_error());
+ let mut rebuild_argv = vec![
+ "xerj".to_owned(),
+ "autoindex".to_owned(),
+ cfg.root.to_string_lossy().into_owned(),
+ "--no-graph".to_owned(),
+ "--url".to_owned(),
+ cfg.url.clone(),
+ "--state-dir".to_owned(),
+ replacement_state.to_string_lossy().into_owned(),
+ "--prefix".to_owned(),
+ replacement_prefix,
+ "--workers".to_owned(),
+ cfg.workers.to_string(),
+ "--pdf-workers".to_owned(),
+ cfg.pdf_workers.to_string(),
+ "--pdf-timeout-secs".to_owned(),
+ cfg.pdf_timeout_secs.to_string(),
+ "--bulk-mb".to_owned(),
+ cfg.bulk_mb.to_string(),
+ "--bulk-timeout-secs".to_owned(),
+ cfg.bulk_timeout_secs.to_string(),
+ "--snapshot-max-gb".to_owned(),
+ (cfg.snapshot_max_bytes >> 30).to_string(),
+ "--max-file-gb".to_owned(),
+ cfg.max_file_gb.to_string(),
+ "--sample".to_owned(),
+ cfg.sample.to_string(),
+ ];
+ if cfg.no_semantic {
+ rebuild_argv.push("--no-semantic".to_owned());
}
- if !delta.is_empty() {
- return Err(delta.into_error());
+ if cfg.follow_symlinks {
+ rebuild_argv.push("--follow-symlinks".to_owned());
+ }
+ anyhow::bail!(
+ "this state directory contains a legacy nonempty plan that cannot become generation \
+ authority: {reasons}. Start an independent rebuild using this argv JSON (no shell \
+ quoting required): {:?}. Keep XERJ_API_KEY set when the endpoint requires \
+ authentication",
+ rebuild_argv
+ );
+ }
+ if !genesis_recovery {
+ if let Some(prior_plan) = preflight.plan.as_ref() {
+ let comparison_keys = if cfg.fresh {
+ inventory.keys.clone()
+ } else {
+ // Ordinary resume preserves planned-key identity for supported
+ // same-path replacement and legacy plans.
+ select_resume_plan_keys(
+ &inventory.files,
+ &inventory.keys,
+ prior_plan,
+ &state_dir.join("journal.ndjson"),
+ )?
+ .into_iter()
+ .zip(inventory.keys.iter())
+ .map(|(planned, current)| planned.unwrap_or_else(|| current.clone()))
+ .collect()
+ };
+ let delta =
+ UnsupportedInventoryDelta::between(&inventory.files, &comparison_keys, prior_plan);
+ if cfg.fresh {
+ // Even an apparently unchanged content-key set can have alias,
+ // path, graph, catalog, or partial-publication history that only
+ // the old plan can reconcile. Route 1 cannot safely discard it.
+ return Err(delta.into_fresh_error());
+ }
+ if !delta.is_empty() {
+ return Err(delta.into_error());
+ }
}
}
let mut journal = state::Journal::open_after_preflight(
@@ -1561,7 +2106,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
cfg.bulk_timeout_secs,
cfg.fresh,
)?;
- let resumed_with_plan = journal.plan.is_some();
+ sync_executor::gc_snapshots(&state_dir, &journal)?;
+ let resumed_with_plan = journal.plan.is_some() && !genesis_recovery;
if inventory.files.is_empty() && !resumed_with_plan {
println!("no files found under {}", cfg.root.display());
pr.finish(true, 0, "no-files", &[]);
@@ -1698,10 +2244,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
plan.duplicate_files = inventory.duplicates.clone();
}
alias_paths_to_replace.extend(inventory.duplicates.iter().map(|alias| alias.rel.clone()));
- let files = inventory.files;
- let keys = inventory.keys;
- let digests = inventory.digests;
- let duplicate_files = inventory.duplicates;
+ let files = inventory.files.clone();
+ let keys = inventory.keys.clone();
+ let digests = inventory.digests.clone();
+ let duplicate_files = inventory.duplicates.clone();
let paths_discovered = files.len() + duplicate_files.len();
let planned_bytes: u64 = files.iter().map(|f| f.size).sum();
if !duplicate_files.is_empty() {
@@ -1731,20 +2277,32 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// open, phase-B workers are what hold bulk buffers. Reserving for the
// larger of the two can only make this optional accelerator hand capacity
// back — never take headroom the run still needs.
- let (pdf_spool_budget, pdf_spool_capacity_warning) =
+ //
+ // `--no-graph` is the exception, and it is a decision this rebase had to
+ // make between two landed changes. #248's artifact is a phase A→B
+ // accelerator, but every `--no-graph` route below returns from the
+ // generated path — which publishes from its own sealed snapshot and never
+ // reaches the legacy phase B. A retained artifact could therefore never be
+ // replayed, so admission is disabled instead of spooling gigabytes nothing
+ // will read. Nothing observable changes: the generated run document does
+ // not carry `pdf_extraction_reuse`.
+ let (pdf_spool_budget, pdf_spool_capacity_warning) = if cfg.no_graph {
+ (extract::pdf::ExtractionSpoolBudget::new(0, 0), None)
+ } else {
extract::pdf::ExtractionSpoolBudget::for_state_dir(
&state_dir,
cfg.workers.max(scan_threads),
cfg.pdf_workers,
cfg.bulk_mb,
- );
+ )
+ };
let phase_a_context = PhaseAContext {
state_dir: &state_dir,
budget: &pdf_spool_budget,
capacity_warning: pdf_spool_capacity_warning.as_deref(),
progress: &pr,
};
- let mut plan: Plan = if let Some(p) = journal.plan.clone() {
+ let mut plan: Plan = if let Some(p) = journal.plan.clone().filter(|_| !genesis_recovery) {
p
} else {
pr.phase("scan", files.len() as u64, planned_bytes);
@@ -1830,6 +2388,22 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
return Ok((0, None));
}
+ if cfg.no_graph && !resumed_with_plan {
+ begin_non_graph_generation(
+ &es,
+ &mut journal,
+ &state_dir,
+ &cfg,
+ &root_str,
+ &inventory,
+ plan,
+ )?;
+ let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);
+ sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;
+ let summary = finish_generated_run(&es, &mut journal, &cfg)?;
+ return Ok((0, Some(summary)));
+ }
+
if plan
.datasets
.iter()
@@ -3585,6 +4159,9 @@ fn run_status(cfg: StatusCfg) -> Result<i32> {
let mut done = 0u64;
let mut records = 0u64;
let mut finished = false;
+ let mut generation: Option<u64> = None;
+ let mut generation_files: Option<u64> = None;
+ let mut generation_records: Option<u64> = None;
let mut graph_line: Option<String> = None;
if let Ok(f) = std::fs::File::open(&jp) {
use std::io::BufRead;
@@ -3600,10 +4177,17 @@ fn run_status(cfg: StatusCfg) -> Result<i32> {
}
Some("finish") => {
finished = true;
+ generation = v.pointer("/summary/generation").and_then(Value::as_u64);
+ generation_files =
+ v.pointer("/summary/files_indexed").and_then(Value::as_u64);
+ generation_records =
+ v.pointer("/summary/records_total").and_then(Value::as_u64);
// Latest finish wins — the summary embeds the run
// doc, whose `graph` block is the edge count of
// record for this journal.
- if let Some(g) = v.pointer("/summary/graph") {
+ if let Some(g) =
+ v.pointer("/summary/graph").filter(|graph| !graph.is_null())
+ {
graph_line = Some(format!(
"graph: {} edges written to {} (brain {})",
g.get("edges_written").and_then(Value::as_u64).unwrap_or(0),
@@ -3618,12 +4202,15 @@ fn run_status(cfg: StatusCfg) -> Result<i32> {
}
}
println!(
- "journal {} — root {} — {} files done, {} records, {}",
+ "journal {} — root {} — {} files done, {} records, {}{}",
jp.display(),
root,
- done,
- records,
- if finished { "FINISHED" } else { "in progress" }
+ generation_files.unwrap_or(done),
+ generation_records.unwrap_or(records),
+ if finished { "FINISHED" } else { "in progress" },
+ generation
+ .map(|generation| format!(" (generation {generation})"))
+ .unwrap_or_default()
);
if let Some(line) = graph_line {
println!(" {line}");
@@ -4043,6 +4630,7 @@ mod map_metadata_tests {
FileAssignment {
rel: "report.dat".into(),
path_id: "path:report.dat".into(),
+ is_symlink: Some(false),
family: "json".into(),
gzip: false,
content_digest: Some("digest".into()),
@@ -4174,5 +4762,57 @@ mod map_metadata_tests {
}
}
+#[cfg(test)]
+mod generation_contract_identity_tests {
+ use super::*;
+
+ fn dataset(slug: &str, index: &str, family: &str) -> PlanDataset {
+ PlanDataset {
+ slug: slug.into(),
+ index: index.into(),
+ family: family.into(),
+ group: None,
+ specs: Vec::new(),
+ time_field: None,
+ semantic_field: None,
+ sampled_records: 0,
+ file_count: 0,
+ }
+ }
+
+ #[test]
+ fn contract_identities_are_order_independent_and_change_with_contracts() {
+ let mut first = Plan {
+ datasets: vec![
+ dataset("b", "prefix-b", "csv"),
+ dataset("a", "prefix-a", "json"),
+ ],
+ ..Plan::default()
+ };
+ let expected = generation_contract_identities(&first).unwrap();
+ first.datasets.reverse();
+ assert_eq!(generation_contract_identities(&first).unwrap(), expected);
+
+ first.datasets[0].family = "text".into();
+ let schema_changed = generation_contract_identities(&first).unwrap();
+ assert_ne!(schema_changed.0, expected.0);
+
+ first.datasets[0].family = "json".into();
+ first.datasets[0].index = "different-a".into();
+ let index_changed = generation_contract_identities(&first).unwrap();
+ assert_eq!(index_changed.0, expected.0);
+ assert_ne!(index_changed.1, expected.1);
+ }
+
+ #[test]
+ fn internal_contract_versions_are_explicit() {
+ assert_eq!(PREPARED_RECORDS_IDENTITY, "prepared-records-v1");
+ assert_eq!(DOCUMENT_IDS_IDENTITY, "document-ids-v1");
+ assert_eq!(DETECTOR_DISABLED_IDENTITY, "disabled");
+ }
+}
+
#[cfg(test)]
mod failure_resume_http_tests;
+#[cfg(test)]
+mod incremental_reconcile_http_tests;
diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rs
new file mode 100644
index 000000000..a8026438b
--- /dev/null
+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs
@@ -0,0 +1,587 @@
+//! Deterministic projection of a changed inventory onto a frozen typed plan.
+//!
+//! Fresh-run clustering is intentionally not reused here: cluster formation
+//! and slug election depend on corpus order and membership. Incremental runs
+//! must preserve the committed dataset/schema identity and fail closed when a
+//! file would require a new dataset or mapping.
+
+use crate::content::Inventory;
+use crate::infer::{FieldAcc, FieldSpec};
+use crate::state::{FileAssignment, JunkFile, Plan, PlanDataset};
+use crate::FileScan;
+use anyhow::{Context, Result};
+use std::cmp::Ordering;
+use std::collections::{HashMap, HashSet};
+
+/// Project the current, byte-verified inventory onto committed dataset schemas.
+///
+/// Assignment precedence is stable content identity, then stable native path
+/// identity, then deterministic schema matching. Dataset definitions are
+/// copied byte-for-byte except for their derived `file_count`.
+pub(crate) fn reconcile_plan(
+ inventory: &Inventory,
+ previous: &Plan,
+ scans: Vec<FileScan>,
+) -> Result<Plan> {
+ anyhow::ensure!(
+ inventory.files.len() == inventory.keys.len()
+ && inventory.files.len() == inventory.digests.len()
+ && inventory.files.len() == scans.len(),
+ "inventory and scan cardinalities disagree"
+ );
+
+ let datasets: HashMap<&str, &PlanDataset> = previous
+ .datasets
+ .iter()
+ .map(|dataset| (dataset.slug.as_str(), dataset))
+ .collect();
+ anyhow::ensure!(
+ datasets.len() == previous.datasets.len(),
+ "frozen plan has duplicate dataset slugs"
+ );
+
+ let mut prior_path_owner: HashMap<&str, (&str, &FileAssignment)> = HashMap::new();
+ for (content_id, assignment) in &previous.files {
+ anyhow::ensure!(
+ !assignment.path_id.is_empty(),
+ "frozen assignment {} has no native path identity",
+ assignment.rel
+ );
+ anyhow::ensure!(
+ prior_path_owner
+ .insert(&assignment.path_id, (content_id, assignment))
+ .is_none(),
+ "frozen plan assigns one native path identity more than once"
+ );
+ }
+ for alias in &previous.duplicate_files {
+ let assignment = previous
+ .files
+ .get(&alias.file_key)
+ .with_context(|| format!("frozen alias {} has no canonical assignment", alias.rel))?;
+ anyhow::ensure!(
+ !alias.path_id.is_empty(),
+ "frozen alias {} has no native path identity",
+ alias.rel
+ );
+ anyhow::ensure!(
+ prior_path_owner
+ .insert(&alias.path_id, (&alias.file_key, assignment))
+ .is_none(),
+ "frozen plan assigns one native path identity more than once"
+ );
+ }
+
+ let mut files = HashMap::new();
+ let mut junk_files = Vec::new();
+ for (((file, content_id), content_digest), scan) in inventory
+ .files
+ .iter()
+ .zip(&inventory.keys)
+ .zip(&inventory.digests)
+ .zip(scans)
+ {
+ if let Some((status, reason)) = scan.junk {
+ junk_files.push(JunkFile {
+ file_key: content_id.clone(),
+ rel: file.rel.clone(),
+ format: crate::format_str(scan.sniffed.as_ref()),
+ status,
+ reason,
+ bytes: file.size,
+ });
+ continue;
+ }
+ let sniffed = scan
+ .sniffed
+ .as_ref()
+ .with_context(|| format!("{} has sketches but no detected family", file.rel))?;
+ anyhow::ensure!(
+ !scan.sketches.is_empty(),
+ "{} has no dataset sketches and was not classified as junk",
+ file.rel
+ );
+
+ let content_owner = previous.files.get(content_id);
+ let path_owner = prior_path_owner.get(file.rel_id.as_str()).copied();
+ if let (Some(content), Some((path_content_id, path))) = (content_owner, path_owner) {
+ anyhow::ensure!(
+ std::ptr::eq(content, path)
+ || previous
+ .files
+ .get(path_content_id)
+ .is_some_and(|candidate| std::ptr::eq(candidate, content)),
+ "{} has conflicting committed content and path owners",
+ file.rel
+ );
+ }
+ let retained = content_owner.or_else(|| path_owner.map(|(_, assignment)| assignment));
+
+ let mut assignments = Vec::with_capacity(scan.sketches.len());
+ for (group, fields, _records) in &scan.sketches {
+ let slug = if let Some(owner) = retained {
+ retained_slug(owner, group, sniffed.family.as_str(), fields, &datasets)
+ .with_context(|| format!("project retained file {}", file.rel))?
+ } else {
+ classify_new(group, sniffed.family.as_str(), fields, &previous.datasets)
+ .with_context(|| format!("project new file {}", file.rel))?
+ };
+ assignments.push((group.clone(), slug));
+ }
+ assignments.sort();
+ assignments.dedup();
+ anyhow::ensure!(
+ assignments.len() == scan.sketches.len(),
+ "{} has duplicate dataset groups",
+ file.rel
+ );
+ let replaced = files.insert(
+ content_id.clone(),
+ FileAssignment {
+ rel: file.rel.clone(),
+ path_id: file.rel_id.clone(),
+ is_symlink: Some(file.is_symlink),
+ family: sniffed.family.as_str().to_owned(),
+ gzip: sniffed.gzip,
+ content_digest: Some(content_digest.clone()),
+ assignments,
+ },
+ );
+ anyhow::ensure!(
+ replaced.is_none(),
+ "byte-verified inventory contains duplicate content identity {content_id}"
+ );
+ }
+
+ let live_content: HashSet<&str> = files.keys().map(String::as_str).collect();
+ let mut duplicate_files = inventory
+ .duplicates
+ .iter()
+ .filter(|alias| live_content.contains(alias.file_key.as_str()))
+ .cloned()
+ .collect::<Vec<_>>();
+ duplicate_files.sort_by(|left, right| {
+ left.file_key
+ .cmp(&right.file_key)
+ .then_with(|| left.path_id.cmp(&right.path_id))
+ .then_with(|| left.rel.cmp(&right.rel))
+ });
+ junk_files.sort_by(|left, right| {
+ left.file_key
+ .cmp(&right.file_key)
+ .then_with(|| left.rel.cmp(&right.rel))
+ });
+
+ let mut frozen_datasets = previous.datasets.clone();
+ for dataset in &mut frozen_datasets {
+ dataset.file_count = files
+ .values()
+ .filter(|assignment| {
+ assignment
+ .assignments
+ .iter()
+ .any(|(_, slug)| slug == &dataset.slug)
+ })
+ .count();
+ }
+
+ Ok(Plan {
+ datasets: frozen_datasets,
+ files,
+ junk_files,
+ duplicate_files,
+ alias_paths_indexed: previous.alias_paths_indexed,
+ })
+}
+
+fn retained_slug(
+ owner: &FileAssignment,
+ group: &Option<String>,
+ family: &str,
+ fields: &HashMap<String, FieldAcc>,
+ datasets: &HashMap<&str, &PlanDataset>,
+) -> Result<String> {
+ anyhow::ensure!(
+ owner.family == family,
+ "detected family changed from {} to {family}",
+ owner.family
+ );
+ let matches = owner
+ .assignments
+ .iter()
+ .filter(|(assigned_group, _)| assigned_group == group)
+ .collect::<Vec<_>>();
+ anyhow::ensure!(
+ matches.len() == 1,
+ "frozen assignment has {} matches for group {:?}",
+ matches.len(),
+ group
+ );
+ let slug = &matches[0].1;
+ let dataset = datasets
+ .get(slug.as_str())
+ .with_context(|| format!("frozen assignment references absent dataset {slug}"))?;
+ ensure_compatible(fields, dataset)?;
+ Ok(slug.clone())
+}
+
+fn classify_new(
+ group: &Option<String>,
+ family: &str,
+ fields: &HashMap<String, FieldAcc>,
+ datasets: &[PlanDataset],
+) -> Result<String> {
+ let mut candidates = Vec::new();
+ for dataset in datasets
+ .iter()
+ .filter(|dataset| dataset.family == family && &dataset.group == group)
+ {
+ if ensure_compatible(fields, dataset).is_err() {
+ continue;
+ }
+ let (intersection, union) = field_overlap(fields, &dataset.specs);
+ let threshold_met = if group.is_some() {
+ intersection * 2 >= union // 0.5
+ } else {
+ intersection * 10 >= union * 7 // 0.7
+ };
+ if threshold_met {
+ candidates.push((dataset, intersection, union));
+ }
+ }
+ candidates.sort_by(|(left, li, lu), (right, ri, ru)| {
+ ratio_cmp(*ri, *ru, *li, *lu).then_with(|| left.slug.cmp(&right.slug))
+ });
+ candidates
+ .first()
+ .map(|(dataset, _, _)| dataset.slug.clone())
+ .with_context(|| {
+ format!(
+ "no frozen dataset accepts family {family}, group {:?}, and fields {:?}; \
+ this requires unsupported dataset/schema evolution (use a new prefix)",
+ group,
+ sorted_field_names(fields)
+ )
+ })
+}
+
+fn field_overlap(fields: &HashMap<String, FieldAcc>, specs: &[FieldSpec]) -> (usize, usize) {
+ // Keep null-only and object-shaped fields in the schema fingerprint too.
+ // FieldAcc deliberately does not count those as scalar observations, but
+ // allowing an unknown one through would let the server create a dynamic
+ // mapping outside the frozen plan.
+ let observed: HashSet<&str> = fields.keys().map(String::as_str).collect();
+ let frozen: HashSet<&str> = specs.iter().map(|spec| spec.name.as_str()).collect();
+ (
+ observed.intersection(&frozen).count(),
+ observed.union(&frozen).count().max(1),
+ )
+}
+
+fn ratio_cmp(left_n: usize, left_d: usize, right_n: usize, right_d: usize) -> Ordering {
+ (left_n * right_d).cmp(&(right_n * left_d))
+}
+
+fn ensure_compatible(fields: &HashMap<String, FieldAcc>, dataset: &PlanDataset) -> Result<()> {
+ let specs: HashMap<&str, &FieldSpec> = dataset
+ .specs
+ .iter()
+ .map(|spec| (spec.name.as_str(), spec))
+ .collect();
+ for (name, acc) in fields {
+ let spec = specs.get(name.as_str()).with_context(|| {
+ format!(
+ "field {name} is absent from frozen dataset {}",
+ dataset.slug
+ )
+ })?;
+ anyhow::ensure!(
+ compatible_with_spec(acc, spec),
+ "field {name} is incompatible with frozen {} mapping in dataset {}",
+ spec.es_type,
+ dataset.slug
+ );
+ }
+ Ok(())
+}
+
+fn compatible_with_spec(acc: &FieldAcc, spec: &FieldSpec) -> bool {
+ if acc.n == 0 {
+ return true;
+ }
+ let at_least_95 = |accepted: u64| accepted.saturating_mul(100) >= acc.n.saturating_mul(95);
+ match spec.es_type.as_str() {
+ "boolean" => at_least_95(acc.bool_ok),
+ "long" => at_least_95(acc.long_ok),
+ "double" => at_least_95(acc.double_ok),
+ "date" => {
+ let string_dates = acc.date_hits.values().copied().sum::<u64>();
+ let numeric_dates = match spec.date_enc.as_deref() {
+ Some("epoch_millis" | "epoch_seconds") => acc.long_ok,
+ _ => 0,
+ };
+ at_least_95(string_dates.saturating_add(numeric_dates).min(acc.n))
+ }
+ "keyword" | "text" | "semantic_text" => true,
+ _ => false,
+ }
+}
+
+fn sorted_field_names(fields: &HashMap<String, FieldAcc>) -> Vec<&str> {
+ let mut names = fields.keys().map(String::as_str).collect::<Vec<_>>();
+ names.sort_unstable();
+ names
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::sniff::{Family, Sniffed};
+ use crate::state::{DuplicateFile, PlanDataset};
+ use crate::walk::FileEntry;
+ use serde_json::json;
+ use std::path::PathBuf;
+
+ fn acc(values: &[serde_json::Value]) -> FieldAcc {
+ let mut acc = FieldAcc::default();
+ for value in values {
+ acc.add(value);
+ }
+ acc
+ }
+
+ fn spec(name: &str, es_type: &str) -> FieldSpec {
+ FieldSpec {
+ name: name.into(),
+ es_type: es_type.into(),
+ date_enc: None,
+ semantic: None,
+ cardinality_est: 0,
+ cardinality_overflow: false,
+ null_ratio: 0.0,
+ avg_len: 0.0,
+ coverage: 1.0,
+ examples: vec![],
+ notes: vec![],
+ date_min: None,
+ date_max: None,
+ date_evidence: vec![],
+ }
+ }
+
+ fn dataset(slug: &str, specs: Vec<FieldSpec>) -> PlanDataset {
+ PlanDataset {
+ slug: slug.into(),
+ index: format!("ax-{slug}"),
+ family: "jsonl".into(),
+ group: None,
+ specs,
+ time_field: None,
+ semantic_field: None,
+ sampled_records: 1,
+ file_count: 1,
+ }
+ }
+
+ fn file(rel: &str, path_id: &str) -> FileEntry {
+ FileEntry {
+ path: PathBuf::from(rel),
+ rel: rel.into(),
+ rel_id: path_id.into(),
+ is_symlink: false,
+ size: 10,
+ }
+ }
+
+ fn scan(fields: HashMap<String, FieldAcc>) -> FileScan {
+ FileScan {
+ sniffed: Some(Sniffed {
+ family: Family::Jsonl,
+ gzip: false,
+ binary_kind: None,
+ csv: None,
+ encoding: "utf-8",
+ }),
+ sketches: vec![(None, fields, 1)],
+ junk: None,
+ }
+ }
+
+ fn inventory(rel: &str, path_id: &str, key: &str) -> Inventory {
+ Inventory {
+ files: vec![file(rel, path_id)],
+ keys: vec![key.into()],
+ digests: vec![format!("digest-{key}")],
+ duplicates: vec![],
+ }
+ }
+
+ fn assignment(rel: &str, path_id: &str, slug: &str) -> FileAssignment {
+ FileAssignment {
+ rel: rel.into(),
+ path_id: path_id.into(),
+ is_symlink: Some(false),
+ family: "jsonl".into(),
+ gzip: false,
+ content_digest: Some("old-digest".into()),
+ assignments: vec![(None, slug.into())],
+ }
+ }
+
+ #[test]
+ fn same_path_replacement_retains_dataset_and_never_re_elects_slug() {
+ let mut previous = Plan {
+ datasets: vec![
+ dataset("z-owner", vec![spec("id", "long"), spec("body", "text")]),
+ dataset("a-other", vec![spec("id", "long"), spec("body", "text")]),
+ ],
+ ..Plan::default()
+ };
+ previous
+ .files
+ .insert("old".into(), assignment("a.jsonl", "unix:61", "z-owner"));
+ let fields = HashMap::from([
+ ("id".into(), acc(&[json!(7)])),
+ ("body".into(), acc(&[json!("hello world")])),
+ ]);
+ let plan = reconcile_plan(
+ &inventory("a.jsonl", "unix:61", "new"),
+ &previous,
+ vec![scan(fields)],
+ )
+ .unwrap();
+ assert_eq!(
+ plan.files["new"].assignments,
+ vec![(None, "z-owner".into())]
+ );
+ }
+
+ #[test]
+ fn new_file_uses_highest_jaccard_then_stable_slug_tie_break() {
+ let previous = Plan {
+ datasets: vec![
+ dataset("z-tie", vec![spec("id", "long"), spec("body", "text")]),
+ dataset("a-tie", vec![spec("id", "long"), spec("body", "text")]),
+ dataset(
+ "lower",
+ vec![
+ spec("id", "long"),
+ spec("body", "text"),
+ spec("extra", "keyword"),
+ ],
+ ),
+ ],
+ ..Plan::default()
+ };
+ let fields = HashMap::from([
+ ("id".into(), acc(&[json!(7)])),
+ ("body".into(), acc(&[json!("hello")])),
+ ]);
+ let plan = reconcile_plan(
+ &inventory("new.jsonl", "unix:6e", "new"),
+ &previous,
+ vec![scan(fields)],
+ )
+ .unwrap();
+ assert_eq!(plan.files["new"].assignments, vec![(None, "a-tie".into())]);
+ assert_eq!(plan.datasets[0].file_count, 0);
+ assert_eq!(plan.datasets[1].file_count, 1);
+ }
+
+ #[test]
+ fn new_field_and_incompatible_type_fail_before_plan_changes() {
+ let previous = Plan {
+ datasets: vec![dataset("rows", vec![spec("id", "long")])],
+ ..Plan::default()
+ };
+ let unknown = HashMap::from([
+ ("id".into(), acc(&[json!(7)])),
+ ("surprise".into(), acc(&[json!("new mapping")])),
+ ]);
+ let error = reconcile_plan(
+ &inventory("new.jsonl", "unix:6e", "new"),
+ &previous,
+ vec![scan(unknown)],
+ )
+ .unwrap_err();
+ assert!(format!("{error:#}").contains("unsupported dataset/schema evolution"));
+
+ // Object-shaped values are not counted as scalar FieldAcc evidence,
+ // but their field name must still be rejected rather than dynamically
+ // mapped by the server.
+ let object_field = HashMap::from([
+ ("id".into(), acc(&[json!(7)])),
+ ("object".into(), acc(&[json!({"nested": true})])),
+ ]);
+ let error = reconcile_plan(
+ &inventory("new.jsonl", "unix:6e", "new"),
+ &previous,
+ vec![scan(object_field)],
+ )
+ .unwrap_err();
+ assert!(format!("{error:#}").contains("unsupported dataset/schema evolution"));
+
+ let wrong_type = HashMap::from([("id".into(), acc(&[json!("not-a-number")]))]);
+ let error = reconcile_plan(
+ &inventory("new.jsonl", "unix:6e", "new"),
+ &previous,
+ vec![scan(wrong_type)],
+ )
+ .unwrap_err();
+ assert!(format!("{error:#}").contains("unsupported dataset/schema evolution"));
+ }
+
+ #[test]
+ fn rename_by_content_retains_assignment_and_rebuilds_aliases_and_counts() {
+ let mut previous = Plan {
+ datasets: vec![dataset("rows", vec![spec("id", "long")])],
+ ..Plan::default()
+ };
+ previous
+ .files
+ .insert("same".into(), assignment("old.jsonl", "unix:6f", "rows"));
+ let mut current = inventory("renamed.jsonl", "unix:72", "same");
+ current.duplicates.push(DuplicateFile {
+ file_key: "same".into(),
+ rel: "alias.jsonl".into(),
+ path_id: "unix:61".into(),
+ is_symlink: Some(false),
+ duplicate_of: "renamed.jsonl".into(),
+ bytes: 10,
+ });
+ let fields = HashMap::from([("id".into(), acc(&[json!(7)]))]);
+ let plan = reconcile_plan(&current, &previous, vec![scan(fields)]).unwrap();
+ assert_eq!(plan.files["same"].rel, "renamed.jsonl");
+ assert_eq!(plan.datasets[0].file_count, 1);
+ assert_eq!(plan.duplicate_files.len(), 1);
+ }
+
+ #[test]
+ fn duplicate_content_identity_and_conflicting_owners_fail_closed() {
+ let mut previous = Plan {
+ datasets: vec![dataset("rows", vec![spec("id", "long")])],
+ ..Plan::default()
+ };
+ previous
+ .files
+ .insert("content-a".into(), assignment("a.jsonl", "unix:61", "rows"));
+ previous
+ .files
+ .insert("content-b".into(), assignment("b.jsonl", "unix:62", "rows"));
+ let fields = || HashMap::from([("id".into(), acc(&[json!(7)]))]);
+
+ let conflict = inventory("a.jsonl", "unix:62", "content-a");
+ let error = reconcile_plan(&conflict, &previous, vec![scan(fields())]).unwrap_err();
+ assert!(format!("{error:#}").contains("conflicting committed content and path owners"));
+
+ let duplicate = Inventory {
+ files: vec![file("c.jsonl", "unix:63"), file("d.jsonl", "unix:64")],
+ keys: vec!["same-key".into(), "same-key".into()],
+ digests: vec!["digest".into(), "digest".into()],
+ duplicates: vec![],
+ };
+ let error = reconcile_plan(&duplicate, &previous, vec![scan(fields()), scan(fields())])
+ .unwrap_err();
+ assert!(format!("{error:#}").contains("duplicate content identity"));
+ }
+}
diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rs
index a0d8c5485..4802cd260 100644
--- a/engine/crates/xerj-autoindex/src/sniff.rs
+++ b/engine/crates/xerj-autoindex/src/sniff.rs
@@ -85,15 +85,30 @@ fn read_prefix(path: &Path, gzip: bool, n: usize) -> Result<Vec<u8>> {
}
pub fn sniff(path: &Path) -> Result<Sniffed> {
- let head = read_prefix(path, false, 8)?;
+ sniff_with_name(path, path)
+}
+
+/// Classify bytes from `content_path` while retaining the logical filename
+/// signals (currently source-code extensions) from `logical_path`.
+///
+/// Durable preparation uses this to classify an immutable snapshot blob
+/// without losing the original name merely because the blob itself is named
+/// by an ordinal.
+pub fn sniff_with_name(content_path: &Path, logical_path: &Path) -> Result<Sniffed> {
+ let head = read_prefix(content_path, false, 8)?;
let gzip = head.len() >= 2 && head[0] == 0x1f && head[1] == 0x8b;
- let prefix = read_prefix(path, gzip, 8192)?;
- let mut s = sniff_bytes(&prefix, path, gzip)?;
+ let prefix = read_prefix(content_path, gzip, 8192)?;
+ let mut s = sniff_bytes(&prefix, content_path, logical_path, gzip)?;
s.gzip = gzip;
Ok(s)
}
-fn sniff_bytes(prefix: &[u8], path: &Path, gzip: bool) -> Result<Sniffed> {
+fn sniff_bytes(
+ prefix: &[u8],
+ content_path: &Path,
+ logical_path: &Path,
+ gzip: bool,
+) -> Result<Sniffed> {
let mk = |family: Family| Sniffed {
family,
gzip: false,
@@ -117,7 +132,7 @@ fn sniff_bytes(prefix: &[u8], path: &Path, gzip: bool) -> Result<Sniffed> {
if prefix.starts_with(b"PK\x03\x04") {
// zip container: DOCX iff it holds word/document.xml
if !gzip {
- if let Ok(f) = std::fs::File::open(path) {
+ if let Ok(f) = std::fs::File::open(content_path) {
if let Ok(mut z) = zip::ZipArchive::new(f) {
let is_docx = (0..z.len()).any(|i| {
z.by_index_raw(i)
@@ -172,7 +187,7 @@ fn sniff_bytes(prefix: &[u8], path: &Path, gzip: bool) -> Result<Sniffed> {
// reach here after the binary guards above, so a text `.py`/`.rs`/`.go`/…
// routes to the tree-sitter AST extractor (crate::extract::code). Extension
// is the right signal — code vs prose is not reliably content-sniffable.
- if let Some(ext) = path.extension().and_then(|e| e.to_str()) {
+ if let Some(ext) = logical_path.extension().and_then(|e| e.to_str()) {
if crate::extract::code::is_code_ext(ext) {
let mut s = mk(Family::Code);
s.encoding = encoding;
diff --git a/engine/crates/xerj-autoindex/src/state.rs b/engine/crates/xerj-autoindex/src/state.rs
index ddb559fa2..16a13c522 100644
--- a/engine/crates/xerj-autoindex/src/state.rs
+++ b/engine/crates/xerj-autoindex/src/state.rs
@@ -12,9 +12,10 @@ use std::io::{BufRead, Write};
use std::path::{Path, PathBuf};
#[cfg(test)]
-static FILE_DONE_IO_FAILPOINT: std::sync::atomic::AtomicU8 = std::sync::atomic::AtomicU8::new(0);
+static FILE_DONE_IO_FAILPOINT: std::sync::Mutex<Option<(u8, PathBuf)>> =
+ std::sync::Mutex::new(None);
#[cfg(test)]
-static SYNC_IO_FAILPOINT: std::sync::atomic::AtomicU8 = std::sync::atomic::AtomicU8::new(0);
+static SYNC_IO_FAILPOINT: std::sync::Mutex<Option<(u8, PathBuf)>> = std::sync::Mutex::new(None);
#[cfg(test)]
pub(crate) static FILE_DONE_IO_FAILPOINT_TEST_LOCK: std::sync::Mutex<()> =
std::sync::Mutex::new(());
@@ -22,13 +23,13 @@ pub(crate) static FILE_DONE_IO_FAILPOINT_TEST_LOCK: std::sync::Mutex<()> =
pub(crate) static SYNC_IO_FAILPOINT_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
#[cfg(test)]
-pub(crate) fn fail_next_file_done_io(boundary: u8) {
- FILE_DONE_IO_FAILPOINT.store(boundary, std::sync::atomic::Ordering::SeqCst);
+pub(crate) fn fail_next_file_done_io(boundary: u8, journal_path: &Path) {
+ *FILE_DONE_IO_FAILPOINT.lock().unwrap() = Some((boundary, journal_path.to_path_buf()));
}
#[cfg(test)]
-fn fail_next_sync_io(boundary: u8) {
- SYNC_IO_FAILPOINT.store(boundary, std::sync::atomic::Ordering::SeqCst);
+fn fail_next_sync_io(boundary: u8, journal_path: &Path) {
+ *SYNC_IO_FAILPOINT.lock().unwrap() = Some((boundary, journal_path.to_path_buf()));
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -186,6 +187,8 @@ mod tests {
"{tail}"
);
}
+ }
+
#[test]
fn preflight_holds_the_same_exclusive_lock_through_authoritative_open() {
let dir = tempfile::tempdir().unwrap();
@@ -420,7 +423,7 @@ fn apply_legacy_plan(
}
#[cfg(test)]
-fn consume_io_failpoint(what: &str, boundary: u8) -> bool {
+fn consume_io_failpoint(what: &str, boundary: u8, journal_path: &Path) -> bool {
let failpoint = if what == "file_done" {
&FILE_DONE_IO_FAILPOINT
} else if what.starts_with("sync_") {
@@ -428,14 +431,18 @@ fn consume_io_failpoint(what: &str, boundary: u8) -> bool {
} else {
return false;
};
- failpoint
- .compare_exchange(
- boundary,
- 0,
- std::sync::atomic::Ordering::SeqCst,
- std::sync::atomic::Ordering::SeqCst,
- )
- .is_ok()
+ let mut armed = failpoint.lock().unwrap();
+ if armed
+ .as_ref()
+ .is_some_and(|(expected_boundary, expected_path)| {
+ *expected_boundary == boundary && expected_path == journal_path
+ })
+ {
+ *armed = None;
+ true
+ } else {
+ false
+ }
}
type PreflightReplay = (
@@ -531,11 +538,14 @@ fn read_plan_for_preflight(
}
Some(kind) if crate::sync::is_sync_record_kind(kind) => {
crate::sync::replay_record(&value, &mut committed_manifest, &mut pending_sync)?;
- if kind == "sync_commit" {
+ if matches!(kind, "sync_bootstrap" | "sync_commit") {
plan = committed_manifest
.as_ref()
.map(|manifest| manifest.plan.clone());
}
+ if kind == "sync_bootstrap" {
+ legacy_migration_reasons.clear();
+ }
}
Some(kind) if kind.starts_with("sync_") => {
anyhow::bail!("unknown durable sync record kind {kind}");
@@ -776,11 +786,14 @@ impl Journal {
&mut committed_manifest,
&mut pending_sync,
)?;
- if kind == "sync_commit" {
+ if matches!(kind, "sync_bootstrap" | "sync_commit") {
plan = committed_manifest
.as_ref()
.map(|manifest| manifest.plan.clone());
}
+ if kind == "sync_bootstrap" {
+ legacy_migration_reasons.clear();
+ }
}
Some(kind) if kind.starts_with("sync_") => {
anyhow::bail!(
@@ -954,11 +967,11 @@ impl Journal {
line.push('\n');
let start = self.file.metadata()?.len();
#[cfg(test)]
- if consume_io_failpoint(what, 1) {
+ if consume_io_failpoint(what, 1, &self.path) {
anyhow::bail!("injected {what} append failure before any bytes");
}
#[cfg(test)]
- let partial_write = consume_io_failpoint(what, 3);
+ let partial_write = consume_io_failpoint(what, 3, &self.path);
#[cfg(not(test))]
let partial_write = false;
let write_result = if partial_write {
@@ -977,7 +990,7 @@ impl Journal {
return self.rollback_transaction(start, what, "append", error);
}
#[cfg(test)]
- let injected_sync_failure = consume_io_failpoint(what, 2);
+ let injected_sync_failure = consume_io_failpoint(what, 2, &self.path);
#[cfg(not(test))]
let injected_sync_failure = false;
let sync_result = if injected_sync_failure {
@@ -1057,6 +1070,32 @@ impl Journal {
Ok(())
}
+ pub fn sync_bootstrap_genesis(&mut self) -> Result<()> {
+ anyhow::ensure!(
+ self.pending_sync.is_none()
+ && self.committed_manifest.is_none()
+ && self.plan.is_none()
+ && self.done.is_empty()
+ && self.pending_replacements.is_empty(),
+ "generation-zero bootstrap is only valid for an empty new journal"
+ );
+ let manifest = crate::sync::CommittedManifest::genesis()?;
+ let mut committed = None;
+ crate::sync::replay_record(
+ &serde_json::json!({"kind": "sync_bootstrap", "manifest": manifest}),
+ &mut committed,
+ &mut None,
+ )?;
+ self.append_transaction(
+ &serde_json::json!({"kind": "sync_bootstrap", "manifest": manifest}),
+ "sync_bootstrap",
+ )?;
+ self.plan = Some(Plan::default());
+ self.committed_manifest = committed;
+ self.legacy_migration_reasons.clear();
+ Ok(())
+ }
+
pub fn sync_operation_state(
&mut self,
operation_id: &str,
@@ -1221,14 +1260,28 @@ mod sync_journal_tests {
sampled_records: 1,
file_count: 0,
});
- journal.write_plan(&plan).unwrap();
+ journal.sync_bootstrap_genesis().unwrap();
journal
}
fn pending(journal: &Journal) -> PendingSync {
let base = journal.committed_manifest.as_ref().unwrap();
let mut plan = base.plan.clone();
- plan.datasets[0].file_count = 1;
+ if plan.datasets.is_empty() {
+ plan.datasets.push(PlanDataset {
+ slug: "reports".into(),
+ index: "prefix-reports".into(),
+ family: "pdf".into(),
+ group: None,
+ specs: Vec::new(),
+ time_field: None,
+ semantic_field: None,
+ sampled_records: 1,
+ file_count: 1,
+ });
+ } else {
+ plan.datasets[0].file_count = 1;
+ }
plan.files.insert(
"content-a".into(),
FileAssignment {
@@ -1253,13 +1306,14 @@ mod sync_journal_tests {
prefix: "prefix".into(),
follow_symlinks: false,
chunker_identity: "chunker-v1".into(),
+ embedding_identity_sha256: "a".repeat(64),
embedding_backend: "lexical".into(),
- embedding_model: "feature-hash".into(),
- embedding_tokenizer: "builtin".into(),
embedding_dimension: 384,
+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),
+ embedding_resumable: true,
graph_enabled: false,
- brain: "none".into(),
- detector_identity: "detectors-v1".into(),
+ brain: "disabled".into(),
+ detector_identity: "disabled".into(),
schema_identity: "schema-v1".into(),
index_identity: "index-v1".into(),
source_policy: SourceExecutionPolicy::AbortOnSourceChange {
@@ -1311,6 +1365,29 @@ mod sync_journal_tests {
assert_eq!(reopened.pending_sync.as_ref().unwrap().tx_id, "sync-1");
}
+ #[test]
+ fn new_journal_can_bootstrap_empty_generation_authority_once() {
+ let dir = tempfile::tempdir().unwrap();
+ let mut journal = Journal::open(dir.path(), "root", "url", "prefix", 300, false).unwrap();
+ journal.sync_bootstrap_genesis().unwrap();
+ let committed = journal.committed_manifest.as_ref().unwrap();
+ assert_eq!(committed.generation, 0);
+ assert!(committed.groups.is_empty());
+ assert_eq!(
+ serde_json::to_value(journal.plan.as_ref().unwrap()).unwrap(),
+ serde_json::to_value(Plan::default()).unwrap()
+ );
+ assert!(journal.sync_bootstrap_genesis().is_err());
+ drop(journal);
+
+ let reopened = Journal::open(dir.path(), "root", "url", "prefix", 300, false).unwrap();
+ assert_eq!(reopened.committed_manifest.unwrap().generation, 0);
+ assert_eq!(
+ serde_json::to_value(reopened.plan.unwrap()).unwrap(),
+ serde_json::to_value(Plan::default()).unwrap()
+ );
+ }
+
#[test]
fn replay_rejects_sync_begin_followed_by_legacy_plan_without_changing_authority() {
let _guard = SYNC_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
@@ -1535,7 +1612,7 @@ mod sync_journal_tests {
let mut journal = journal_with_plan(dir.path());
let desired = pending(&journal);
let before = std::fs::read(dir.path().join("journal.ndjson")).unwrap();
- fail_next_sync_io(boundary);
+ fail_next_sync_io(boundary, journal.path());
assert!(journal.sync_begin(&desired).is_err());
assert!(journal.pending_sync.is_none());
drop(journal);
@@ -1569,7 +1646,7 @@ mod sync_journal_tests {
journal.sync_validated().unwrap();
}
let before = std::fs::read(dir.path().join("journal.ndjson")).unwrap();
- fail_next_sync_io(boundary);
+ fail_next_sync_io(boundary, journal.path());
let result = match stage {
"operation" => {
journal.sync_operation_state("upsert:group-a", SyncOperationState::Started)
@@ -1809,7 +1886,7 @@ mod compatibility_tests {
let mut journal =
Journal::open(dir.path(), "root", "http://engine", "ax", 300, false).unwrap();
journal.file_replace_start("file-key", "new").unwrap();
- fail_next_file_done_io(boundary);
+ fail_next_file_done_io(boundary, journal.path());
let error = journal
.file_done(&FileDone {
file_key: "file-key".into(),
@@ -1847,7 +1924,7 @@ mod compatibility_tests {
journal
.file_replace_start("file-c", "generation-c")
.unwrap();
- fail_next_file_done_io(3);
+ fail_next_file_done_io(3, journal.path());
let error = journal
.file_done(&FileDone {
file_key: "file-c".into(),
diff --git a/engine/crates/xerj-autoindex/src/sync.rs b/engine/crates/xerj-autoindex/src/sync.rs
index c60f4c151..9a20f4352 100644
--- a/engine/crates/xerj-autoindex/src/sync.rs
+++ b/engine/crates/xerj-autoindex/src/sync.rs
@@ -10,7 +10,7 @@ use serde::{Deserialize, Serialize};
use serde_json::Value;
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
-pub const EXECUTION_IDENTITY_VERSION: u32 = 1;
+pub const EXECUTION_IDENTITY_VERSION: u32 = 2;
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
pub struct ManifestPath {
@@ -66,10 +66,13 @@ pub struct ExecutionIdentity {
pub prefix: String,
pub follow_symlinks: bool,
pub chunker_identity: String,
+ /// Opaque, server-authenticated identity of the exact embedding execution
+ /// space. Autoindex must never synthesize model/tokenizer labels.
+ pub embedding_identity_sha256: String,
pub embedding_backend: String,
- pub embedding_model: String,
- pub embedding_tokenizer: String,
pub embedding_dimension: usize,
+ pub embedding_semantic_contract: String,
+ pub embedding_resumable: bool,
pub graph_enabled: bool,
pub brain: String,
pub detector_identity: String,
@@ -105,6 +108,27 @@ pub enum LegacyBootstrap {
}
impl CommittedManifest {
+ pub fn genesis() -> Result<Self> {
+ match Self::bootstrap_legacy(Plan::default())? {
+ LegacyBootstrap::Ready(manifest) => Ok(*manifest),
+ LegacyBootstrap::MigrationRequired { .. } => {
+ anyhow::bail!("empty generation-zero manifest unexpectedly requires migration")
+ }
+ }
+ }
+
+ fn validate_genesis(&self) -> Result<()> {
+ anyhow::ensure!(
+ self.generation == 0
+ && self.manifest_digest == Self::genesis()?.manifest_digest
+ && serde_json::to_value(&self.plan)? == serde_json::to_value(Plan::default())?
+ && self.groups.is_empty()
+ && self.execution.is_none(),
+ "generation-zero bootstrap manifest is not exact genesis"
+ );
+ Ok(())
+ }
+
pub fn bootstrap_legacy(plan: Plan) -> Result<LegacyBootstrap> {
let mut reasons = Vec::new();
for (file_key, file) in &plan.files {
@@ -355,6 +379,13 @@ fn validate_supported_manifest_delta(
// the only configuration transition this vocabulary can represent. A
// durable snapshot reference and digest are generation inputs, not engine
// configuration, so they must advance with each desired generation.
+ if base.generation == 0 && base.execution.is_none() && base.groups.is_empty() {
+ anyhow::ensure!(
+ base.manifest_digest == CommittedManifest::genesis()?.manifest_digest,
+ "generation-one migration base is not exact genesis"
+ );
+ return Ok(());
+ }
if base.generation > 0 || base.execution.is_some() {
let same_execution_configuration = |left: &ExecutionIdentity, right: &ExecutionIdentity| {
let mut left = left.clone();
@@ -385,11 +416,10 @@ fn validate_supported_manifest_delta(
base.plan.alias_paths_indexed == desired.plan.alias_paths_indexed,
"plan flags changed without a supported operation"
);
- anyhow::ensure!(
- canonical_json_bytes(&base.plan.junk_files)?
- == canonical_json_bytes(&desired.plan.junk_files)?,
- "junk plan changed without a supported operation"
- );
+ // Junk is a local inference result, not remotely published generation
+ // membership. Adding/removing an unsupported file may advance the
+ // manifest for truthful map/status output, but it intentionally derives
+ // no remote data or catalog operation.
let normalize_datasets = |datasets: &[crate::state::PlanDataset]| {
let mut datasets = datasets.to_vec();
for dataset in &mut datasets {
@@ -632,16 +662,25 @@ fn validate_manifest(manifest: &GenerationManifest, legacy: bool) -> Result<()>
execution.version == EXECUTION_IDENTITY_VERSION,
"unsupported execution identity version"
);
+ anyhow::ensure!(
+ !execution.graph_enabled
+ && execution.brain == "disabled"
+ && execution.detector_identity == "disabled",
+ "incremental generation manifests require graph-disabled execution identity"
+ );
for (name, value) in [
("root_identity", execution.root_identity.as_str()),
("url", execution.url.as_str()),
("prefix", execution.prefix.as_str()),
("chunker_identity", execution.chunker_identity.as_str()),
+ (
+ "embedding_identity_sha256",
+ execution.embedding_identity_sha256.as_str(),
+ ),
("embedding_backend", execution.embedding_backend.as_str()),
- ("embedding_model", execution.embedding_model.as_str()),
(
- "embedding_tokenizer",
- execution.embedding_tokenizer.as_str(),
+ "embedding_semantic_contract",
+ execution.embedding_semantic_contract.as_str(),
),
("detector_identity", execution.detector_identity.as_str()),
("schema_identity", execution.schema_identity.as_str()),
@@ -649,6 +688,22 @@ fn validate_manifest(manifest: &GenerationManifest, legacy: bool) -> Result<()>
] {
anyhow::ensure!(!value.is_empty(), "execution identity {name} is empty");
}
+ anyhow::ensure!(
+ execution.embedding_identity_sha256.len() == 64
+ && execution
+ .embedding_identity_sha256
+ .bytes()
+ .all(|byte| byte.is_ascii_hexdigit() && !byte.is_ascii_uppercase()),
+ "embedding execution identity must be a lowercase SHA-256 digest"
+ );
+ anyhow::ensure!(
+ execution.embedding_dimension > 0,
+ "embedding execution dimension must be positive"
+ );
+ anyhow::ensure!(
+ execution.embedding_resumable,
+ "a durable generation requires a resumable embedding execution identity"
+ );
match &execution.source_policy {
SourceExecutionPolicy::DurableSnapshot {
reference,
@@ -901,7 +956,12 @@ pub fn apply_begin(
pub fn is_sync_record_kind(kind: &str) -> bool {
matches!(
kind,
- "sync_begin" | "sync_operation_state" | "sync_validated" | "sync_commit" | "sync_abort"
+ "sync_bootstrap"
+ | "sync_begin"
+ | "sync_operation_state"
+ | "sync_validated"
+ | "sync_commit"
+ | "sync_abort"
)
}
@@ -912,6 +972,26 @@ pub fn replay_record(
) -> Result<()> {
let kind = record_str(record, "kind")?;
match kind {
+ "sync_bootstrap" => {
+ anyhow::ensure!(
+ pending.is_none(),
+ "sync_bootstrap cannot replace pending sync"
+ );
+ anyhow::ensure!(
+ committed.is_none(),
+ "sync_bootstrap cannot replace committed authority"
+ );
+ let manifest: CommittedManifest = serde_json::from_value(
+ record
+ .get("manifest")
+ .cloned()
+ .context("sync_bootstrap has no manifest")?,
+ )
+ .context("decode durable sync_bootstrap")?;
+ manifest.validate_genesis()?;
+ *committed = Some(manifest);
+ Ok(())
+ }
"sync_begin" => {
let begin: PendingSync =
serde_json::from_value(record.clone()).context("decode durable sync_begin")?;
@@ -1015,6 +1095,34 @@ mod tests {
}
}
+ #[test]
+ fn bootstrap_replay_accepts_only_exact_empty_genesis() {
+ let genesis = CommittedManifest::genesis().unwrap();
+ let record = serde_json::json!({
+ "kind": "sync_bootstrap",
+ "manifest": genesis,
+ });
+ let mut committed = None;
+ let mut pending = None;
+ replay_record(&record, &mut committed, &mut pending).unwrap();
+ assert_eq!(committed.unwrap().generation, 0);
+
+ let mut non_genesis = CommittedManifest::genesis().unwrap();
+ non_genesis.generation = 1;
+ let record = serde_json::json!({
+ "kind": "sync_bootstrap",
+ "manifest": non_genesis,
+ });
+ assert!(replay_record(&record, &mut None, &mut None).is_err());
+
+ let mut committed = Some(CommittedManifest::genesis().unwrap());
+ let record = serde_json::json!({
+ "kind": "sync_bootstrap",
+ "manifest": CommittedManifest::genesis().unwrap(),
+ });
+ assert!(replay_record(&record, &mut committed, &mut None).is_err());
+ }
+
#[test]
fn canonical_prefers_real_then_native_bytes_including_non_utf8_identity() {
let groups = reconcile_groups(
@@ -1154,13 +1262,14 @@ mod tests {
prefix: "prefix".into(),
follow_symlinks: false,
chunker_identity: "chunker-v1".into(),
+ embedding_identity_sha256: "a".repeat(64),
embedding_backend: "lexical".into(),
- embedding_model: "feature-hash".into(),
- embedding_tokenizer: "builtin".into(),
embedding_dimension: 384,
+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),
+ embedding_resumable: true,
graph_enabled: false,
- brain: "none".into(),
- detector_identity: "detectors-v1".into(),
+ brain: "disabled".into(),
+ detector_identity: "disabled".into(),
schema_identity: "schema-v1".into(),
index_identity: "index-v1".into(),
source_policy: SourceExecutionPolicy::AbortOnSourceChange {
@@ -1183,11 +1292,7 @@ mod tests {
sampled_records: 0,
file_count: 0,
});
- let LegacyBootstrap::Ready(legacy) =
- CommittedManifest::bootstrap_legacy(legacy_plan.clone()).unwrap()
- else {
- panic!("empty legacy membership is migratable");
- };
+ let legacy = CommittedManifest::genesis().unwrap();
let first = GenerationManifest {
generation: 1,
execution: Some(identity("inventory-a")),
@@ -1204,10 +1309,21 @@ mod tests {
assert!(
PendingSync::new("changed-execution".into(), &committed, changed_execution).is_err()
);
+ let mut changed_generated_flag = first.clone();
+ changed_generated_flag.generation = 2;
+ changed_generated_flag.plan.alias_paths_indexed =
+ !changed_generated_flag.plan.alias_paths_indexed;
+ assert!(PendingSync::new(
+ "changed-generated-flag".into(),
+ &committed,
+ changed_generated_flag
+ )
+ .is_err());
let mut changed_plan = first;
+ changed_plan.generation = 2;
changed_plan.plan.alias_paths_indexed = true;
- assert!(PendingSync::new("changed-plan".into(), &legacy, changed_plan).is_err());
+ assert!(PendingSync::new("changed-plan".into(), &committed, changed_plan).is_err());
}
#[test]
diff --git a/engine/crates/xerj-autoindex/src/sync_executor.rs b/engine/crates/xerj-autoindex/src/sync_executor.rs
index 06423deb4..234d00de3 100644
--- a/engine/crates/xerj-autoindex/src/sync_executor.rs
+++ b/engine/crates/xerj-autoindex/src/sync_executor.rs
@@ -13,20 +13,28 @@ use crate::sync::{
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
-use std::collections::HashMap;
+use std::collections::{BTreeMap, HashMap};
use std::fs::{File, OpenOptions};
use std::io::{BufReader, BufWriter, Read, Write};
use std::path::Path;
-const SNAPSHOT_VERSION: u32 = 1;
+const SNAPSHOT_VERSION: u32 = 2;
#[cfg(test)]
static SNAPSHOT_FAILPOINT: std::sync::atomic::AtomicU8 = std::sync::atomic::AtomicU8::new(0);
#[cfg(test)]
static SNAPSHOT_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
#[cfg(test)]
+static POST_SEAL_SOURCE_REPLACEMENT: std::sync::Mutex<Option<(std::path::PathBuf, Vec<u8>)>> =
+ std::sync::Mutex::new(None);
+#[cfg(test)]
static REPLAY_FAIL_AFTER_APPLY: std::sync::atomic::AtomicBool =
std::sync::atomic::AtomicBool::new(false);
+#[cfg(test)]
+pub(crate) static REPLAY_FAILPOINT_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+#[cfg(test)]
+static GC_FAIL_AFTER_RENAME: std::sync::atomic::AtomicBool =
+ std::sync::atomic::AtomicBool::new(false);
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct SnapshotFile {
@@ -52,15 +60,99 @@ pub struct PreparedArtifact {
pub struct SourceSnapshot {
pub version: u32,
pub tx_id: String,
+ /// Publication timestamp sealed once and reused by every replay.
+ pub started: String,
+ pub preparation_contract_digest: String,
+ pub footprint: SnapshotFootprint,
pub files: Vec<SnapshotFile>,
pub snapshot_digest: String,
}
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
+pub struct SnapshotFootprint {
+ pub source_bytes: u64,
+ pub prepared_bytes: u64,
+ pub total_bytes: u64,
+ pub hard_budget_bytes: u64,
+}
+
+/// Logical payload accounting for source and prepared-record bytes.
+/// This is deliberately not a filesystem-allocation limit: directory,
+/// manifest, and block-allocation overhead are excluded.
+struct PayloadBudget {
+ used: u64,
+ limit: u64,
+}
+
+impl PayloadBudget {
+ fn charge(&mut self, bytes: usize) -> std::io::Result<()> {
+ let bytes = u64::try_from(bytes)
+ .map_err(|_| std::io::Error::other("snapshot payload write length overflow"))?;
+ let next = self
+ .used
+ .checked_add(bytes)
+ .ok_or_else(|| std::io::Error::other("snapshot payload byte count overflow"))?;
+ if next > self.limit {
+ return Err(std::io::Error::other(format!(
+ "snapshot logical payload write of {bytes} bytes would raise staged payload from \
+ {} to {next} bytes, exceeding configured limit of {} bytes; no generation was \
+ committed and the partial snapshot is removed automatically",
+ self.used, self.limit
+ )));
+ }
+ self.used = next;
+ Ok(())
+ }
+}
+
+struct BudgetWriter<'a, W> {
+ inner: W,
+ budget: &'a mut PayloadBudget,
+}
+
+impl<W: Write> Write for BudgetWriter<'_, W> {
+ fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
+ self.budget.charge(buf.len())?;
+ match self.inner.write(buf) {
+ Ok(written) if written == buf.len() => Ok(written),
+ Ok(written) => {
+ self.budget.used -= (buf.len() - written) as u64;
+ Ok(written)
+ }
+ Err(error) => {
+ self.budget.used -= buf.len() as u64;
+ Err(error)
+ }
+ }
+ }
+
+ fn flush(&mut self) -> std::io::Result<()> {
+ self.inner.flush()
+ }
+}
+
+struct StagingCleanup {
+ path: std::path::PathBuf,
+ armed: bool,
+}
+
+impl Drop for StagingCleanup {
+ fn drop(&mut self) {
+ if self.armed {
+ let _ = std::fs::remove_dir_all(&self.path);
+ }
+ }
+}
+
/// Remote mutations must be convergent: calling `apply` twice for one
/// operation after an accepted-but-unrecorded response must produce exactly
/// the same live state. `validate` is the final generation-wide barrier.
#[cfg_attr(not(test), allow(dead_code))]
pub trait SyncOperationBackend {
+ fn provision_generation(&mut self, _desired: &GenerationManifest) -> Result<()> {
+ Ok(())
+ }
+
fn apply(
&mut self,
operation: &SyncOperation,
@@ -75,6 +167,17 @@ pub trait SyncOperationBackend {
desired: &GenerationManifest,
snapshot: &SourceSnapshot,
) -> Result<()>;
+
+ /// Publish and exactly read back the complete agent-facing catalog for
+ /// this generation. This runs before the validation/authority barrier.
+ fn publish_generation_catalog(
+ &mut self,
+ _base: &CommittedManifest,
+ _desired: &GenerationManifest,
+ _snapshot: &SourceSnapshot,
+ ) -> Result<()> {
+ Ok(())
+ }
}
/// Production ES-compatible operation backend for graph-disabled generations.
@@ -212,64 +315,162 @@ impl<'a> EsSyncBackend<'a> {
Ok(total)
}
- fn reconcile_file_catalog(
+ fn catalog_generation(&self, run_id: &str) -> Result<BTreeMap<String, Value>> {
+ let mut documents = BTreeMap::new();
+ let mut search_after: Option<Value> = None;
+ loop {
+ let mut body = serde_json::json!({
+ "size": 1000,
+ "sort": [{"_id": "asc"}],
+ "query": {"term": {"run_id": run_id}}
+ });
+ if let Some(after) = &search_after {
+ body["search_after"] = after.clone();
+ }
+ let response = self.es.search(crate::catalog::CATALOG_INDEX, &body)?;
+ let hits = response
+ .pointer("/hits/hits")
+ .and_then(Value::as_array)
+ .context("catalog generation query has no hits")?;
+ for hit in hits {
+ let id = hit
+ .get("_id")
+ .and_then(Value::as_str)
+ .context("catalog hit has no _id")?
+ .to_owned();
+ let source = hit
+ .get("_source")
+ .cloned()
+ .context("catalog hit has no _source")?;
+ anyhow::ensure!(
+ documents.insert(id.clone(), source).is_none(),
+ "catalog generation query returned duplicate ID {id}"
+ );
+ }
+ if hits.len() < 1000 {
+ break;
+ }
+ search_after = hits.last().and_then(|hit| hit.get("sort")).cloned();
+ anyhow::ensure!(
+ search_after.is_some(),
+ "full catalog generation page has no continuation sort key"
+ );
+ }
+ Ok(documents)
+ }
+
+ fn exact_dataset_catalog_stats(
&self,
- old: Option<&ManifestGroup>,
- new: Option<&ManifestGroup>,
desired: &GenerationManifest,
- ) -> Result<()> {
- for content_id in old
- .into_iter()
- .map(|group| group.content_id.as_str())
- .chain(new.into_iter().map(|group| group.content_id.as_str()))
- {
- self.es.delete_by_query(
- crate::catalog::CATALOG_INDEX,
- &serde_json::json!({"term": {"file_key": content_id}}),
- )?;
- }
- let Some(group) = new else {
- return Ok(());
- };
- let assignment = desired
- .plan
- .files
- .get(&group.content_id)
- .context("catalog group has no desired file assignment")?;
- let format = if assignment.gzip {
- format!("{}(gzip)", assignment.family)
- } else {
- assignment.family.clone()
- };
- let (id, doc) = crate::catalog::file_doc(
- &group.content_id,
- &group.canonical.rel,
- &format,
- "indexed",
- None,
- group.expected_records,
- 0,
- group.content_size,
- &format!("generation-{}", desired.generation),
- );
- let mut body = Vec::new();
- append_index_action(crate::catalog::CATALOG_INDEX, &id, &doc, &mut body)?;
- for alias in &group.aliases {
- let (id, doc) = crate::catalog::duplicate_file_doc(
- &group.content_id,
- &alias.rel,
- &alias.path_id,
- &group.canonical.rel,
- group.content_size,
- &format!("generation-{}", desired.generation),
+ ) -> Result<BTreeMap<String, crate::generation_catalog::DatasetCatalogStats>> {
+ let mut out = BTreeMap::new();
+ for dataset in &desired.plan.datasets {
+ let groups: Vec<&ManifestGroup> = desired
+ .groups
+ .iter()
+ .filter(|group| group.dataset_slugs.contains(&dataset.slug))
+ .collect();
+ let content_ids: Vec<&str> = groups
+ .iter()
+ .map(|group| group.content_id.as_str())
+ .collect();
+ let mut body = serde_json::json!({
+ "size": 0,
+ "track_total_hits": true,
+ "query": {"terms": {"ax_file": content_ids}}
+ });
+ if let Some(field) = &dataset.time_field {
+ body["aggs"] = serde_json::json!({
+ "time_min": {"min": {"field": field}},
+ "time_max": {"max": {"field": field}}
+ });
+ }
+ let response = self.es.search(&dataset.index, &body)?;
+ let record_count = response
+ .pointer("/hits/total/value")
+ .and_then(Value::as_u64)
+ .context("dataset exact read-back has no total")?;
+ let expected = groups.iter().try_fold(0u64, |sum, group| {
+ sum.checked_add(group.expected_records)
+ .context("dataset expected record count overflow")
+ })?;
+ anyhow::ensure!(
+ record_count == expected,
+ "dataset {} exact read-back count {record_count} disagrees with sealed count \
+ {expected}",
+ dataset.slug
+ );
+ let mut formats: Vec<String> = desired
+ .plan
+ .files
+ .values()
+ .filter(|assignment| {
+ assignment
+ .assignments
+ .iter()
+ .any(|(_, slug)| slug == &dataset.slug)
+ })
+ .map(|assignment| {
+ if assignment.gzip {
+ format!("{}(gzip)", assignment.family)
+ } else {
+ assignment.family.clone()
+ }
+ })
+ .collect();
+ formats.sort();
+ formats.dedup();
+ let bytes = groups.iter().try_fold(0u64, |sum, group| {
+ sum.checked_add(group.content_size)
+ .context("dataset source byte count overflow")
+ })?;
+ let time = |name: &str| {
+ response
+ .pointer(&format!("/aggregations/{name}/value_as_string"))
+ .and_then(Value::as_str)
+ .map(ToOwned::to_owned)
+ };
+ out.insert(
+ dataset.slug.clone(),
+ crate::generation_catalog::DatasetCatalogStats {
+ record_count,
+ junk_records: 0,
+ bytes,
+ formats,
+ time_min: time("time_min"),
+ time_max: time("time_max"),
+ sample_queries: crate::catalog::build_sample_queries(dataset, &[]),
+ notes: dataset
+ .group
+ .iter()
+ .map(|group| format!("source table: {group}"))
+ .chain(dataset.specs.iter().flat_map(|spec| {
+ spec.notes
+ .iter()
+ .map(move |note| format!("{}: {note}", spec.name))
+ }))
+ .collect(),
+ },
);
- append_index_action(crate::catalog::CATALOG_INDEX, &id, &doc, &mut body)?;
}
- checked_bulk(self.es, body)
+ Ok(out)
}
}
impl SyncOperationBackend for EsSyncBackend<'_> {
+ fn provision_generation(&mut self, desired: &GenerationManifest) -> Result<()> {
+ let execution = desired
+ .execution
+ .as_ref()
+ .context("desired generation has no execution identity")?;
+ let (_, index_identity) = crate::generation_contract_identities(&desired.plan)?;
+ anyhow::ensure!(
+ execution.index_identity == index_identity,
+ "desired generation index identity disagrees with its frozen mappings"
+ );
+ crate::ensure_generation_mappings(self.es, &desired.plan)
+ }
+
fn apply(
&mut self,
operation: &SyncOperation,
@@ -313,7 +514,73 @@ impl SyncOperationBackend for EsSyncBackend<'_> {
)?;
}
}
- self.reconcile_file_catalog(old, new, desired)
+ Ok(())
+ }
+
+ fn publish_generation_catalog(
+ &mut self,
+ base: &CommittedManifest,
+ desired: &GenerationManifest,
+ snapshot: &SourceSnapshot,
+ ) -> Result<()> {
+ for index in desired
+ .plan
+ .datasets
+ .iter()
+ .map(|dataset| dataset.index.as_str())
+ .collect::<std::collections::BTreeSet<_>>()
+ {
+ self.es.refresh(index)?;
+ }
+ self.es.refresh(crate::catalog::CATALOG_INDEX)?;
+ let prior_run_id =
+ base.execution
+ .as_ref()
+ .and_then(|execution| match &execution.source_policy {
+ SourceExecutionPolicy::DurableSnapshot { reference, .. } => {
+ reference.strip_prefix("sync-snapshots/")
+ }
+ SourceExecutionPolicy::AbortOnSourceChange { .. } => None,
+ });
+ let prior_documents = prior_run_id
+ .map(|run_id| self.catalog_generation(run_id))
+ .transpose()?
+ .unwrap_or_default();
+ let prior_ids = prior_documents.keys().cloned().collect();
+ let stats = self.exact_dataset_catalog_stats(desired)?;
+ let projection = crate::generation_catalog::project_generation(
+ base,
+ desired,
+ &crate::generation_catalog::GenerationCatalogMetadata {
+ generation_id: snapshot.tx_id.clone(),
+ started: snapshot.started.clone(),
+ },
+ &stats,
+ &BTreeMap::new(),
+ &prior_ids,
+ )?;
+ let mut body = Vec::new();
+ for id in &projection.stale_ids {
+ let action =
+ serde_json::json!({"delete": {"_index": crate::catalog::CATALOG_INDEX, "_id": id}});
+ body.extend_from_slice(serde_json::to_string(&action)?.as_bytes());
+ body.push(b'\n');
+ }
+ for (id, document) in &projection.documents {
+ append_index_action(crate::catalog::CATALOG_INDEX, id, document, &mut body)?;
+ }
+ checked_bulk(self.es, body)?;
+ self.es.refresh(crate::catalog::CATALOG_INDEX)?;
+ projection.validate_observed(&self.catalog_generation(&snapshot.tx_id)?)?;
+ if let Some(prior_run_id) = prior_run_id {
+ let remaining = self.catalog_generation(prior_run_id)?;
+ anyhow::ensure!(
+ remaining.is_empty(),
+ "prior catalog generation {prior_run_id} still has {} managed documents",
+ remaining.len()
+ );
+ }
+ Ok(())
}
fn validate(
@@ -394,6 +661,7 @@ pub fn replay_pending_operations(
pending.validate_against(&base)?;
let snapshot = open_snapshot(state_dir, &pending.tx_id)?;
verify_snapshot_binding(&pending, &snapshot)?;
+ backend.provision_generation(&pending.desired)?;
for operation in &pending.operations {
let state = journal
@@ -411,9 +679,110 @@ pub fn replay_pending_operations(
replay_fail_after_apply()?;
journal.sync_operation_state(&operation.operation_id, SyncOperationState::Committed)?;
}
+ backend.publish_generation_catalog(&base, &pending.desired, &snapshot)?;
backend.validate(&base, &pending.desired, &snapshot)?;
journal.sync_validated()?;
- journal.sync_commit()
+ journal.sync_commit()?;
+ gc_snapshots(state_dir, journal).context(
+ "generation committed durably, but snapshot cleanup failed; inspect the attached cause \
+ (remove a refused symlink or repair the reported filesystem permission/I/O problem), \
+ then retry the same command to continue bounded cleanup without republishing data",
+ )
+}
+
+const SNAPSHOT_GC_BATCH_SIZE: usize = 4096;
+
+fn protected_snapshot(
+ state_dir: &Path,
+ execution: &crate::sync::ExecutionIdentity,
+) -> Result<String> {
+ let SourceExecutionPolicy::DurableSnapshot {
+ reference,
+ snapshot_digest,
+ } = &execution.source_policy
+ else {
+ anyhow::bail!("generated authority does not reference a durable snapshot");
+ };
+ let tx_id = reference
+ .strip_prefix("sync-snapshots/")
+ .context("protected snapshot reference is not state-relative")?;
+ validate_tx_id(tx_id)?;
+ let snapshot = open_snapshot(state_dir, tx_id)?;
+ anyhow::ensure!(
+ snapshot.snapshot_digest == *snapshot_digest,
+ "protected snapshot digest disagrees with journal authority"
+ );
+ Ok(tx_id.to_owned())
+}
+
+/// Reclaim only snapshots proven unreferenced by replayed journal authority.
+/// The caller owns the journal lock for the complete validation/rename/fsync
+/// sequence.
+pub fn gc_snapshots(state_dir: &Path, journal: &Journal) -> Result<()> {
+ let root = state_dir.join("sync-snapshots");
+ let mut protected = std::collections::HashSet::new();
+ if let Some(execution) = journal
+ .committed_manifest
+ .as_ref()
+ .and_then(|manifest| manifest.execution.as_ref())
+ {
+ protected.insert(protected_snapshot(state_dir, execution)?);
+ }
+ if let Some(execution) = journal
+ .pending_sync
+ .as_ref()
+ .and_then(|pending| pending.desired.execution.as_ref())
+ {
+ protected.insert(protected_snapshot(state_dir, execution)?);
+ }
+ let entries = match std::fs::read_dir(&root) {
+ Ok(entries) => entries,
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()),
+ Err(error) => return Err(error.into()),
+ };
+ let mut entries = entries.collect::<std::io::Result<Vec<_>>>()?;
+ entries.sort_by_key(|entry| entry.file_name());
+ // Validate the complete directory before the first mutation. In
+ // particular, a late symlink must not be discovered after earlier
+ // snapshots have already been deleted.
+ for entry in &entries {
+ anyhow::ensure!(
+ !entry.file_type()?.is_symlink(),
+ "snapshot cleanup refuses symlink {}",
+ entry.path().display()
+ );
+ }
+ let directory = File::open(&root)?;
+ for entry in entries.into_iter().take(SNAPSHOT_GC_BATCH_SIZE) {
+ let metadata = entry.file_type()?;
+ if !metadata.is_dir() {
+ continue;
+ }
+ let name = entry.file_name().to_string_lossy().into_owned();
+ if protected.contains(&name) {
+ continue;
+ }
+ let tombstone = if name.ends_with(".gc") {
+ entry.path()
+ } else {
+ let tombstone = root.join(format!(".{name}.gc"));
+ anyhow::ensure!(
+ !tombstone.exists(),
+ "snapshot tombstone already exists: {}",
+ tombstone.display()
+ );
+ std::fs::rename(entry.path(), &tombstone)?;
+ directory.sync_all()?;
+ #[cfg(test)]
+ if GC_FAIL_AFTER_RENAME.swap(false, std::sync::atomic::Ordering::SeqCst) {
+ anyhow::bail!("injected snapshot GC crash after durable tombstone rename");
+ }
+ tombstone
+ };
+ std::fs::remove_dir_all(&tombstone)?;
+ directory.sync_all()?;
+ }
+ Ok(())
}
/// Test helper proving a pending generation binds before mutable discovery.
@@ -742,7 +1111,14 @@ pub fn create_snapshot(
tx_id: &str,
inventory: &Inventory,
) -> Result<SourceSnapshot> {
- create_snapshot_inner(state_dir, tx_id, inventory, None)
+ create_snapshot_inner(
+ state_dir,
+ tx_id,
+ inventory,
+ None,
+ "source-snapshot-v1",
+ u64::MAX,
+ )
}
/// Seal deterministic bulk actions together with source bytes. Extraction and
@@ -754,8 +1130,17 @@ pub fn create_prepared_snapshot(
tx_id: &str,
inventory: &Inventory,
plan: &Plan,
+ preparation_contract_digest: &str,
+ hard_budget_bytes: u64,
) -> Result<SourceSnapshot> {
- create_snapshot_inner(state_dir, tx_id, inventory, Some(plan))
+ create_snapshot_inner(
+ state_dir,
+ tx_id,
+ inventory,
+ Some(plan),
+ preparation_contract_digest,
+ hard_budget_bytes,
+ )
}
fn create_snapshot_inner(
@@ -763,6 +1148,8 @@ fn create_snapshot_inner(
tx_id: &str,
inventory: &Inventory,
plan: Option<&Plan>,
+ preparation_contract_digest: &str,
+ hard_budget_bytes: u64,
) -> Result<SourceSnapshot> {
validate_tx_id(tx_id)?;
ensure_inventory_lengths(inventory)?;
@@ -770,19 +1157,69 @@ fn create_snapshot_inner(
std::fs::create_dir_all(&root)?;
let final_dir = root.join(tx_id);
if final_dir.exists() {
- return open_snapshot(state_dir, tx_id);
+ let existing = open_snapshot(state_dir, tx_id)?;
+ anyhow::ensure!(
+ existing.preparation_contract_digest == preparation_contract_digest,
+ "existing final snapshot was prepared under a different contract"
+ );
+ let requested = inventory
+ .keys
+ .iter()
+ .zip(&inventory.digests)
+ .zip(&inventory.files)
+ .map(|((content_id, digest), file)| (content_id.as_str(), digest.as_str(), file.size))
+ .collect::<Vec<_>>();
+ let sealed = existing
+ .files
+ .iter()
+ .map(|file| {
+ (
+ file.content_id.as_str(),
+ file.content_digest.as_str(),
+ file.content_size,
+ )
+ })
+ .collect::<Vec<_>>();
+ anyhow::ensure!(
+ sealed == requested,
+ "existing final snapshot inventory differs from this preparation attempt"
+ );
+ anyhow::ensure!(
+ existing.footprint.total_bytes <= hard_budget_bytes,
+ "existing final snapshot exceeds the current logical payload limit"
+ );
+ return Ok(existing);
}
let staging = root.join(format!(".{tx_id}.partial"));
if staging.exists() {
std::fs::remove_dir_all(&staging)?;
}
std::fs::create_dir(&staging)?;
+ let mut cleanup = StagingCleanup {
+ path: staging.clone(),
+ armed: true,
+ };
let blobs = staging.join("blobs");
std::fs::create_dir(&blobs)?;
let prepared_dir = staging.join("prepared");
if plan.is_some() {
std::fs::create_dir(&prepared_dir)?;
}
+ let source_bytes = inventory.files.iter().try_fold(0u64, |total, file| {
+ total
+ .checked_add(file.size)
+ .context("snapshot source byte overflow")
+ })?;
+ if source_bytes > hard_budget_bytes {
+ anyhow::bail!(
+ "snapshot source footprint {source_bytes} bytes exceeds logical payload limit \
+ {hard_budget_bytes} bytes before preparation"
+ );
+ }
+ let mut budget = PayloadBudget {
+ used: 0,
+ limit: hard_budget_bytes,
+ };
let mut files = Vec::with_capacity(inventory.files.len());
for (ordinal, ((source, content_id), content_digest)) in inventory
.files
@@ -794,10 +1231,23 @@ fn create_snapshot_inner(
crate::content::verify(&source.path, source.size, content_digest)?;
let relative_blob = format!("blobs/{ordinal:08}");
let destination = staging.join(&relative_blob);
- copy_synced(&source.path, &destination)?;
+ copy_synced(&source.path, &destination, &mut budget)?;
crate::content::verify(&destination, source.size, content_digest)?;
+ #[cfg(test)]
+ apply_post_seal_source_replacement(&source.path)?;
let prepared = plan
- .map(|plan| prepare_artifact(&staging, ordinal, source, content_id, &destination, plan))
+ .map(|plan| {
+ prepare_artifact(
+ &staging,
+ ordinal,
+ tx_id,
+ source,
+ content_id,
+ &destination,
+ plan,
+ &mut budget,
+ )
+ })
.transpose()?;
files.push(SnapshotFile {
content_id: content_id.clone(),
@@ -807,13 +1257,42 @@ fn create_snapshot_inner(
prepared,
});
}
+ let prepared_bytes = files.iter().try_fold(0u64, |total, file| {
+ total
+ .checked_add(file.prepared.as_ref().map_or(0, |artifact| artifact.bytes))
+ .context("prepared snapshot byte overflow")
+ })?;
+ let total_bytes = source_bytes
+ .checked_add(prepared_bytes)
+ .context("total snapshot byte overflow")?;
+ anyhow::ensure!(
+ total_bytes == budget.used && total_bytes <= hard_budget_bytes,
+ "snapshot logical payload accounting mismatch"
+ );
+ let footprint = SnapshotFootprint {
+ source_bytes,
+ prepared_bytes,
+ total_bytes,
+ hard_budget_bytes,
+ };
snapshot_failpoint(1)?;
let snapshot = SourceSnapshot {
version: SNAPSHOT_VERSION,
tx_id: tx_id.to_string(),
- snapshot_digest: snapshot_digest(tx_id, &files)?,
+ preparation_contract_digest: preparation_contract_digest.to_owned(),
+ footprint: footprint.clone(),
+ started: chrono::Utc::now().to_rfc3339(),
+ snapshot_digest: String::new(),
files,
};
+ let mut snapshot = snapshot;
+ snapshot.snapshot_digest = snapshot_digest(
+ tx_id,
+ &snapshot.started,
+ preparation_contract_digest,
+ &footprint,
+ &snapshot.files,
+ )?;
write_synced_json(&staging.join("manifest.json"), &snapshot)?;
sync_dir(&blobs)?;
if plan.is_some() {
@@ -822,6 +1301,7 @@ fn create_snapshot_inner(
sync_dir(&staging)?;
snapshot_failpoint(2)?;
std::fs::rename(&staging, &final_dir)?;
+ cleanup.armed = false;
sync_dir(&root)?;
snapshot_failpoint(3)?;
open_snapshot(state_dir, tx_id)
@@ -836,8 +1316,33 @@ pub fn open_snapshot(state_dir: &Path, tx_id: &str) -> Result<SourceSnapshot> {
snapshot.version == SNAPSHOT_VERSION && snapshot.tx_id == tx_id,
"source snapshot identity mismatch"
);
+ let source_bytes = snapshot.files.iter().try_fold(0u64, |sum, file| {
+ sum.checked_add(file.content_size)
+ .context("snapshot source footprint overflow")
+ })?;
+ let prepared_bytes = snapshot.files.iter().try_fold(0u64, |sum, file| {
+ sum.checked_add(file.prepared.as_ref().map_or(0, |artifact| artifact.bytes))
+ .context("snapshot prepared footprint overflow")
+ })?;
+ let total_bytes = source_bytes
+ .checked_add(prepared_bytes)
+ .context("snapshot total footprint overflow")?;
+ anyhow::ensure!(
+ snapshot.footprint.source_bytes == source_bytes
+ && snapshot.footprint.prepared_bytes == prepared_bytes
+ && snapshot.footprint.total_bytes == total_bytes
+ && total_bytes <= snapshot.footprint.hard_budget_bytes,
+ "source snapshot footprint does not match its sealed artifacts"
+ );
anyhow::ensure!(
- snapshot.snapshot_digest == snapshot_digest(tx_id, &snapshot.files)?,
+ snapshot.snapshot_digest
+ == snapshot_digest(
+ tx_id,
+ &snapshot.started,
+ &snapshot.preparation_contract_digest,
+ &snapshot.footprint,
+ &snapshot.files,
+ )?,
"source snapshot manifest digest mismatch"
);
for file in &snapshot.files {
@@ -866,13 +1371,16 @@ pub fn open_snapshot(state_dir: &Path, tx_id: &str) -> Result<SourceSnapshot> {
Ok(snapshot)
}
+#[allow(clippy::too_many_arguments)]
fn prepare_artifact(
staging: &Path,
ordinal: usize,
+ generation_identity: &str,
source: &crate::walk::FileEntry,
content_id: &str,
snapshot_blob: &Path,
plan: &Plan,
+ budget: &mut PayloadBudget,
) -> Result<PreparedArtifact> {
let assignment = plan
.files
@@ -907,7 +1415,7 @@ fn prepare_artifact(
.iter()
.map(|(group, slug)| (group.clone(), slug.as_str()))
.collect();
- let sniffed = crate::sniff::sniff(&source.path)
+ let sniffed = crate::sniff::sniff_with_name(snapshot_blob, &source.path)
.with_context(|| format!("sniff {} for durable preparation", source.rel))?;
let relative_ndjson = format!("prepared/{ordinal:08}.ndjson");
let path = staging.join(&relative_ndjson);
@@ -915,7 +1423,10 @@ fn prepare_artifact(
.create_new(true)
.write(true)
.open(&path)?;
- let mut writer = BufWriter::new(file);
+ let mut writer = BudgetWriter {
+ inner: BufWriter::new(file),
+ budget,
+ };
let mut records = 0u64;
let mut passages = 0u64;
let mut vectors = 0u64;
@@ -952,7 +1463,10 @@ fn prepare_artifact(
fields.insert("ax_file".into(), Value::String(content_id.to_string()));
fields.insert("ax_locator".into(), Value::String(record.locator.clone()));
fields.insert("ax_dataset".into(), Value::String(slug.to_string()));
- fields.insert("ax_run".into(), Value::String("generation-pending".into()));
+ fields.insert(
+ "ax_run".into(),
+ Value::String(generation_identity.to_owned()),
+ );
fields.insert(
"ax_format".into(),
Value::String(source.path.extension().map_or_else(
@@ -962,11 +1476,10 @@ fn prepare_artifact(
);
let id = crate::ids::doc_id(slug, content_id, &record.locator);
let action = serde_json::json!({"index": {"_index": dataset.index, "_id": id}});
- if writeln!(writer, "{action}")
+ if let Err(error) = writeln!(writer, "{action}")
.and_then(|_| writeln!(writer, "{}", Value::Object(fields.clone())))
- .is_err()
{
- sink_error = Some(anyhow::anyhow!("write prepared NDJSON"));
+ sink_error = Some(anyhow::Error::new(error).context("write prepared NDJSON"));
return false;
}
records += 1;
@@ -992,7 +1505,7 @@ fn prepare_artifact(
return Err(error);
}
writer.flush()?;
- writer.get_ref().sync_all()?;
+ writer.inner.get_ref().sync_all()?;
drop(writer);
let bytes = std::fs::metadata(&path)?.len();
Ok(PreparedArtifact {
@@ -1036,14 +1549,27 @@ fn ensure_inventory_lengths(inventory: &Inventory) -> Result<()> {
Ok(())
}
-fn snapshot_digest(tx_id: &str, files: &[SnapshotFile]) -> Result<String> {
+fn snapshot_digest(
+ tx_id: &str,
+ started: &str,
+ preparation_contract_digest: &str,
+ footprint: &SnapshotFootprint,
+ files: &[SnapshotFile],
+) -> Result<String> {
let mut files = files.to_vec();
files.sort_by(|left, right| {
left.content_id
.cmp(&right.content_id)
.then_with(|| left.relative_blob.cmp(&right.relative_blob))
});
- let encoded = serde_json::to_vec(&(SNAPSHOT_VERSION, tx_id, files))?;
+ let encoded = serde_json::to_vec(&(
+ SNAPSHOT_VERSION,
+ tx_id,
+ started,
+ preparation_contract_digest,
+ footprint,
+ files,
+ ))?;
Ok(format!(
"axs1-{:032x}",
xxhash_rust::xxh3::xxh3_128(&encoded)
@@ -1062,14 +1588,18 @@ fn validate_tx_id(tx_id: &str) -> Result<()> {
Ok(())
}
-fn copy_synced(source: &Path, destination: &Path) -> Result<()> {
+fn copy_synced(source: &Path, destination: &Path, budget: &mut PayloadBudget) -> Result<()> {
let mut input = File::open(source)?;
- let mut output = OpenOptions::new()
+ let output = OpenOptions::new()
.create_new(true)
.write(true)
.open(destination)?;
+ let mut output = BudgetWriter {
+ inner: output,
+ budget,
+ };
std::io::copy(&mut input, &mut output)?;
- output.sync_all()?;
+ output.inner.sync_all()?;
Ok(())
}
@@ -1086,6 +1616,29 @@ fn sync_dir(path: &Path) -> Result<()> {
Ok(())
}
+#[cfg(test)]
+fn arm_source_mutation_after_seal(path: &Path, replacement: &[u8]) {
+ *POST_SEAL_SOURCE_REPLACEMENT.lock().unwrap() =
+ Some((path.to_path_buf(), replacement.to_vec()));
+}
+
+#[cfg(test)]
+fn apply_post_seal_source_replacement(path: &Path) -> Result<()> {
+ let replacement = {
+ let mut armed = POST_SEAL_SOURCE_REPLACEMENT.lock().unwrap();
+ if armed.as_ref().is_some_and(|(expected, _)| expected == path) {
+ armed.take().map(|(_, bytes)| bytes)
+ } else {
+ None
+ }
+ };
+ if let Some(replacement) = replacement {
+ std::fs::write(path, replacement)
+ .with_context(|| format!("inject post-seal source replacement {}", path.display()))?;
+ }
+ Ok(())
+}
+
#[cfg(test)]
fn fail_next_snapshot(boundary: u8) {
SNAPSHOT_FAILPOINT.store(boundary, std::sync::atomic::Ordering::SeqCst);
@@ -1183,13 +1736,14 @@ mod tests {
prefix: "ax".into(),
follow_symlinks: false,
chunker_identity: "chunker-v1".into(),
+ embedding_identity_sha256: "a".repeat(64),
embedding_backend: "lexical".into(),
- embedding_model: "feature-hash".into(),
- embedding_tokenizer: "none".into(),
embedding_dimension: 384,
+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),
+ embedding_resumable: true,
graph_enabled: false,
- brain: "brain".into(),
- detector_identity: "none".into(),
+ brain: "disabled".into(),
+ detector_identity: "disabled".into(),
schema_identity: "schema-v1".into(),
index_identity: "index-v1".into(),
source_policy: SourceExecutionPolicy::DurableSnapshot {
@@ -1225,9 +1779,16 @@ mod tests {
struct FakeBackend {
live: BTreeMap<String, LiveGroup>,
applications: Vec<String>,
+ provisions: usize,
+ validations: usize,
}
impl SyncOperationBackend for FakeBackend {
+ fn provision_generation(&mut self, _desired: &GenerationManifest) -> Result<()> {
+ self.provisions += 1;
+ Ok(())
+ }
+
fn apply(
&mut self,
operation: &SyncOperation,
@@ -1269,6 +1830,7 @@ mod tests {
desired: &GenerationManifest,
_snapshot: &SourceSnapshot,
) -> Result<()> {
+ self.validations += 1;
let expected: BTreeMap<String, LiveGroup> = desired
.groups
.iter()
@@ -1309,16 +1871,6 @@ mod tests {
journal.sync_begin(&pending).unwrap();
}
- fn empty_base_for(plan: &Plan) -> Plan {
- let mut base = plan.clone();
- base.files.clear();
- base.duplicate_files.clear();
- for dataset in &mut base.datasets {
- dataset.file_count = 0;
- }
- base
- }
-
#[test]
fn inventory_bridge_retains_counts_only_for_identical_content() {
let corpus = tempfile::tempdir().unwrap();
@@ -1376,8 +1928,15 @@ mod tests {
.unwrap();
let inventory = inventory(corpus.path());
let plan = plan_for(&inventory);
- let snapshot =
- create_prepared_snapshot(state.path(), "tx-prepared", &inventory, &plan).unwrap();
+ let snapshot = create_prepared_snapshot(
+ state.path(),
+ "tx-prepared",
+ &inventory,
+ &plan,
+ "test-preparation-v1",
+ u64::MAX,
+ )
+ .unwrap();
let prepared = snapshot.files[0].prepared.as_ref().unwrap();
assert_eq!(prepared.records, 2);
assert!(prepared.bytes > 0);
@@ -1403,6 +1962,153 @@ mod tests {
assert!(open_snapshot(state.path(), "tx-prepared").is_err());
}
+ #[test]
+ fn preparation_sniffs_and_extracts_only_the_sealed_blob_after_live_mutation() {
+ let _guard = SNAPSHOT_TEST_LOCK
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ let corpus = tempfile::tempdir().unwrap();
+ let state = tempfile::tempdir().unwrap();
+ let source = corpus.path().join("records.jsonl");
+ let sealed = b"{\"message\":\"sealed alpha\"}\n{\"message\":\"sealed beta\"}\n";
+ let replacement = b"id,value\n9,live mutation\n";
+ std::fs::write(&source, sealed).unwrap();
+ let inventory = inventory(corpus.path());
+ let plan = plan_for(&inventory);
+ arm_source_mutation_after_seal(&source, replacement);
+
+ let snapshot = create_prepared_snapshot(
+ state.path(),
+ "tx-post-seal-mutation",
+ &inventory,
+ &plan,
+ "test-preparation-v1",
+ u64::MAX,
+ )
+ .unwrap();
+
+ assert_eq!(std::fs::read(&source).unwrap(), replacement);
+ let prepared = snapshot.files[0].prepared.as_ref().unwrap();
+ assert_eq!(prepared.records, 2);
+ let root = state.path().join("sync-snapshots/tx-post-seal-mutation");
+ let ndjson = std::fs::read_to_string(root.join(&prepared.relative_ndjson)).unwrap();
+ assert!(ndjson.contains("sealed alpha"), "{ndjson}");
+ assert!(ndjson.contains("sealed beta"), "{ndjson}");
+ assert!(!ndjson.contains("live mutation"), "{ndjson}");
+ for document in ndjson
+ .lines()
+ .skip(1)
+ .step_by(2)
+ .map(|line| serde_json::from_str::<Value>(line).unwrap())
+ {
+ assert_eq!(document["ax_format"], "jsonl");
+ assert!(document.get("message").is_some(), "{document}");
+ assert!(document.get("id").is_none(), "{document}");
+ assert!(document.get("value").is_none(), "{document}");
+ }
+ assert_eq!(
+ std::fs::read(root.join(&snapshot.files[0].relative_blob)).unwrap(),
+ sealed
+ );
+ }
+
+ #[test]
+ fn prepared_snapshot_budget_aborts_and_removes_staging() {
+ let corpus = tempfile::tempdir().unwrap();
+ std::fs::write(corpus.path().join("a.txt"), "budgeted source bytes").unwrap();
+ let inventory =
+ crate::content::resolve(crate::walk::walk(corpus.path(), false).unwrap()).unwrap();
+ let plan = plan_for(&inventory);
+ let state = tempfile::tempdir().unwrap();
+ let error = create_prepared_snapshot(
+ state.path(),
+ "tx-budget",
+ &inventory,
+ &plan,
+ "test-preparation-v1",
+ inventory.files[0].size + 1,
+ )
+ .unwrap_err();
+ assert!(format!("{error:#}").contains("logical payload"));
+ assert!(!state
+ .path()
+ .join("sync-snapshots/.tx-budget.partial")
+ .exists());
+ assert!(!state.path().join("sync-snapshots/tx-budget").exists());
+ }
+
+ #[test]
+ fn payload_writer_refuses_bytes_before_they_reach_disk() {
+ let state = tempfile::tempdir().unwrap();
+ let path = state.path().join("payload");
+ let file = File::create(&path).unwrap();
+ let mut budget = PayloadBudget { used: 0, limit: 3 };
+ let mut writer = BudgetWriter {
+ inner: file,
+ budget: &mut budget,
+ };
+ writer.write_all(b"abc").unwrap();
+ assert!(writer.write_all(b"d").is_err());
+ writer.flush().unwrap();
+ drop(writer);
+ assert_eq!(std::fs::metadata(path).unwrap().len(), 3);
+ assert_eq!(budget.used, 3);
+ }
+
+ #[test]
+ fn gc_tombstone_crash_is_retryable_and_symlinks_are_refused() {
+ let state = tempfile::tempdir().unwrap();
+ let mut journal = Journal::open(state.path(), "root", "url", "prefix", 300, false).unwrap();
+ journal.sync_bootstrap_genesis().unwrap();
+ let snapshots = state.path().join("sync-snapshots");
+ std::fs::create_dir_all(snapshots.join("orphan")).unwrap();
+ std::fs::write(snapshots.join("orphan/blob"), b"x").unwrap();
+ GC_FAIL_AFTER_RENAME.store(true, std::sync::atomic::Ordering::SeqCst);
+ assert!(gc_snapshots(state.path(), &journal).is_err());
+ assert!(!snapshots.join("orphan").exists());
+ assert!(snapshots.join(".orphan.gc").exists());
+ gc_snapshots(state.path(), &journal).unwrap();
+ assert!(!snapshots.join(".orphan.gc").exists());
+
+ #[cfg(unix)]
+ {
+ std::fs::create_dir(snapshots.join("a-orphan")).unwrap();
+ std::os::unix::fs::symlink(state.path(), snapshots.join("hostile")).unwrap();
+ assert!(gc_snapshots(state.path(), &journal).is_err());
+ assert!(snapshots.join("a-orphan").exists());
+ assert!(snapshots.join("hostile").exists());
+ }
+ }
+
+ #[test]
+ fn gc_validates_every_protected_snapshot_before_deleting_orphans() {
+ let _guard = SNAPSHOT_TEST_LOCK
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ let corpus = tempfile::tempdir().unwrap();
+ std::fs::write(corpus.path().join("a.txt"), "protected bytes").unwrap();
+ let inventory =
+ crate::content::resolve(crate::walk::walk(corpus.path(), false).unwrap()).unwrap();
+ let plan = plan_for(&inventory);
+ let state = tempfile::tempdir().unwrap();
+ let snapshot = create_snapshot(state.path(), "tx-protected", &inventory).unwrap();
+ let mut journal = Journal::open(state.path(), "root", "url", "prefix", 300, false).unwrap();
+ journal.sync_bootstrap_genesis().unwrap();
+ let groups = groups_from_inventory(&inventory, &plan, &[]).unwrap();
+ begin(&mut journal, "tx-protected", &snapshot, plan, groups);
+ let orphan = state.path().join("sync-snapshots/orphan");
+ std::fs::create_dir(&orphan).unwrap();
+ std::fs::write(orphan.join("blob"), b"orphan").unwrap();
+ let protected_blob = state
+ .path()
+ .join("sync-snapshots/tx-protected")
+ .join(&snapshot.files[0].relative_blob);
+ std::fs::write(protected_blob, b"corrupt").unwrap();
+ assert!(gc_snapshots(state.path(), &journal).is_err());
+ assert!(orphan.exists(), "validation must precede every deletion");
+ assert!(state.path().join("sync-snapshots/tx-protected").exists());
+ }
+
#[test]
fn prepared_and_metadata_streams_preserve_pairs_under_tiny_budgets() {
let input = concat!(
@@ -1490,13 +2196,14 @@ mod tests {
"prefix": "ax",
"follow_symlinks": false,
"chunker_identity": "chunker",
+ "embedding_identity_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"embedding_backend": "lexical",
- "embedding_model": "feature-hash",
- "embedding_tokenizer": "none",
"embedding_dimension": 384,
+ "embedding_semantic_contract": "semantic_text-derived-vector.v1",
+ "embedding_resumable": true,
"graph_enabled": false,
- "brain": "brain",
- "detector_identity": "none",
+ "brain": "disabled",
+ "detector_identity": "disabled",
"schema_identity": "schema",
"index_identity": "index",
"source_policy": {
@@ -1545,7 +2252,7 @@ mod tests {
let snapshot = create_snapshot(state.path(), "tx-add", &inventory).unwrap();
let mut journal =
Journal::open(state.path(), "root", "http://engine", "ax", 300, false).unwrap();
- journal.write_plan(&empty_base_for(&plan)).unwrap();
+ journal.sync_bootstrap_genesis().unwrap();
begin(&mut journal, "tx-add", &snapshot, plan, groups);
let mut backend = FakeBackend::default();
@@ -1580,7 +2287,7 @@ mod tests {
let first_snapshot = create_snapshot(state.path(), "tx-first", &first_inventory).unwrap();
let mut journal =
Journal::open(state.path(), "root", "http://engine", "ax", 300, false).unwrap();
- journal.write_plan(&empty_base_for(&first_plan)).unwrap();
+ journal.sync_bootstrap_genesis().unwrap();
begin(
&mut journal,
"tx-first",
@@ -1671,4 +2378,39 @@ mod tests {
.groups
.is_empty());
}
+
+ #[test]
+ fn serialized_graph_identity_with_zero_operations_fails_before_backend_or_commit() {
+ let state = tempfile::tempdir().unwrap();
+ let inventory = Inventory {
+ files: vec![],
+ keys: vec![],
+ digests: vec![],
+ duplicates: vec![],
+ };
+ let snapshot = create_snapshot(state.path(), "tx-hostile", &inventory).unwrap();
+ let mut journal =
+ Journal::open(state.path(), "root", "http://engine", "ax", 300, false).unwrap();
+ journal.sync_bootstrap_genesis().unwrap();
+ let base = journal.committed_manifest.as_ref().unwrap().clone();
+ let desired = desired(1, "tx-hostile", &snapshot, Plan::default(), vec![]);
+ let valid = PendingSync::new("tx-hostile".into(), &base, desired).unwrap();
+ assert!(valid.operations.is_empty());
+ journal.sync_begin(&valid).unwrap();
+
+ let mut encoded = serde_json::to_value(journal.pending_sync.as_ref().unwrap()).unwrap();
+ encoded["desired"]["execution"]["graph_enabled"] = Value::Bool(true);
+ encoded["desired"]["execution"]["brain"] = Value::String("hostile".into());
+ encoded["desired"]["execution"]["detector_identity"] =
+ Value::String("hostile-detectors".into());
+ journal.pending_sync = Some(serde_json::from_value(encoded).unwrap());
+
+ let mut backend = FakeBackend::default();
+ assert!(replay_pending_operations(state.path(), &mut journal, &mut backend).is_err());
+ assert_eq!(backend.provisions, 0);
+ assert!(backend.applications.is_empty());
+ assert_eq!(backend.validations, 0);
+ assert_eq!(journal.committed_manifest.as_ref().unwrap().generation, 0);
+ assert!(journal.pending_sync.is_some());
+ }
}
diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rs
index 6def96b9b..1d4d5a70b 100644
--- a/engine/crates/xerj-server/src/brain.rs
+++ b/engine/crates/xerj-server/src/brain.rs
@@ -510,6 +510,7 @@ fn index_cfg(cfg: &BrainCfg, brain: &str, api_key: Option<String>) -> IndexCfg {
pdf_timeout_secs: 120,
bulk_mb: BULK_MB,
bulk_timeout_secs: 300,
+ snapshot_max_bytes: 64u64 << 30,
prefix: "ax".into(),
state_dir: None,
fresh: cfg.fresh,
diff --git a/landing/docs/recipes/zero-config-autoindex.html b/landing/docs/recipes/zero-config-autoindex.html
index edc3d0b29..ae18044c0 100644
--- a/landing/docs/recipes/zero-config-autoindex.html
+++ b/landing/docs/recipes/zero-config-autoindex.html
@@ -239,8 +239,7 @@ live indices at http://localhost:9280:
ax-docs 1 docs
ax-exports 800 docs
autoindex-catalog 9 docs</pre>
- <p><code class="inline">--fresh</code> ignores the journal and rebuilds the plan in place (ids stay idempotent), which is how you pick up files added since the last run.</p>
- <p>A rerun that resumes an existing plan indexes changed files and reports added ones as skipped. Deleting a file is refused: nothing here removes the documents it already published, so the rerun stops before touching the destination and names what is gone. Restore the file and rerun, or rebuild — in place by deleting the named indices and the state directory, or isolated under a new <code class="inline">--state-dir</code>, <code class="inline">--prefix</code> and <code class="inline">--brain</code> (or <code class="inline">--no-graph</code>), validated before you switch readers. The shared <code class="inline">autoindex-catalog</code> and the old target require explicit cleanup.</p>
+ <p><code class="inline">--fresh</code> starts without resume state only when the selected state directory has no durable plan; it never removes stale destination records. For an independent rebuild, use a new <code class="inline">--state-dir</code>, new <code class="inline">--prefix</code>, and new <code class="inline">--brain</code> when graph detection is enabled (or add <code class="inline">--no-graph</code>). Validate before switching readers. The shared <code class="inline">autoindex-catalog</code> and old target require explicit, validated cleanup. Generated journals with <code class="inline">--no-graph</code> reconcile additions, changes, deletions, renames, and no-op reruns. Legacy journals and graph-enabled generations refuse membership changes before remote mutation.</p>
<hr>
diff --git a/landing/llms-full.txt b/landing/llms-full.txt
index 698924812..e64ac3d47 100644
--- a/landing/llms-full.txt
+++ b/landing/llms-full.txt
@@ -140,18 +140,17 @@ destabilize the server.
xerj autoindex map [--url U] [--json] [--dataset SLUG]
xerj autoindex status [--url U] [--state-dir P]
---fresh ignores the journal and rebuilds the plan in place (ids stay
-idempotent); it is how you pick up files added since the last run. A rerun
-that resumes an existing plan indexes changed files and reports added ones as
-skipped. Deleting an indexed file is refused before any destination change,
-because nothing here removes the documents it already published: restore the
-file and rerun, or rebuild — in place by deleting the named indices and the
-state directory, or isolated under a new --state-dir, --prefix and --brain (or
---no-graph), validated before switching readers. The shared autoindex-catalog
-and the old target require explicit cleanup.
+--fresh starts without resume state only when the selected state directory has
+no durable plan. It never removes stale destination records. An independent
+rebuild requires a new --state-dir, new --prefix, and new --brain when graph
+detection is enabled (or --no-graph), followed by validation before switching
+readers. The shared autoindex-catalog and old target require explicit,
+validated cleanup. Generated journals with --no-graph reconcile additions,
+changes, deletions, renames, and no-op reruns. Legacy journals and graph-enabled
+generations refuse membership changes before remote mutation.
Exit codes: 0 complete · 3 completed-with-junk (junk recorded, never fatal) ·
-2 usage · 1 endpoint/journal error or a refused corpus removal.
+2 usage · 1 endpoint/journal error or refused unsafe corpus-state transition.
Pipeline: walk → sniff → sample → infer → map → extract → bulk → correlate →
catalog → verify.