AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(autoindex): enable incremental non-graph reconciliation
buger61b31ef3d8e7
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ef625a381..98cb1e89f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -386,6 +386,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0prints the funnel on demand, including which repo-level numbers arecontaminated and why.+### Changed — fail-closed autoindex restart semantics++- **`autoindex --fresh` no longer discards a durable resume plan.** A durable+ plan contains the alias, path, graph, and stale-record knowledge needed for+ safe reconciliation. `--fresh` is accepted only when the selected state+ directory has no durable plan and never cleans the destination. An+ independent rebuild must use a new state directory, new prefix, and new brain+ namespace when graph detection is enabled (or disable graph writes), be+ validated, and only then replace the old reader target. The shared catalog+ and old target require explicit, validated cleanup. Generated journals with+ `--no-graph` reconcile additions, changes, deletions, renames, and no-op+ reruns. Legacy journals and graph-enabled generations refuse membership+ changes before remote mutation.+## [1.0.0-rc.13] - 2026-08-08### Security@@ -655,7 +669,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0caveats. They are tracked for the following release rather than rushed intothis one.-### Changed — runtime-field types (can break existing mappings)- **`mappings.runtime` is now validated against ES's runtime-field typediff --git a/demo/usecases/autoindex/RECIPE_DRAFT.md b/demo/usecases/autoindex/RECIPE_DRAFT.mdindex 023080bac..e24b4819b 100644--- a/demo/usecases/autoindex/RECIPE_DRAFT.md+++ b/demo/usecases/autoindex/RECIPE_DRAFT.md@@ -203,11 +203,15 @@ done in 0.1s — 3 datasets, 5801 records live, 0 junk records, 1 junk/skipped fjust polite re-runs: in the robustness evaluation, a `kill -9` at 18 s into a200 MB file followed by a re-run converged to **byte-identical counts acrossall 9 datasets** — no duplicates. `xerj autoindex status` shows the journal-and live index counts; `--fresh` ignores the journal and rebuilds the plan in-place (ids stay idempotent), which is how a rerun picks up files added since-the last one. Deleting an indexed file is refused before any destination-change — nothing here removes the documents it already published — and the-error names what is gone plus the recovery routes.+and live index counts. `--fresh` starts without resume state only when the+selected state directory has no durable plan; it never removes stale+destination records. For an independent rebuild, use a new `--state-dir`, new+`--prefix`, and new `--brain` when graph detection is enabled (or add+`--no-graph`). Validate before switching readers. The shared+`autoindex-catalog` and old target require explicit, validated cleanup.+Generated journals with `--no-graph` reconcile additions, changes, deletions,+renames, and no-op reruns. Legacy journals and graph-enabled generations refuse+membership changes before remote mutation.## Reproduce it yourselfdiff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.mdindex b75f062df..88b27917f 100644--- a/docs/recipes/zero-config-indexing.md+++ b/docs/recipes/zero-config-indexing.md@@ -228,17 +228,14 @@ live indices at http://localhost:9280:autoindex-catalog 9 docs```-`--fresh` ignores the journal and rebuilds the plan in place (ids stay-idempotent), which is how you pick up files added since the last run.--A rerun that resumes an existing plan indexes changed files and reports added-ones as skipped. Deleting a file is refused: nothing here removes the-documents it already published, so the rerun stops before touching the-destination and names what is gone. Restore the file and rerun, or rebuild —-in place by deleting the named indices and the state directory, or isolated-under a new `--state-dir`, `--prefix` and `--brain` (or `--no-graph`),-validated before you switch readers. The shared `autoindex-catalog` and the-old target require explicit cleanup.+`--fresh` starts without resume state only when the selected state directory+has no durable plan; it never removes stale destination records. For an+independent rebuild, use a new `--state-dir`, new `--prefix`, and new+`--brain` when graph detection is enabled (or add `--no-graph`). Validate+before switching readers. The shared `autoindex-catalog` and old target require+explicit, validated cleanup. Generated journals with `--no-graph` reconcile+additions, changes, deletions, renames, and no-op reruns. Legacy journals and+graph-enabled generations refuse membership changes before remote mutation.## Reproduce it yourselfdiff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex e225b4732..6499a7f80 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -25,6 +25,7 @@ pub struct IndexCfg {pub pdf_timeout_secs: u64,pub bulk_mb: usize,pub bulk_timeout_secs: u64,+ pub snapshot_max_bytes: u64,pub prefix: String,pub state_dir: Option<PathBuf>,pub fresh: bool,@@ -75,6 +76,9 @@ pub enum Cmd {const FRESH_HELP: &str ="start without resume state only when the selected state directory has no \durable plan; an existing plan is refused and destination records are never reset";+const RESUME_POLICY_HELP: &str =+ "generated --no-graph journals reconcile add, change, delete, rename, and no-op runs; \+legacy journals and graph-enabled generations refuse membership changes before remote mutation";pub fn print_help() {println!(@@ -103,6 +107,8 @@ pub fn print_help() {valid range 1..=3600)\n\--prefix <P> index prefix (default ax)\n\--state-dir <PATH> resume journal location (default ~/.xerj/autoindex/<hash>/)\n\+ --snapshot-max-gb <N> logical payload cap for sealed source+prepared records\n\+ bytes (default 64); excludes filesystem/manifest overhead\n\--fresh {fresh_help}\n\--follow-symlinks follow symlinks (loop-safe); off by default\n\--max-file-gb <N> skip+record oversized non-streamable files (default 2)\n\@@ -143,6 +149,14 @@ pub fn print_help() {--pdf-workers bounds both phase-A parser processes and phase-B replay\n\materialization; --workers does not widen that PDF memory gate.\n\\n\+ INCREMENTAL RECONCILIATION:\n\+ Generated journals with --no-graph reconcile added, removed, moved, and changed\n\+ files. Legacy journals and graph-enabled generations remain fail-closed. Each\n\+ changed generation currently copies and prepares the full corpus (O(N)); the\n\+ latest full snapshot remains retained, and cleanup re-reads protected artifacts\n\+ to verify them. --snapshot-max-gb limits logical staged payload bytes before each\n\+ write; it is not a physical disk-space or peak-allocation guarantee.\n\+ \n\EMBEDDINGS:\n\autoindex sends semantic_text to the running server; it does not choose the\n\server's embedding backend. The default is lexical (not neural). For the\n\@@ -170,16 +184,17 @@ pub fn print_help() {cannot be computed honestly, never a filler number.\n\\n\RESUME POLICY:\n\- A durable plan supports no-op resume and same-path content replacement.\n\- Added or removed content groups are refused before remote mutation.\n\- An independent rebuild needs a new --state-dir, new --prefix, and, when\n\- graph detection is enabled, new --brain (or --no-graph). Validate before\n\- switching readers; the shared autoindex-catalog and old target require\n\- explicit, validated cleanup.\n\+ {resume_policy_help}.\n\+ If a legacy or graph-enabled journal refuses a change, do not use --fresh as\n\+ cleanup. Restore its original destination, or rebuild with a new --state-dir and\n\+ new --prefix plus a new --brain when graph is enabled (or use --no-graph).\n\+ Validate before switching readers; explicitly clean the shared\n\+ autoindex-catalog and old target only after validation.\n\\n\EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\2 usage; 1 endpoint/journal failure or unsupported corpus delta\n",- fresh_help = FRESH_HELP+ fresh_help = FRESH_HELP,+ resume_policy_help = RESUME_POLICY_HELP);}@@ -198,6 +213,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {let mut pdf_timeout_secs = 120u64;let mut bulk_mb = 8usize;let mut bulk_timeout_secs = 300u64;+ let mut snapshot_max_bytes = 64u64 << 30;let mut bulk_timeout_explicit = false;let mut prefix = "ax".to_string();let mut state_dir: Option<PathBuf> = None;@@ -272,6 +288,17 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {return Err("--bulk-timeout-secs must be in the range 1..=3600 seconds".into());}}+ "--snapshot-max-gb" => {+ let gib: u64 = it+ .next()+ .ok_or("--snapshot-max-gb needs a positive integer")?+ .parse()+ .map_err(|_| "--snapshot-max-gb needs a positive integer")?;+ snapshot_max_bytes = gib+ .checked_mul(1u64 << 30)+ .filter(|bytes| *bytes > 0)+ .ok_or("--snapshot-max-gb is too large")?;+ }"--in-flight" => {let _ = it.next(); // reserved (bulks are worker-synchronous in v1)}@@ -407,6 +434,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {pdf_timeout_secs,bulk_mb,bulk_timeout_secs,+ snapshot_max_bytes,prefix,state_dir,fresh,@@ -475,6 +503,24 @@ mod tests {assert_eq!(index(&["data"]).bulk_timeout_secs, 300);}+ #[test]+ fn snapshot_budget_defaults_and_accepts_gibibytes() {+ assert_eq!(index(&["data"]).snapshot_max_bytes, 64u64 << 30);+ assert_eq!(+ index(&["data", "--snapshot-max-gb", "7"]).snapshot_max_bytes,+ 7u64 << 30+ );+ for value in ["0", "nope", "18446744073709551615"] {+ assert!(super::parse(+ ["data", "--snapshot-max-gb", value]+ .into_iter()+ .map(str::to_owned)+ .collect()+ )+ .is_err());+ }+ }+#[test]fn fresh_help_warns_that_it_is_not_destination_reconciliation() {assert!(super::FRESH_HELP.contains("no durable plan"));@@ -482,6 +528,20 @@ mod tests {assert!(super::FRESH_HELP.contains("destination records are never reset"));}+ #[test]+ fn resume_policy_help_distinguishes_supported_and_refused_state() {+ let help = super::RESUME_POLICY_HELP;+ for claim in [+ "generated --no-graph journals",+ "add, change, delete, rename, and no-op",+ "legacy journals",+ "graph-enabled generations",+ "before remote mutation",+ ] {+ assert!(help.contains(claim), "missing resume-policy claim: {claim}");+ }+ }+#[test]fn graph_is_on_by_default_with_derived_brain() {let cfg = index(&["data"]);diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rsindex 8a228f0de..59174c85a 100644--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs@@ -263,6 +263,7 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {pdf_timeout_secs: 30,bulk_mb: 8,bulk_timeout_secs: 300,+ snapshot_max_bytes: 64 << 30,prefix: "ax".into(),state_dir: Some(state_dir.to_owned()),fresh: false,diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rsindex e3e750aa8..fb2b1285a 100644--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs@@ -133,7 +133,11 @@ impl MockEndpoint {match listener.accept() {Ok((stream, _)) => handle(stream, &server_state),Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {- if server_state.lock().unwrap().stop {+ if server_state+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner())+ .stop+ {break;}thread::yield_now();@@ -157,10 +161,13 @@ impl MockEndpoint {impl Drop for MockEndpoint {fn drop(&mut self) {- self.state.lock().unwrap().stop = true;+ self.state+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner())+ .stop = true;// Wake the nonblocking listener even on heavily loaded test hosts.let _ = TcpStream::connect(self.url.trim_start_matches("http://"));- self.join.take().unwrap().join().unwrap();+ let _ = self.join.take().unwrap().join();}}@@ -281,6 +288,10 @@ fn handle(mut stream: TcpStream, state: &Arc<Mutex<MockState>>) {.pointer("/query/term/ax_file").and_then(Value::as_str).map(str::to_owned);+ let file_key = query+ .pointer("/query/term/file_key")+ .and_then(Value::as_str)+ .map(str::to_owned);let mut locked = state.lock().unwrap();locked.delete_calls += 1;if let Some(key) = ax_file {@@ -288,17 +299,45 @@ fn handle(mut stream: TcpStream, state: &Arc<Mutex<MockState>>) {.docs.retain(|_, doc| doc.get("ax_file").and_then(Value::as_str) != Some(key.as_str()));}+ if let Some(key) = file_key {+ locked+ .catalog_docs+ .retain(|_, doc| doc.get("file_key").and_then(Value::as_str) != Some(key.as_str()));+ }(200, json!({"deleted": 0, "failures": []}))} else if method == "GET" && path.ends_with("/_count") {(200, json!({"count": state.lock().unwrap().docs.len()}))} else if method == "POST" && path.ends_with("/_search") {- // Report the REAL number of stored docs: `run_index` now verifies- // live counts against the journal (#195), so a mock that always- // answered 0 hits would (rightly) fail every successful run.- let total = state.lock().unwrap().docs.len();+ // Report the REAL number of stored docs: `run_index` verifies live+ // counts against the journal (#195), so a mock that always answered 0+ // hits would (rightly) fail every successful run. The generated path+ // additionally reads back per-file and per-catalog projections, so the+ // count is filtered by whichever identity the query names.+ let query: Value = serde_json::from_slice(&body).unwrap_or(Value::Null);+ let ax_file = query+ .pointer("/query/term/ax_file")+ .or_else(|| query.pointer("/query/bool/must/0/term/ax_file"))+ .and_then(Value::as_str);+ let file_key = query+ .pointer("/query/term/file_key")+ .and_then(Value::as_str);+ let locked = state.lock().unwrap();+ let documents = if file_key.is_some() {+ &locked.catalog_docs+ } else {+ &locked.docs+ };+ let count = documents+ .values()+ .filter(|doc| {+ ax_file.is_none_or(|key| doc.get("ax_file").and_then(Value::as_str) == Some(key))+ && file_key+ .is_none_or(|key| doc.get("file_key").and_then(Value::as_str) == Some(key))+ })+ .count();(200,- json!({"hits":{"total":{"value":total},"hits":[]},"aggregations":{}}),+ json!({"hits":{"total":{"value":count},"hits":[]},"aggregations":{}}),)} else {// ping, index creation/mapping, refresh, and catalog operations@@ -326,6 +365,18 @@ fn bulk_response(body: &[u8], state: &Arc<Mutex<MockState>>) -> Value {.and_then(|action| action.pointer("/index/_index").cloned()).and_then(|index| index.as_str().map(str::to_owned)).is_some_and(|index| index != catalog::CATALOG_INDEX);+ // Graph edge bulks are accepted and discarded: this module counts *data*+ // bulks to place injected failures, so edge bulks must not shift the+ // numbering (the graph pipeline has its own e2e suite in detect::e2e).+ let is_graph = lines+ .first()+ .and_then(|line| serde_json::from_slice::<Value>(line).ok())+ .and_then(|action| action.pointer("/index/_index").cloned())+ .and_then(|index| index.as_str().map(str::to_owned))+ .is_some_and(|index| index.ends_with("-edges"));+ if is_graph {+ return json!({"errors": false, "items": []});+ }if !is_data {// Catalog bulks mix `index` and `delete` actions (stale aliases, and// the junk sweep of #238), so walk the NDJSON rather than assuming@@ -448,6 +499,7 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {pdf_timeout_secs: 30,bulk_mb: 64,bulk_timeout_secs: 3_600,+ snapshot_max_bytes: 64 << 30,prefix: "failure-test".into(),state_dir: Some(state_dir.to_owned()),fresh: false,@@ -455,11 +507,10 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {max_file_gb: 1,sample: 50,no_semantic: true,- // These tests count data bulks to place injected failures; graph- // edge bulks would shift that numbering, so keep detection off here- // (the graph pipeline has its own e2e suite in detect::e2e).+ // This module preserves legacy graph-enabled failure behavior;+ // incremental non-graph behavior has its own HTTP suite.brain: None,- no_graph: true,+ no_graph: false,dry_run: false,json: false,quiet: true,@@ -1294,25 +1345,44 @@ fn existing_completion_is_invalidated_and_partial_visibility_is_deleted_on_resum{let locked = endpoint.state.lock().unwrap();assert!(locked.failed_once);- assert!(locked.docs.len() < rows);+ assert!(+ locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .count()+ < rows+ );}assert_eq!(run_index(config).unwrap(), 0);assert_eq!(file_done_count(state_dir.path()), 2);let locked = endpoint.state.lock().unwrap();- assert_eq!(locked.docs.len(), rows);+ assert_eq!(+ locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .count(),+ rows+ );assert_eq!(locked.delete_calls, 2,"fresh publication skips delete; replacement and repair each clean once");- assert!(locked.docs.values().all(|doc| {- doc["value"]- .as_str()- .is_some_and(|value| value.starts_with("replacement-"))- }));+ assert!(locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .all(|doc| {+ doc["value"]+ .as_str()+ .is_some_and(|value| value.starts_with("replacement-"))+ }));let mut locators: Vec<usize> = locked.docs.values()+ .filter(|doc| doc.get("value").is_some()).map(|doc| {doc["ax_locator"].as_str()@@ -1371,12 +1441,24 @@ fn resume_repairs_kills_after_plan_delete_and_final_bulk_before_file_done() {assert_eq!(run_index(config).unwrap(), 0);let locked = endpoint.state.lock().unwrap();- assert_eq!(locked.docs.len(), rows, "boundary {boundary}");- assert!(locked.docs.values().all(|doc| {- doc["value"]- .as_str()- .is_some_and(|value| value.starts_with("new-"))- }));+ assert_eq!(+ locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .count(),+ rows,+ "boundary {boundary}"+ );+ assert!(locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .all(|doc| {+ doc["value"]+ .as_str()+ .is_some_and(|value| value.starts_with("new-"))+ }));}}@@ -1394,7 +1476,7 @@ fn file_done_append_and_fsync_failures_are_fatal_and_resume_repairs() {assert_eq!(run_index(config.clone()).unwrap(), 0);fs::write(&path, "id,value\n0,new\n1,new\n").unwrap();- state::fail_next_file_done_io(io_boundary);+ state::fail_next_file_done_io(io_boundary, &state_dir.path().join("journal.ndjson"));let error = run_index(config.clone()).unwrap_err();let message = format!("{error:#}");assert!(@@ -1426,10 +1508,18 @@ fn file_done_append_and_fsync_failures_are_fatal_and_resume_repairs() {assert_eq!(run_index(config).unwrap(), 0);let locked = endpoint.state.lock().unwrap();- assert_eq!(locked.docs.len(), 2);+ assert_eq!(+ locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .count(),+ 2+ );assert!(locked.docs.values()+ .filter(|doc| doc.get("value").is_some()).all(|doc| doc["value"].as_str() == Some("new")));}}@@ -1497,10 +1587,18 @@ fn pending_generation_b_is_superseded_when_source_changes_to_c() {assert_eq!(starts.last().copied(), Some(committed_generation));drop(replay_c);let locked = endpoint.state.lock().unwrap();- assert_eq!(locked.docs.len(), 2);+ assert_eq!(+ locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .count(),+ 2+ );assert!(locked.docs.values()+ .filter(|doc| doc.get("value").is_some()).all(|doc| doc["value"].as_str() == Some("generation-c")));}@@ -1533,9 +1631,12 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {assert_unsupported_delta_without_remote_mutation(&endpoint, config.clone(), &["b.csv"], &[]);{let locked = endpoint.state.lock().unwrap();- assert_eq!(locked.docs.len(), 2);+ assert_eq!(locked.docs.len(), 3);assert!(locked.docs.values().all(|doc| {- doc["ax_path"].as_str() == Some("a.csv") && doc["value"].as_str() == Some("original")+ doc["ax_path"].as_str() == Some("a.csv")+ && doc+ .get("value")+ .is_none_or(|value| value.as_str() == Some("original"))}));}let replay = state::Journal::open(@@ -1559,11 +1660,10 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {assert_unsupported_delta_without_remote_mutation(&endpoint, config, &["b.csv"], &[]);assert_eq!(event_count(state_dir.path(), "plan"), plans_before);let locked = endpoint.state.lock().unwrap();- assert_eq!(locked.docs.len(), 2);- assert!(locked- .docs- .values()- .all(|doc| doc["value"].as_str() == Some("original")));+ assert_eq!(locked.docs.len(), 3);+ assert!(locked.docs.values().all(|doc| doc+ .get("value")+ .is_none_or(|value| value.as_str() == Some("original"))));}/// #238: a file added after the plan was frozen is skipped and reported in the@@ -1723,7 +1823,7 @@ fn partial_file_done_is_rolled_back_before_another_worker_commits() {fs::write(corpus.path().join("a.csv"), "id,value\n0,a-new\n1,a-new\n").unwrap();fs::write(corpus.path().join("b.csv"), "id,value\n2,b-new\n3,b-new\n").unwrap();- state::fail_next_file_done_io(3);+ state::fail_next_file_done_io(3, &state_dir.path().join("journal.ndjson"));let error = run_index(config.clone()).unwrap_err();assert!(format!("{error:#}").contains("partial file_done write"));@@ -1745,12 +1845,23 @@ fn partial_file_done_is_rolled_back_before_another_worker_commits() {assert_eq!(run_index(config).unwrap(), 0);let locked = endpoint.state.lock().unwrap();- assert_eq!(locked.docs.len(), 4);- assert!(locked.docs.values().all(|doc| {- doc["value"]- .as_str()- .is_some_and(|value| value.ends_with("-new"))- }));+ assert_eq!(+ locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .count(),+ 4+ );+ assert!(locked+ .docs+ .values()+ .filter(|doc| doc.get("value").is_some())+ .all(|doc| {+ doc["value"]+ .as_str()+ .is_some_and(|value| value.ends_with("-new"))+ }));}/// #195: an index write block rejects every bulk item with status 403 (thediff --git a/engine/crates/xerj-autoindex/src/generation_catalog.rs b/engine/crates/xerj-autoindex/src/generation_catalog.rsnew file mode 100644index 000000000..c78df338c--- /dev/null+++ b/engine/crates/xerj-autoindex/src/generation_catalog.rs@@ -0,0 +1,608 @@+//! Exact catalog projection for one committed corpus generation.+//!+//! Data-group operations are intentionally not catalog authority. A generation+//! which only adds junk, removes an alias, or changes one of many files still+//! has to publish a complete agent-facing data map. This module builds that+//! complete, retry-stable projection without performing HTTP I/O.++use crate::catalog;+use crate::sync::{CommittedManifest, GenerationManifest, SyncOperationKind};+use anyhow::{Context, Result};+use serde_json::{json, Value};+use std::collections::{BTreeMap, BTreeSet};++#[derive(Debug, Clone, PartialEq)]+pub struct DatasetCatalogStats {+ pub record_count: u64,+ pub junk_records: u64,+ pub bytes: u64,+ pub formats: Vec<String>,+ pub time_min: Option<String>,+ pub time_max: Option<String>,+ pub sample_queries: Vec<Value>,+ pub notes: Vec<String>,+}++#[derive(Debug, Clone, PartialEq)]+pub struct GenerationCatalogMetadata {+ /// Stable ID sealed into the pending generation. This is the `run_id`+ /// attached to every catalog document in this projection.+ pub generation_id: String,+ /// Stable timestamp sealed before publication. Retries must reuse it.+ pub started: String,+}++#[derive(Debug, Clone, PartialEq)]+pub struct CatalogProjection {+ pub generation_id: String,+ /// Complete desired managed document set, keyed by catalog `_id`.+ pub documents: BTreeMap<String, Value>,+ /// IDs managed by the previous generation which no longer exist.+ pub stale_ids: BTreeSet<String>,+}++impl CatalogProjection {+ /// Validate an exact read-back of all desired managed documents.+ ///+ /// The HTTP backend should fetch these IDs after refresh and pass the+ /// resulting `_source` values here. Extra unrelated catalog documents are+ /// outside this corpus projection and therefore are not accepted through+ /// this API.+ pub fn validate_observed(&self, observed: &BTreeMap<String, Value>) -> Result<()> {+ anyhow::ensure!(+ observed.len() == self.documents.len(),+ "catalog read-back contains {} documents; expected {}",+ observed.len(),+ self.documents.len()+ );+ for (id, expected) in &self.documents {+ let actual = observed+ .get(id)+ .with_context(|| format!("catalog read-back is missing {id}"))?;+ anyhow::ensure!(+ actual == expected,+ "catalog read-back for {id} disagrees with the sealed generation projection"+ );+ }+ Ok(())+ }+}++/// Build the complete catalog projection for `desired`.+///+/// `dataset_stats` must be exact, generation-bound values obtained from the+/// sealed prepared artifacts or from a refreshed exact read-back. Supplying+/// sampled estimates here would make the data map lie.+///+/// `prior_managed_ids` is the exact set observed through the authenticated+/// catalog endpoint for the prior committed catalog generation. The manifest+/// cannot derive correlation IDs, so callers must query or durably persist+/// this set.+pub fn project_generation(+ _base: &CommittedManifest,+ desired: &GenerationManifest,+ metadata: &GenerationCatalogMetadata,+ dataset_stats: &BTreeMap<String, DatasetCatalogStats>,+ correlations: &BTreeMap<String, Value>,+ prior_managed_ids: &BTreeSet<String>,+) -> Result<CatalogProjection> {+ anyhow::ensure!(+ !metadata.generation_id.is_empty() && !metadata.started.is_empty(),+ "catalog generation metadata is incomplete"+ );+ let execution = desired+ .execution+ .as_ref()+ .context("catalog generation has no execution identity")?;+ let mut documents = BTreeMap::new();++ for group in &desired.groups {+ let assignment = desired.plan.files.get(&group.content_id).with_context(|| {+ format!(+ "catalog group {} has no desired file assignment",+ group.group_id+ )+ })?;+ let format = assignment_format(assignment.family.as_str(), assignment.gzip);+ let (id, doc) = catalog::file_doc(+ &group.content_id,+ &group.canonical.rel,+ &format,+ "indexed",+ None,+ group.expected_records,+ 0,+ group.content_size,+ &metadata.generation_id,+ );+ insert_unique(&mut documents, id, doc)?;+ for alias in &group.aliases {+ let (id, doc) = catalog::duplicate_file_doc(+ &group.content_id,+ &alias.rel,+ &alias.path_id,+ &group.canonical.rel,+ group.content_size,+ &metadata.generation_id,+ );+ insert_unique(&mut documents, id, doc)?;+ }+ }++ for junk in &desired.plan.junk_files {+ let (id, doc) = catalog::file_doc(+ &junk.file_key,+ &junk.rel,+ &junk.format,+ &junk.status,+ Some(&junk.reason),+ 0,+ 0,+ junk.bytes,+ &metadata.generation_id,+ );+ insert_unique(&mut documents, id, doc)?;+ }++ let mut total_records = 0u64;+ let mut total_junk_records = 0u64;+ for dataset in &desired.plan.datasets {+ let stats = dataset_stats.get(&dataset.slug).with_context(|| {+ format!(+ "catalog generation has no exact statistics for dataset {}",+ dataset.slug+ )+ })?;+ total_records = total_records+ .checked_add(stats.record_count)+ .context("catalog total record count overflow")?;+ total_junk_records = total_junk_records+ .checked_add(stats.junk_records)+ .context("catalog total junk-record count overflow")?;+ let mut formats = stats.formats.clone();+ formats.sort();+ formats.dedup();+ let (id, doc) = catalog::dataset_doc(&catalog::DatasetDocInput {+ pd: dataset,+ record_count: stats.record_count,+ junk_records: stats.junk_records,+ bytes: stats.bytes,+ file_count: dataset.file_count,+ formats,+ time_min: stats.time_min.clone(),+ time_max: stats.time_max.clone(),+ sample_queries: stats.sample_queries.clone(),+ notes: stats.notes.clone(),+ run_id: &metadata.generation_id,+ });+ insert_unique(&mut documents, id, doc)?;+ }++ for (id, correlation) in correlations {+ let mut correlation = correlation.clone();+ anyhow::ensure!(+ correlation.get("doc_kind").and_then(Value::as_str) == Some("correlation"),+ "catalog correlation {id} has the wrong doc_kind"+ );+ correlation["run_id"] = Value::String(metadata.generation_id.clone());+ insert_unique(&mut documents, id.clone(), correlation)?;+ }++ let files_total = desired+ .groups+ .iter()+ .map(|group| 1usize + group.aliases.len())+ .sum::<usize>()+ .checked_add(desired.plan.junk_files.len())+ .context("catalog file count overflow")?;+ let run_doc = json!({+ "doc_kind": "run",+ "run_id": metadata.generation_id,+ "root": execution.root_identity,+ "url": execution.url,+ "prefix": execution.prefix,+ "started": metadata.started,+ "generation": desired.generation,+ "files_total": files_total,+ "unique_content_files": desired.groups.len(),+ "files_indexed": desired.groups.len(),+ "duplicate_files": desired.plan.duplicate_files.len(),+ "files_junk": desired.plan.junk_files.len(),+ "records_total": total_records,+ "junk_records_total": total_junk_records,+ "semantic": desired+ .plan+ .datasets+ .iter()+ .any(|dataset| dataset.semantic_field.is_some()),+ });+ insert_unique(+ &mut documents,+ format!("run:{}", metadata.generation_id),+ run_doc,+ )?;++ anyhow::ensure!(+ documents.values().all(|doc| {+ doc.get("run_id").and_then(Value::as_str) == Some(metadata.generation_id.as_str())+ }),+ "every desired catalog document must carry the current generation run_id"+ );++ let desired_ids: BTreeSet<String> = documents.keys().cloned().collect();+ let stale_ids = prior_managed_ids+ .difference(&desired_ids)+ .filter(|id| !desired_ids.contains(*id))+ .cloned()+ .collect();++ Ok(CatalogProjection {+ generation_id: metadata.generation_id.clone(),+ documents,+ stale_ids,+ })+}++/// Provenance rule used by the publication backend.+///+/// Upserts publish newly prepared bytes and therefore must carry the desired+/// generation ID. Metadata-only operations must preserve the existing+/// document `ax_run`; deletes leave no live provenance value.+#[cfg_attr(not(test), allow(dead_code))]+pub fn expected_ax_run<'a>(+ kind: &SyncOperationKind,+ desired_generation_id: &'a str,+ existing_ax_run: Option<&'a str>,+) -> Result<Option<&'a str>> {+ match kind {+ SyncOperationKind::Upsert => Ok(Some(desired_generation_id)),+ SyncOperationKind::Metadata => existing_ax_run+ .map(Some)+ .context("metadata-only publication has no existing ax_run to preserve"),+ SyncOperationKind::Delete => Ok(None),+ }+}++#[cfg(test)]+fn managed_non_run_ids(plan: &crate::state::Plan) -> BTreeSet<String> {+ let mut ids = BTreeSet::new();+ ids.extend(plan.files.keys().map(|key| format!("file:{key}")));+ ids.extend(+ plan.junk_files+ .iter()+ .map(|junk| format!("file:{}", junk.file_key)),+ );+ ids.extend(+ plan.duplicate_files+ .iter()+ .map(|alias| catalog::duplicate_file_id(&alias.file_key, &alias.rel, &alias.path_id)),+ );+ ids.extend(+ plan.datasets+ .iter()+ .map(|dataset| format!("ds:{}", dataset.slug)),+ );+ ids+}++fn assignment_format(family: &str, gzip: bool) -> String {+ if gzip {+ format!("{family}(gzip)")+ } else {+ family.to_owned()+ }+}++fn insert_unique(documents: &mut BTreeMap<String, Value>, id: String, doc: Value) -> Result<()> {+ anyhow::ensure!(+ documents.insert(id.clone(), doc).is_none(),+ "catalog projection contains duplicate document ID {id}"+ );+ Ok(())+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::state::{DuplicateFile, FileAssignment, JunkFile, Plan, PlanDataset};+ use crate::sync::{+ ExecutionIdentity, ManifestGroup, ManifestPath, SourceExecutionPolicy,+ EXECUTION_IDENTITY_VERSION,+ };+ fn unix_path_id(path: &str) -> String {+ let encoded = path+ .as_bytes()+ .iter()+ .map(|byte| format!("{byte:02x}"))+ .collect::<String>();+ format!("unix:{encoded}")+ }++ fn dataset() -> PlanDataset {+ PlanDataset {+ slug: "reports".into(),+ index: "ax-reports".into(),+ family: "csv".into(),+ group: None,+ specs: Vec::new(),+ time_field: None,+ semantic_field: None,+ sampled_records: 2,+ file_count: 1,+ }+ }++ fn assignment(path: &str) -> FileAssignment {+ FileAssignment {+ rel: path.into(),+ path_id: unix_path_id(path),+ is_symlink: Some(false),+ family: "csv".into(),+ gzip: false,+ content_digest: Some("digest".into()),+ assignments: vec![(None, "reports".into())],+ }+ }++ fn group(key: &str, path: &str, aliases: Vec<ManifestPath>) -> ManifestGroup {+ ManifestGroup {+ group_id: format!("group-{key}"),+ content_id: key.into(),+ content_digest: "digest".into(),+ content_size: 10,+ canonical: ManifestPath {+ path_id: unix_path_id(path),+ rel: path.into(),+ is_symlink: false,+ },+ aliases,+ dataset_slugs: vec!["reports".into()],+ expected_records: 2,+ expected_passages: 0,+ expected_vectors: 0,+ }+ }++ fn execution(generation_id: &str) -> ExecutionIdentity {+ ExecutionIdentity {+ version: EXECUTION_IDENTITY_VERSION,+ root_identity: "/corpus".into(),+ url: "http://engine".into(),+ prefix: "ax".into(),+ follow_symlinks: false,+ chunker_identity: "prepared-records-v1".into(),+ embedding_identity_sha256: "a".repeat(64),+ embedding_backend: "lexical".into(),+ embedding_dimension: 384,+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),+ embedding_resumable: true,+ graph_enabled: false,+ brain: "none".into(),+ detector_identity: "disabled".into(),+ schema_identity: "schema".into(),+ index_identity: "index".into(),+ source_policy: SourceExecutionPolicy::DurableSnapshot {+ reference: format!("sync-snapshots/{generation_id}"),+ snapshot_digest: "snapshot".into(),+ },+ }+ }++ fn manifest(+ generation: u64,+ id: &str,+ plan: Plan,+ groups: Vec<ManifestGroup>,+ ) -> GenerationManifest {+ GenerationManifest {+ generation,+ execution: Some(execution(id)),+ plan,+ groups,+ }+ }++ fn committed(generation: GenerationManifest) -> CommittedManifest {+ CommittedManifest {+ generation: generation.generation,+ manifest_digest: "base".into(),+ plan: generation.plan,+ groups: generation.groups,+ execution: generation.execution,+ }+ }++ fn stats() -> BTreeMap<String, DatasetCatalogStats> {+ BTreeMap::from([(+ "reports".into(),+ DatasetCatalogStats {+ record_count: 2,+ junk_records: 0,+ bytes: 10,+ formats: vec!["csv".into()],+ time_min: None,+ time_max: None,+ sample_queries: Vec::new(),+ notes: Vec::new(),+ },+ )])+ }++ fn metadata(id: &str) -> GenerationCatalogMetadata {+ GenerationCatalogMetadata {+ generation_id: id.into(),+ started: "2026-08-03T00:00:00Z".into(),+ }+ }++ #[test]+ fn every_current_document_moves_to_latest_catalog_generation() {+ let mut base_plan = Plan {+ datasets: vec![dataset()],+ ..Plan::default()+ };+ base_plan+ .files+ .insert("keep".into(), assignment("keep.csv"));+ base_plan+ .files+ .insert("change".into(), assignment("change.csv"));+ let base_generation = manifest(+ 1,+ "generation-1",+ base_plan.clone(),+ vec![+ group("keep", "keep.csv", vec![]),+ group("change", "change.csv", vec![]),+ ],+ );+ let base = committed(base_generation);++ let desired = manifest(+ 2,+ "generation-2",+ base_plan,+ vec![+ group("keep", "keep.csv", vec![]),+ group("change", "change.csv", vec![]),+ ],+ );+ let projection = project_generation(+ &base,+ &desired,+ &metadata("generation-2"),+ &stats(),+ &BTreeMap::new(),+ &managed_non_run_ids(&base.plan),+ )+ .unwrap();++ assert!(projection.documents.len() >= 4);+ assert!(projection+ .documents+ .values()+ .all(|doc| { doc.get("run_id").and_then(Value::as_str) == Some("generation-2") }));+ assert_eq!(+ projection.documents["file:keep"]["run_id"], "generation-2",+ "an unchanged peer remains visible in the latest data map"+ );+ }++ #[test]+ fn junk_transitions_and_removed_aliases_have_exact_stale_ids() {+ let alias = ManifestPath {+ path_id: unix_path_id("alias.csv"),+ rel: "alias.csv".into(),+ is_symlink: false,+ };+ let duplicate = DuplicateFile {+ file_key: "indexed".into(),+ rel: alias.rel.clone(),+ path_id: alias.path_id.clone(),+ is_symlink: Some(false),+ duplicate_of: "indexed.csv".into(),+ bytes: 10,+ };+ let mut base_plan = Plan {+ datasets: vec![dataset()],+ duplicate_files: vec![duplicate.clone()],+ ..Plan::default()+ };+ base_plan+ .files+ .insert("indexed".into(), assignment("indexed.csv"));+ base_plan.junk_files.push(JunkFile {+ file_key: "old-junk".into(),+ rel: "old.bin".into(),+ format: "binary".into(),+ status: "junk".into(),+ reason: "binary".into(),+ bytes: 10,+ });+ let base = committed(manifest(+ 1,+ "generation-1",+ base_plan,+ vec![group("indexed", "indexed.csv", vec![alias])],+ ));++ let mut desired_plan = Plan {+ datasets: vec![dataset()],+ ..Plan::default()+ };+ desired_plan.junk_files.push(JunkFile {+ file_key: "indexed".into(),+ rel: "indexed.csv".into(),+ format: "binary".into(),+ status: "junk".into(),+ reason: "became binary".into(),+ bytes: 10,+ });+ let desired = manifest(2, "generation-2", desired_plan, Vec::new());+ let mut no_dataset_stats = stats();+ no_dataset_stats.get_mut("reports").unwrap().record_count = 0;+ let projection = project_generation(+ &base,+ &desired,+ &metadata("generation-2"),+ &no_dataset_stats,+ &BTreeMap::new(),+ &managed_non_run_ids(&base.plan),+ )+ .unwrap();++ assert_eq!(projection.documents["file:indexed"]["status"], "junk");+ assert!(projection.stale_ids.contains("file:old-junk"));+ assert!(projection.stale_ids.contains(&catalog::duplicate_file_id(+ &duplicate.file_key,+ &duplicate.rel,+ &duplicate.path_id+ )));+ assert!(!projection.stale_ids.contains("file:indexed"));+ }++ #[test]+ fn provenance_contract_changes_only_upsert_ax_run() {+ assert_eq!(+ expected_ax_run(&SyncOperationKind::Upsert, "g2", Some("g1")).unwrap(),+ Some("g2")+ );+ assert_eq!(+ expected_ax_run(&SyncOperationKind::Metadata, "g2", Some("g1")).unwrap(),+ Some("g1")+ );+ assert_eq!(+ expected_ax_run(&SyncOperationKind::Delete, "g2", Some("g1")).unwrap(),+ None+ );+ assert!(expected_ax_run(&SyncOperationKind::Metadata, "g2", None).is_err());+ }++ #[test]+ fn exact_readback_rejects_missing_or_different_documents() {+ let plan = Plan {+ datasets: vec![dataset()],+ ..Plan::default()+ };+ let base = committed(manifest(1, "g1", plan.clone(), Vec::new()));+ let desired = manifest(2, "g2", plan, Vec::new());+ let projection = project_generation(+ &base,+ &desired,+ &metadata("g2"),+ &stats(),+ &BTreeMap::new(),+ &managed_non_run_ids(&base.plan),+ )+ .unwrap();+ projection.validate_observed(&projection.documents).unwrap();++ let mut missing = projection.documents.clone();+ missing.pop_first();+ assert!(projection.validate_observed(&missing).is_err());++ let mut changed = projection.documents.clone();+ changed.get_mut("ds:reports").unwrap()["record_count"] = json!(999);+ assert!(projection.validate_observed(&changed).is_err());+ }+}diff --git a/engine/crates/xerj-autoindex/src/generation_catalog_http_tests.rs b/engine/crates/xerj-autoindex/src/generation_catalog_http_tests.rsnew file mode 100644index 000000000..82ae54453--- /dev/null+++ b/engine/crates/xerj-autoindex/src/generation_catalog_http_tests.rs@@ -0,0 +1,519 @@+//! Backend-contract tests for whole-generation catalog publication.+//!+//! The in-memory endpoint models the observable HTTP contract: a bulk response+//! may be accepted remotely and lost locally, so applying one exact projection+//! twice must converge. Correlation IDs are supplied as independently observed+//! prior-generation IDs because the current committed manifest does not persist+//! them; a real backend must query them by the prior `run_id`.++use crate::catalog;+use crate::generation_catalog::{+ expected_ax_run, project_generation, CatalogProjection, DatasetCatalogStats,+ GenerationCatalogMetadata,+};+use crate::state::{DuplicateFile, FileAssignment, JunkFile, Plan, PlanDataset};+use crate::sync::{+ CommittedManifest, ExecutionIdentity, GenerationManifest, ManifestGroup, ManifestPath,+ SourceExecutionPolicy, SyncOperationKind, EXECUTION_IDENTITY_VERSION,+};+use anyhow::Result;+use serde_json::{json, Value};+use std::collections::{BTreeMap, BTreeSet};++#[derive(Default)]+struct CatalogEndpoint {+ documents: BTreeMap<String, Value>,+ fail_after_accept_once: bool,+}++impl CatalogEndpoint {+ fn publish(+ &mut self,+ projection: &CatalogProjection,+ independently_observed_stale: &BTreeSet<String>,+ ) -> Result<()> {+ for id in projection+ .stale_ids+ .iter()+ .chain(independently_observed_stale)+ {+ self.documents.remove(id);+ }+ self.documents.extend(projection.documents.clone());+ if std::mem::take(&mut self.fail_after_accept_once) {+ anyhow::bail!("response lost after the endpoint accepted the publication");+ }+ Ok(())+ }++ fn desired_readback(&self, projection: &CatalogProjection) -> BTreeMap<String, Value> {+ projection+ .documents+ .keys()+ .filter_map(|id| self.documents.get(id).cloned().map(|doc| (id.clone(), doc)))+ .collect()+ }+}++fn path_id(path: &str) -> String {+ let encoded = path+ .as_bytes()+ .iter()+ .map(|byte| format!("{byte:02x}"))+ .collect::<String>();+ format!("unix:{encoded}")+}++fn dataset() -> PlanDataset {+ PlanDataset {+ slug: "reports".into(),+ index: "ax-reports".into(),+ family: "csv".into(),+ group: None,+ specs: Vec::new(),+ time_field: None,+ semantic_field: None,+ sampled_records: 2,+ file_count: 0,+ }+}++fn assignment(path: &str) -> FileAssignment {+ FileAssignment {+ rel: path.into(),+ path_id: path_id(path),+ is_symlink: Some(false),+ family: "csv".into(),+ gzip: false,+ content_digest: Some(format!("digest-{path}")),+ assignments: vec![(None, "reports".into())],+ }+}++fn group(key: &str, path: &str, aliases: &[&str]) -> ManifestGroup {+ ManifestGroup {+ group_id: format!("group-{key}"),+ content_id: key.into(),+ content_digest: format!("digest-{path}"),+ content_size: 20,+ canonical: ManifestPath {+ path_id: path_id(path),+ rel: path.into(),+ is_symlink: false,+ },+ aliases: aliases+ .iter()+ .map(|alias| ManifestPath {+ path_id: path_id(alias),+ rel: (*alias).into(),+ is_symlink: false,+ })+ .collect(),+ dataset_slugs: vec!["reports".into()],+ expected_records: 2,+ expected_passages: 0,+ expected_vectors: 0,+ }+}++fn junk(key: &str, path: &str, reason: &str) -> JunkFile {+ JunkFile {+ file_key: key.into(),+ rel: path.into(),+ format: "binary".into(),+ status: "junk".into(),+ reason: reason.into(),+ bytes: 9,+ }+}++fn execution(generation_id: &str) -> ExecutionIdentity {+ ExecutionIdentity {+ version: EXECUTION_IDENTITY_VERSION,+ root_identity: "/finance".into(),+ url: "http://engine".into(),+ prefix: "ax".into(),+ follow_symlinks: false,+ chunker_identity: "prepared-records-v1".into(),+ embedding_identity_sha256: "a".repeat(64),+ embedding_backend: "lexical".into(),+ embedding_dimension: 384,+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),+ embedding_resumable: true,+ graph_enabled: false,+ brain: "none".into(),+ detector_identity: "disabled".into(),+ schema_identity: "schema".into(),+ index_identity: "index".into(),+ source_policy: SourceExecutionPolicy::DurableSnapshot {+ reference: format!("sync-snapshots/{generation_id}"),+ snapshot_digest: format!("snapshot-{generation_id}"),+ },+ }+}++fn generation(+ number: u64,+ generation_id: &str,+ mut plan: Plan,+ groups: Vec<ManifestGroup>,+) -> GenerationManifest {+ for dataset in &mut plan.datasets {+ dataset.file_count = groups+ .iter()+ .filter(|group| group.dataset_slugs.contains(&dataset.slug))+ .count();+ }+ GenerationManifest {+ generation: number,+ execution: Some(execution(generation_id)),+ plan,+ groups,+ }+}++fn committed(generation: GenerationManifest) -> CommittedManifest {+ CommittedManifest {+ generation: generation.generation,+ manifest_digest: format!("manifest-{}", generation.generation),+ plan: generation.plan,+ groups: generation.groups,+ execution: generation.execution,+ }+}++fn metadata(id: &str) -> GenerationCatalogMetadata {+ GenerationCatalogMetadata {+ generation_id: id.into(),+ started: "2026-08-03T12:00:00Z".into(),+ }+}++fn stats(records: u64, bytes: u64) -> BTreeMap<String, DatasetCatalogStats> {+ BTreeMap::from([(+ "reports".into(),+ DatasetCatalogStats {+ record_count: records,+ junk_records: 0,+ bytes,+ formats: vec!["csv".into()],+ time_min: None,+ time_max: None,+ sample_queries: vec![json!({"class": "full_text"})],+ notes: Vec::new(),+ },+ )])+}++fn plan(indexed: &[(&str, &str)], junk_files: Vec<JunkFile>) -> Plan {+ let mut plan = Plan {+ datasets: vec![dataset()],+ junk_files,+ ..Plan::default()+ };+ for (key, path) in indexed {+ plan.files.insert((*key).into(), assignment(path));+ }+ plan+}++fn prior_managed_ids(base: &CommittedManifest) -> BTreeSet<String> {+ let mut ids = BTreeSet::new();+ ids.extend(base.plan.files.keys().map(|key| format!("file:{key}")));+ ids.extend(+ base.plan+ .junk_files+ .iter()+ .map(|junk| format!("file:{}", junk.file_key)),+ );+ ids.extend(+ base.plan+ .duplicate_files+ .iter()+ .map(|alias| catalog::duplicate_file_id(&alias.file_key, &alias.rel, &alias.path_id)),+ );+ ids.extend(+ base.plan+ .datasets+ .iter()+ .map(|dataset| format!("ds:{}", dataset.slug)),+ );+ ids+}++#[test]+fn junk_only_add_change_and_delete_converge_without_hiding_unchanged_data() {+ let base_generation = generation(+ 1,+ "g1",+ plan(&[("keep", "keep.csv")], vec![]),+ vec![group("keep", "keep.csv", &[])],+ );+ let base = committed(base_generation);+ let mut endpoint = CatalogEndpoint::default();++ let added = generation(+ 2,+ "g2",+ plan(+ &[("keep", "keep.csv")],+ vec![junk("junk-a", "opaque.bin", "binary")],+ ),+ vec![group("keep", "keep.csv", &[])],+ );+ let projection = project_generation(+ &base,+ &added,+ &metadata("g2"),+ &stats(2, 20),+ &BTreeMap::new(),+ &prior_managed_ids(&base),+ )+ .unwrap();+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();+ assert_eq!(endpoint.documents["file:junk-a"]["status"], "junk");+ assert_eq!(endpoint.documents["file:keep"]["run_id"], "g2");++ let base = committed(added);+ let changed = generation(+ 3,+ "g3",+ plan(+ &[("keep", "keep.csv")],+ vec![junk("junk-a", "opaque.bin", "unsupported archive")],+ ),+ vec![group("keep", "keep.csv", &[])],+ );+ let projection = project_generation(+ &base,+ &changed,+ &metadata("g3"),+ &stats(2, 20),+ &BTreeMap::new(),+ &prior_managed_ids(&base),+ )+ .unwrap();+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();+ assert_eq!(+ endpoint.documents["file:junk-a"]["reason"],+ "unsupported archive"+ );++ let base = committed(changed);+ let deleted = generation(+ 4,+ "g4",+ plan(&[("keep", "keep.csv")], vec![]),+ vec![group("keep", "keep.csv", &[])],+ );+ let projection = project_generation(+ &base,+ &deleted,+ &metadata("g4"),+ &stats(2, 20),+ &BTreeMap::new(),+ &prior_managed_ids(&base),+ )+ .unwrap();+ assert!(projection.stale_ids.contains("file:junk-a"));+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();+ assert!(!endpoint.documents.contains_key("file:junk-a"));+ assert_eq!(endpoint.documents["file:keep"]["run_id"], "g4");+}++#[test]+fn indexed_and_junk_transitions_replace_the_same_catalog_identity() {+ let indexed = generation(+ 1,+ "g1",+ plan(&[("same", "report.csv")], vec![]),+ vec![group("same", "report.csv", &[])],+ );+ let base = committed(indexed);+ let became_junk = generation(+ 2,+ "g2",+ plan(+ &[],+ vec![junk("same", "report.csv", "no records extracted")],+ ),+ Vec::new(),+ );+ let mut empty_stats = stats(0, 0);+ empty_stats.get_mut("reports").unwrap().formats.clear();+ let junk_projection = project_generation(+ &base,+ &became_junk,+ &metadata("g2"),+ &empty_stats,+ &BTreeMap::new(),+ &prior_managed_ids(&base),+ )+ .unwrap();+ assert_eq!(junk_projection.documents["file:same"]["status"], "junk");+ assert!(!junk_projection.stale_ids.contains("file:same"));++ let base = committed(became_junk);+ let indexed_again = generation(+ 3,+ "g3",+ plan(&[("same", "report.csv")], vec![]),+ vec![group("same", "report.csv", &[])],+ );+ let projection = project_generation(+ &base,+ &indexed_again,+ &metadata("g3"),+ &stats(2, 20),+ &BTreeMap::new(),+ &prior_managed_ids(&base),+ )+ .unwrap();+ assert_eq!(projection.documents["file:same"]["status"], "indexed");+ assert!(!projection.stale_ids.contains("file:same"));+}++#[test]+fn accepted_response_retry_is_idempotent_and_exactly_validated() {+ let base = committed(generation(+ 1,+ "g1",+ plan(&[("a", "a.csv")], vec![]),+ vec![group("a", "a.csv", &[])],+ ));+ let desired = generation(+ 2,+ "g2",+ plan(&[("a", "a.csv"), ("b", "b.csv")], vec![]),+ vec![group("a", "a.csv", &[]), group("b", "b.csv", &[])],+ );+ let projection = project_generation(+ &base,+ &desired,+ &metadata("g2"),+ &stats(4, 40),+ &BTreeMap::new(),+ &prior_managed_ids(&base),+ )+ .unwrap();+ let mut endpoint = CatalogEndpoint {+ fail_after_accept_once: true,+ ..CatalogEndpoint::default()+ };+ assert!(endpoint.publish(&projection, &BTreeSet::new()).is_err());+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();+ projection+ .validate_observed(&endpoint.desired_readback(&projection))+ .unwrap();+}++#[test]+fn exact_run_dataset_file_ids_and_payloads_are_generation_bound() {+ let base = committed(generation(1, "g1", plan(&[], vec![]), Vec::new()));+ let desired = generation(+ 2,+ "g2",+ plan(&[("report", "report.csv")], vec![]),+ vec![group("report", "report.csv", &[])],+ );+ let projection = project_generation(+ &base,+ &desired,+ &metadata("g2"),+ &stats(2, 20),+ &BTreeMap::new(),+ &prior_managed_ids(&base),+ )+ .unwrap();++ assert_eq!(+ projection+ .documents+ .keys()+ .cloned()+ .collect::<BTreeSet<_>>(),+ BTreeSet::from(["ds:reports".into(), "file:report".into(), "run:g2".into()])+ );+ assert_eq!(projection.documents["run:g2"]["records_total"], 2);+ assert_eq!(projection.documents["run:g2"]["files_total"], 1);+ assert_eq!(projection.documents["ds:reports"]["record_count"], 2);+ assert_eq!(projection.documents["file:report"]["records"], 2);+ assert!(projection+ .documents+ .values()+ .all(|doc| doc["run_id"] == "g2"));+}++#[test]+fn stale_alias_and_independently_observed_correlation_are_deleted() {+ let alias = DuplicateFile {+ file_key: "report".into(),+ rel: "copy.csv".into(),+ path_id: path_id("copy.csv"),+ is_symlink: Some(false),+ duplicate_of: "report.csv".into(),+ bytes: 20,+ };+ let mut base_plan = plan(&[("report", "report.csv")], vec![]);+ base_plan.duplicate_files.push(alias.clone());+ let base = committed(generation(+ 1,+ "g1",+ base_plan,+ vec![group("report", "report.csv", &["copy.csv"])],+ ));+ let desired = generation(+ 2,+ "g2",+ plan(&[("report", "report.csv")], vec![]),+ vec![group("report", "report.csv", &[])],+ );+ let stale_correlation = "corr:reports:old".to_string();+ let mut observed_prior = prior_managed_ids(&base);+ observed_prior.insert(stale_correlation.clone());+ let projection = project_generation(+ &base,+ &desired,+ &metadata("g2"),+ &stats(2, 20),+ &BTreeMap::new(),+ &observed_prior,+ )+ .unwrap();+ let alias_id = catalog::duplicate_file_id(&alias.file_key, &alias.rel, &alias.path_id);+ assert!(projection.stale_ids.contains(&alias_id));++ let mut endpoint = CatalogEndpoint::default();+ endpoint+ .documents+ .insert(alias_id.clone(), json!({"doc_kind": "file"}));+ endpoint.documents.insert(+ stale_correlation.clone(),+ json!({"doc_kind": "correlation", "run_id": "g1"}),+ );+ endpoint.publish(&projection, &BTreeSet::new()).unwrap();+ assert!(!endpoint.documents.contains_key(&alias_id));+ assert!(!endpoint.documents.contains_key(&stale_correlation));+}++#[test]+fn metadata_publication_preserves_ax_run_while_upsert_changes_it() {+ let mut live_document = json!({"ax_path": "old.csv", "ax_run": "g1"});+ let preserved = expected_ax_run(+ &SyncOperationKind::Metadata,+ "g2",+ live_document.get("ax_run").and_then(Value::as_str),+ )+ .unwrap()+ .unwrap()+ .to_owned();+ live_document["ax_path"] = Value::String("renamed.csv".into());+ live_document["ax_run"] = Value::String(preserved);+ assert_eq!(live_document["ax_run"], "g1");++ let replacement = expected_ax_run(&SyncOperationKind::Upsert, "g3", Some("g1"))+ .unwrap()+ .unwrap();+ live_document["ax_run"] = Value::String(replacement.into());+ assert_eq!(live_document["ax_run"], "g3");+}diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rsnew file mode 100644index 000000000..231d7d288--- /dev/null+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs@@ -0,0 +1,612 @@+//! Real-HTTP end-to-end coverage for incremental corpus generations.+//!+//! The endpoint below is deliberately stateful. It applies bulk index/update+//! actions, delete-by-query, and exact count searches so `run_index` traverses+//! the same HTTP client and validation barriers as a real server.++use super::*;+use std::collections::HashMap;+use std::fs;+use std::io::{BufRead, BufReader, Read, Write};+use std::net::{TcpListener, TcpStream};+use std::path::Path;+use std::sync::{Arc, Mutex};+use std::thread;++static HTTP_E2E_LOCK: Mutex<()> = Mutex::new(());++#[derive(Default)]+struct HttpState {+ docs: HashMap<(String, String), Value>,+ requests: Vec<(String, String)>,+ data_bulk_requests: usize,+ fail_next_data_bulk: bool,+ fail_embedding_identity: bool,+ stop: bool,+ embedding_identity_sha256: String,+ saw_dataset_mapping_update: bool,+ saw_catalog_mapping_update: bool,+}++struct HttpEndpoint {+ url: String,+ state: Arc<Mutex<HttpState>>,+ join: Option<thread::JoinHandle<()>>,+}++impl HttpEndpoint {+ fn start() -> Self {+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();+ listener.set_nonblocking(true).unwrap();+ let url = format!("http://{}", listener.local_addr().unwrap());+ let state = Arc::new(Mutex::new(HttpState {+ embedding_identity_sha256: "a".repeat(64),+ ..HttpState::default()+ }));+ let server_state = Arc::clone(&state);+ let join = thread::spawn(move || loop {+ match listener.accept() {+ Ok((stream, _)) => handle_http(stream, &server_state),+ Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {+ if server_state.lock().unwrap().stop {+ break;+ }+ thread::yield_now();+ }+ Err(error) => panic!("HTTP test endpoint accept failed: {error}"),+ }+ });+ Self {+ url,+ state,+ join: Some(join),+ }+ }++ fn data_docs(&self) -> Vec<Value> {+ self.state+ .lock()+ .unwrap()+ .docs+ .iter()+ .filter(|((index, _), _)| index != catalog::CATALOG_INDEX)+ .map(|(_, doc)| doc.clone())+ .collect()+ }++ fn data_bulk_requests(&self) -> usize {+ self.state.lock().unwrap().data_bulk_requests+ }+}++impl Drop for HttpEndpoint {+ fn drop(&mut self) {+ self.state.lock().unwrap().stop = true;+ let _ = TcpStream::connect(self.url.trim_start_matches("http://"));+ self.join.take().unwrap().join().unwrap();+ }+}++fn handle_http(mut stream: TcpStream, state: &Arc<Mutex<HttpState>>) {+ let mut reader = BufReader::new(stream.try_clone().unwrap());+ let mut request_line = String::new();+ if reader.read_line(&mut request_line).unwrap() == 0 {+ return;+ }+ let mut content_length = 0usize;+ loop {+ let mut line = String::new();+ reader.read_line(&mut line).unwrap();+ if line == "\r\n" || line.is_empty() {+ break;+ }+ if let Some(value) = line+ .to_ascii_lowercase()+ .strip_prefix("content-length:")+ .map(str::trim)+ {+ content_length = value.parse().unwrap();+ }+ }+ let mut body = vec![0; content_length];+ reader.read_exact(&mut body).unwrap();+ let mut parts = request_line.split_whitespace();+ let method = parts.next().unwrap_or("");+ let path = parts.next().unwrap_or("");+ state+ .lock()+ .unwrap()+ .requests+ .push((method.to_owned(), path.to_owned()));++ let (status, response) = if method == "GET" && path == "/v1/embedding/identity" {+ let locked = state.lock().unwrap();+ if locked.fail_embedding_identity {+ (+ 500,+ json!({"error": {+ "type": "injected_identity_failure",+ "reason": "genesis recovery boundary"+ }}),+ )+ } else {+ (+ 200,+ json!({"data": {+ "version": 1,+ "backend": "lexical",+ "identity_sha256": locked.embedding_identity_sha256.clone(),+ "dimensions": 384,+ "semantic_contract": "semantic_text-derived-vector.v1",+ "resumable": true+ }, "took_ms": 0, "request_id": "incremental-http-test"}),+ )+ }+ } else if method == "POST" && path == "/_bulk" {+ let locked = state.lock().unwrap();+ assert!(+ locked.saw_dataset_mapping_update && locked.saw_catalog_mapping_update,+ "bulk publication preceded exact generation mapping updates"+ );+ drop(locked);+ bulk_http(&body, state)+ } else if method == "POST" && path.contains("/_delete_by_query") {+ let deleted = delete_http(path, &body, state);+ (200, json!({"deleted": deleted, "failures": []}))+ } else if method == "POST" && path.ends_with("/_search") {+ (200, search_http(path, &body, state))+ } else if method == "GET" && path.ends_with("/_count") {+ let index = path.trim_start_matches('/').trim_end_matches("/_count");+ let count = state+ .lock()+ .unwrap()+ .docs+ .keys()+ .filter(|(candidate, _)| candidate == index)+ .count();+ (200, json!({"count": count}))+ } else if method == "PUT" {+ let value: Value = serde_json::from_slice(&body).unwrap_or(Value::Null);+ let mut locked = state.lock().unwrap();+ if path.ends_with("/_mapping") {+ assert!(value.get("properties").is_some());+ assert!(value.get("mappings").is_none());+ if path.starts_with(&format!("/{}/", catalog::CATALOG_INDEX)) {+ locked.saw_catalog_mapping_update = true;+ } else {+ locked.saw_dataset_mapping_update = true;+ }+ } else {+ assert!(value.pointer("/mappings/properties").is_some());+ assert!(value.get("properties").is_none());+ }+ (200, json!({"acknowledged": true}))+ } else {+ // Readiness, index creation/mapping, and refresh.+ (200, json!({"acknowledged": true}))+ };+ let bytes = response.to_string();+ write!(+ stream,+ "HTTP/1.1 {status} {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",+ if status == 200 {+ "OK"+ } else {+ "Internal Server Error"+ },+ bytes.len(),+ bytes+ )+ .unwrap();+}++fn bulk_http(body: &[u8], state: &Arc<Mutex<HttpState>>) -> (u16, Value) {+ let lines: Vec<&[u8]> = body+ .split(|byte| *byte == b'\n')+ .filter(|line| !line.is_empty())+ .collect();+ let is_data = lines.iter().step_by(2).any(|line| {+ serde_json::from_slice::<Value>(line)+ .ok()+ .and_then(|action| {+ action+ .pointer("/index/_index")+ .or_else(|| action.pointer("/update/_index"))+ .and_then(Value::as_str)+ .map(|index| index != catalog::CATALOG_INDEX)+ })+ .unwrap_or(false)+ });+ let mut locked = state.lock().unwrap();+ if is_data {+ locked.data_bulk_requests += 1;+ if std::mem::take(&mut locked.fail_next_data_bulk) {+ return (+ 200,+ json!({+ "errors": true,+ "items": [{"index": {+ "status": 500,+ "error": {+ "type": "injected_failure",+ "reason": "leave durable generation pending"+ }+ }}]+ }),+ );+ }+ }+ let mut cursor = 0;+ while cursor < lines.len() {+ let action: Value = serde_json::from_slice(lines[cursor]).unwrap();+ cursor += 1;+ if let Some(meta) = action.get("delete") {+ let index = meta["_index"].as_str().unwrap().to_owned();+ let id = meta["_id"].as_str().unwrap().to_owned();+ locked.docs.remove(&(index, id));+ continue;+ }+ let payload: Value = serde_json::from_slice(lines[cursor]).unwrap();+ cursor += 1;+ if let Some(meta) = action.get("index") {+ let index = meta["_index"].as_str().unwrap().to_owned();+ let id = meta["_id"].as_str().unwrap().to_owned();+ locked.docs.insert((index, id), payload);+ } else if let Some(meta) = action.get("update") {+ let index = meta["_index"].as_str().unwrap().to_owned();+ let id = meta["_id"].as_str().unwrap().to_owned();+ let patch = payload["doc"].as_object().unwrap();+ let target = locked.docs.get_mut(&(index, id)).unwrap();+ let target = target.as_object_mut().unwrap();+ target.extend(patch.clone());+ }+ }+ (200, json!({"errors": false, "items": []}))+}++fn delete_http(path: &str, body: &[u8], state: &Arc<Mutex<HttpState>>) -> usize {+ let index = path+ .trim_start_matches('/')+ .split("/_delete_by_query")+ .next()+ .unwrap();+ let query: Value = serde_json::from_slice(body).unwrap();+ let term = query+ .pointer("/query/term")+ .and_then(Value::as_object)+ .and_then(|term| term.iter().next())+ .map(|(field, value)| (field.clone(), value.clone()));+ let mut locked = state.lock().unwrap();+ let before = locked.docs.len();+ locked.docs.retain(|(candidate, _), doc| {+ if candidate != index {+ return true;+ }+ let Some((field, expected)) = &term else {+ return false;+ };+ doc.get(field) != Some(expected)+ });+ before - locked.docs.len()+}++fn search_http(path: &str, body: &[u8], state: &Arc<Mutex<HttpState>>) -> Value {+ let indices = path+ .trim_start_matches('/')+ .trim_end_matches("/_search")+ .split(',')+ .collect::<Vec<_>>();+ let query: Value = serde_json::from_slice(body).unwrap();+ let term = query+ .pointer("/query/term")+ .or_else(|| query.pointer("/query/bool/must/0/term"))+ .and_then(Value::as_object)+ .and_then(|term| term.iter().next());+ let terms = query+ .pointer("/query/terms")+ .and_then(Value::as_object)+ .and_then(|terms| terms.iter().next());+ let filters = query+ .pointer("/query/bool/filter")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();+ let exists = query+ .pointer("/query/bool/must/1/exists/field")+ .and_then(Value::as_str);+ let locked = state.lock().unwrap();+ let matching: Vec<_> = locked+ .docs+ .iter()+ .filter(|((index, _), doc)| {+ indices.contains(&index.as_str())+ && term+ .map(|(field, value)| doc.get(field) == Some(value))+ .unwrap_or(true)+ && terms+ .map(|(field, values)| {+ values.as_array().is_some_and(|values| {+ values.iter().any(|value| doc.get(field) == Some(value))+ })+ })+ .unwrap_or(true)+ && filters.iter().all(|filter| {+ filter+ .get("term")+ .and_then(Value::as_object)+ .and_then(|term| term.iter().next())+ .is_none_or(|(field, value)| doc.get(field) == Some(value))+ })+ && exists.map(|field| doc.get(field).is_some()).unwrap_or(true)+ })+ .collect();+ let hits = matching+ .iter()+ .map(|((_, id), source)| json!({"_id": id, "_source": source}))+ .collect::<Vec<_>>();+ json!({+ "hits": {+ "total": {"value": matching.len(), "relation": "eq"},+ "hits": hits+ },+ "aggregations": {}+ })+}++fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {+ IndexCfg {+ root: root.to_owned(),+ url: url.to_owned(),+ api_key: None,+ workers: 1,+ pdf_workers: 1,+ pdf_timeout_secs: 30,+ bulk_mb: 1,+ bulk_timeout_secs: 30,+ snapshot_max_bytes: 64 << 30,+ prefix: "incremental-http".into(),+ state_dir: Some(state_dir.to_owned()),+ fresh: false,+ follow_symlinks: false,+ max_file_gb: 1,+ sample: 50,+ no_semantic: !semantic,+ brain: None,+ no_graph: true,+ dry_run: false,+ json: false,+ quiet: true,+ }+}++fn journal_events(state_dir: &Path, kind: &str) -> usize {+ fs::read_to_string(state_dir.join("journal.ndjson"))+ .unwrap()+ .lines()+ .filter_map(|line| serde_json::from_str::<Value>(line).ok())+ .filter(|event| event.get("kind").and_then(Value::as_str) == Some(kind))+ .count()+}++fn paths(docs: &[Value]) -> Vec<String> {+ let mut paths: Vec<_> = docs+ .iter()+ .filter_map(|doc| doc.get("ax_path").and_then(Value::as_str))+ .map(str::to_owned)+ .collect();+ paths.sort();+ paths.dedup();+ paths+}++fn final_snapshot_count(state_dir: &Path) -> usize {+ fs::read_dir(state_dir.join("sync-snapshots"))+ .map(|entries| {+ entries+ .filter_map(Result::ok)+ .filter(|entry| {+ entry.file_type().is_ok_and(|kind| kind.is_dir())+ && !entry.file_name().to_string_lossy().starts_with('.')+ })+ .count()+ })+ .unwrap_or(0)+}++#[test]+fn genesis_recovers_after_snapshot_budget_failure_before_sync_begin() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap();+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("a.csv"), "id,value\n1,alpha\n").unwrap();+ let endpoint = HttpEndpoint::start();+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);+ config.snapshot_max_bytes = 1;++ let error = run_index(config.clone()).unwrap_err();+ assert!(format!("{error:#}").contains("snapshot source footprint"));+ assert_eq!(journal_events(state_dir.path(), "sync_bootstrap"), 1);+ assert_eq!(journal_events(state_dir.path(), "sync_begin"), 0);+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 0);+ assert_eq!(endpoint.data_bulk_requests(), 0);++ config.snapshot_max_bytes = 64 << 30;+ assert_eq!(run_index(config).unwrap(), 0);+ assert_eq!(journal_events(state_dir.path(), "sync_bootstrap"), 1);+ assert_eq!(journal_events(state_dir.path(), "sync_begin"), 1);+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);+ assert_eq!(endpoint.data_docs().len(), 1);+}++#[test]+fn no_semantic_generation_does_not_require_embedding_identity_endpoint() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap();+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("a.csv"), "id,value\n1,alpha\n").unwrap();+ let endpoint = HttpEndpoint::start();+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);+ endpoint.state.lock().unwrap().fail_embedding_identity = true;++ assert_eq!(run_index(config).unwrap(), 0);+ assert_eq!(journal_events(state_dir.path(), "sync_bootstrap"), 1);+ assert_eq!(journal_events(state_dir.path(), "sync_begin"), 1);+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);+ assert_eq!(journal_events(state_dir.path(), "finish"), 1);+ assert_eq!(final_snapshot_count(state_dir.path()), 1);+ assert_eq!(endpoint.data_docs().len(), 1);+}++#[test]+fn initial_generation_and_noop_are_end_to_end_idempotent() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap();+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("a.csv"), "id,value\n1,alpha\n2,beta\n").unwrap();+ let endpoint = HttpEndpoint::start();+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);++ let (code, summary) = run_index_report(config.clone()).unwrap();+ assert_eq!(code, 0);+ let summary = summary.expect("generated run must return its committed run projection");+ assert_eq!(summary["generation"], 1);+ assert_eq!(summary["records_total"], 2);+ let first_docs = endpoint.data_docs();+ assert_eq!(first_docs.len(), 2);+ assert_eq!(paths(&first_docs), ["a.csv"]);+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);+ let root = corpus.path().canonicalize().unwrap();+ let preflight = state::Journal::preflight(+ state_dir.path(),+ &root.to_string_lossy(),+ &endpoint.url,+ "incremental-http",+ )+ .unwrap();+ assert!(+ preflight.committed_manifest.is_some(),+ "sync_commit must be visible to the next preflight"+ );+ drop(preflight);++ let bulks = endpoint.data_bulk_requests();+ let (_, noop_summary) = run_index_report(config).unwrap();+ assert_eq!(noop_summary.unwrap()["generation"], 1);+ assert_eq!(endpoint.data_docs(), first_docs);+ assert_eq!(endpoint.data_bulk_requests(), bulks);+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);+ assert_eq!(journal_events(state_dir.path(), "finish"), 2);+}++#[test]+fn add_change_delete_and_rename_converge_over_real_http() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap();+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("a.csv"), "id,value\n1,alpha\n").unwrap();+ let endpoint = HttpEndpoint::start();+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::write(corpus.path().join("b.csv"), "id,value\n2,bravo\n").unwrap();+ assert_eq!(run_index(config.clone()).unwrap(), 0);+ assert_eq!(paths(&endpoint.data_docs()), ["a.csv", "b.csv"]);++ fs::write(+ corpus.path().join("a.csv"),+ "id,value\n1,alpha-new\n3,charlie\n",+ )+ .unwrap();+ assert_eq!(run_index(config.clone()).unwrap(), 0);+ assert_eq!(endpoint.data_docs().len(), 3);++ fs::remove_file(corpus.path().join("b.csv")).unwrap();+ assert_eq!(run_index(config.clone()).unwrap(), 0);+ assert_eq!(paths(&endpoint.data_docs()), ["a.csv"]);++ fs::rename(+ corpus.path().join("a.csv"),+ corpus.path().join("renamed.csv"),+ )+ .unwrap();+ assert_eq!(run_index(config).unwrap(), 0);+ assert_eq!(paths(&endpoint.data_docs()), ["renamed.csv"]);+ assert_eq!(endpoint.data_docs().len(), 2);+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 5);+}++#[test]+fn pending_generation_replays_sealed_source_after_live_source_mutates() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap();+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ let source = corpus.path().join("a.csv");+ fs::write(&source, "id,value\n1,committed\n").unwrap();+ let endpoint = HttpEndpoint::start();+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::write(&source, "id,value\n1,sealed-pending\n2,sealed-second\n").unwrap();+ endpoint.state.lock().unwrap().fail_next_data_bulk = true;+ assert!(run_index(config.clone()).is_err());+ assert_eq!(journal_events(state_dir.path(), "sync_begin"), 2);+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);++ fs::write(&source, "id,value\n9,mutated-after-sync-begin\n").unwrap();+ assert_eq!(run_index(config).unwrap(), 0);+ let docs = endpoint.data_docs();+ assert_eq!(docs.len(), 2);+ let rendered = serde_json::to_string(&docs).unwrap();+ assert!(rendered.contains("sealed-pending"), "{rendered}");+ assert!(rendered.contains("sealed-second"), "{rendered}");+ assert!(!rendered.contains("mutated-after-sync-begin"), "{rendered}");+}++#[test]+fn pending_semantic_identity_drift_rejects_before_any_further_bulk() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap();+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ let source = corpus.path().join("report.txt");+ fs::write(+ &source,+ "Initial quarterly report discusses durable subscription revenue and operating income.",+ )+ .unwrap();+ let endpoint = HttpEndpoint::start();+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url, true);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::write(+ &source,+ "Updated quarterly report discusses materially stronger subscription revenue and margin.",+ )+ .unwrap();+ endpoint.state.lock().unwrap().fail_next_data_bulk = true;+ assert!(run_index(config.clone()).is_err());+ let bulks_before = endpoint.data_bulk_requests();+ endpoint.state.lock().unwrap().embedding_identity_sha256 = "b".repeat(64);++ let error = run_index(config).unwrap_err();+ let message = format!("{error:#}");+ assert!(+ message.contains("different embedding execution identity"),+ "{message}"+ );+ assert!(+ message.contains("no remote mutation was attempted"),+ "{message}"+ );+ assert_eq!(+ endpoint.data_bulk_requests(),+ bulks_before,+ "identity drift must be rejected before another data bulk"+ );+ assert_eq!(journal_events(state_dir.path(), "sync_commit"), 1);+}diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex ee89b8cf7..47cf70f80 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -12,10 +12,14 @@ pub mod dataset;pub mod detect;pub mod esclient;pub mod extract;+mod generation_catalog;+#[cfg(test)]+mod generation_catalog_http_tests;pub mod ids;pub mod infer;pub mod pool;pub mod progress;+mod reconcile_plan;pub mod resources;pub mod sniff;pub mod state;@@ -42,6 +46,267 @@ use progress::Progress;use sniff::{Family, Sniffed};use state::{DuplicateFile, FileAssignment, FileDone, JunkFile, Plan, PlanDataset};+const PREPARED_RECORDS_IDENTITY: &str = "prepared-records-v1";+const DOCUMENT_IDS_IDENTITY: &str = "document-ids-v1";+const DETECTOR_DISABLED_IDENTITY: &str = "disabled";++fn prepared_records_identity(cfg: &IndexCfg) -> Result<String> {+ let value = json!({+ "contract": PREPARED_RECORDS_IDENTITY,+ "sample": cfg.sample,+ "max_file_gb": cfg.max_file_gb,+ "no_semantic": cfg.no_semantic,+ // Worker counts are operational only. The timeout can change whether+ // a PDF yields records, so it is part of the semantic contract.+ "pdf_timeout_secs": cfg.pdf_timeout_secs,+ });+ Ok(format!(+ "{}-{:032x}",+ PREPARED_RECORDS_IDENTITY,+ xxhash_rust::xxh3::xxh3_128(&serde_json::to_vec(&value)?)+ ))+}++fn preparation_contract_digest(cfg: &IndexCfg, plan: &Plan) -> Result<String> {+ let (schema_identity, index_identity) = generation_contract_identities(plan)?;+ let encoded = serde_json::to_vec(&json!({+ "prepared_records": prepared_records_identity(cfg)?,+ "document_ids": DOCUMENT_IDS_IDENTITY,+ "schema_identity": schema_identity,+ "index_identity": index_identity,+ "plan": plan,+ }))?;+ Ok(format!(+ "axpc1-{:032x}",+ xxhash_rust::xxh3::xxh3_128(&encoded)+ ))+}++pub(crate) fn generation_contract_identities(plan: &Plan) -> Result<(String, String)> {+ let mut datasets = plan.datasets.iter().collect::<Vec<_>>();+ datasets.sort_by(|left, right| {+ left.slug+ .cmp(&right.slug)+ .then_with(|| left.index.cmp(&right.index))+ });+ let schema = datasets+ .iter()+ .map(|dataset| {+ json!({+ "slug": dataset.slug,+ "family": dataset.family,+ "group": dataset.group,+ "specs": dataset.specs,+ "time_field": dataset.time_field,+ "semantic_field": dataset.semantic_field,+ })+ })+ .collect::<Vec<_>>();+ let indices = datasets+ .iter()+ .map(|dataset| {+ json!({+ "index": dataset.index,+ "mapping": build_mapping(&dataset.specs),+ })+ })+ .collect::<Vec<_>>();+ let digest = |label: &str, value: Value| -> Result<String> {+ let bytes = serde_json::to_vec(&json!({"contract": label, "value": value}))?;+ Ok(format!("{:032x}", xxhash_rust::xxh3::xxh3_128(&bytes)))+ };+ Ok((+ digest(PREPARED_RECORDS_IDENTITY, Value::Array(schema))?,+ digest(+ DOCUMENT_IDS_IDENTITY,+ json!({+ "datasets": indices,+ "catalog_index": catalog::CATALOG_INDEX,+ "catalog_mapping": catalog::catalog_mapping(),+ "document_ids": DOCUMENT_IDS_IDENTITY,+ }),+ )?,+ ))+}++pub(crate) fn ensure_generation_mappings(es: &Es, plan: &Plan) -> Result<()> {+ for dataset in &plan.datasets {+ let mut create_body = build_mapping(&dataset.specs);+ create_body["mappings"]["properties"]["ax_paths"] = json!({"type": "keyword"});+ let update_body = json!({+ "properties": create_body["mappings"]["properties"].clone()+ });+ es.ensure_index(&dataset.index, &create_body)+ .with_context(|| format!("create generation index {}", dataset.index))?;+ es.update_mapping(&dataset.index, &update_body)+ .with_context(|| format!("install generation mapping for {}", dataset.index))?;+ }+ let mut catalog_create_body = catalog::catalog_mapping();+ catalog_create_body["mappings"]["properties"]["duplicate_of"] = json!({"type": "keyword"});+ let catalog_update_body = json!({+ "properties": catalog_create_body["mappings"]["properties"].clone()+ });+ es.ensure_index(catalog::CATALOG_INDEX, &catalog_create_body)?;+ es.update_mapping(catalog::CATALOG_INDEX, &catalog_update_body)+ .context("install generation catalog mapping")+}++fn begin_non_graph_generation(+ es: &Es,+ journal: &mut state::Journal,+ state_dir: &Path,+ cfg: &IndexCfg,+ root_identity: &str,+ inventory: &content::Inventory,+ plan: Plan,+) -> Result<()> {+ anyhow::ensure!(+ cfg.no_graph,+ "non-graph generation cutover requires --no-graph"+ );+ anyhow::ensure!(+ journal.pending_sync.is_none(),+ "cannot prepare over an existing pending generation"+ );+ if journal.committed_manifest.is_none() {+ journal.sync_bootstrap_genesis()?;+ }+ let base = journal+ .committed_manifest+ .as_ref()+ .context("generation cutover has no committed base")?+ .clone();+ let tx_id = format!("{}-g{}", journal.run_id, base.generation + 1);+ let preparation_contract = preparation_contract_digest(cfg, &plan)?;+ let snapshot = sync_executor::create_prepared_snapshot(+ state_dir,+ &tx_id,+ inventory,+ &plan,+ &preparation_contract,+ cfg.snapshot_max_bytes,+ )?;+ let chunker_identity = prepared_records_identity(cfg)?;+ let semantic = plan+ .datasets+ .iter()+ .any(|dataset| dataset.semantic_field.is_some());+ let identity = if semantic {+ let identity = es+ .embedding_execution_identity()+ .context("generation cutover could not pin the server embedding execution identity")?;+ anyhow::ensure!(+ identity.resumable,+ "generation cutover requires a resumable embedding execution identity: {}",+ identity+ .non_resumable_reason+ .as_deref()+ .unwrap_or("the server did not provide an immutable identity")+ );+ journal.pin_embedding_identity(+ &identity.identity_sha256,+ identity.resumable,+ identity.non_resumable_reason.as_deref(),+ )?;+ identity+ } else {+ crate::esclient::EmbeddingExecutionIdentity {+ version: 1,+ backend: "disabled".into(),+ identity_sha256: "0".repeat(64),+ dimensions: 1,+ semantic_contract: "disabled-no-semantic-fields-v1".into(),+ resumable: true,+ non_resumable_reason: None,+ }+ };++ let aliases_by_content = plan.duplicate_files.iter().fold(+ HashMap::<&str, Vec<sync::ManifestPath>>::new(),+ |mut aliases, alias| {+ aliases+ .entry(alias.file_key.as_str())+ .or_default()+ .push(sync::ManifestPath {+ path_id: alias.path_id.clone(),+ rel: alias.rel.clone(),+ is_symlink: alias.is_symlink.unwrap_or(false),+ });+ aliases+ },+ );+ let file_by_content = inventory+ .keys+ .iter()+ .zip(&inventory.files)+ .zip(&inventory.digests)+ .map(|((key, file), digest)| (key.as_str(), (file, digest)))+ .collect::<HashMap<_, _>>();+ let mut candidates = Vec::with_capacity(plan.files.len());+ for (content_id, assignment) in &plan.files {+ let (file, digest) = file_by_content+ .get(content_id.as_str())+ .with_context(|| format!("planned content {content_id} is absent from inventory"))?;+ let mut paths = vec![sync::ManifestPath {+ path_id: assignment.path_id.clone(),+ rel: assignment.rel.clone(),+ is_symlink: assignment.is_symlink.unwrap_or(file.is_symlink),+ }];+ paths.extend(+ aliases_by_content+ .get(content_id.as_str())+ .cloned()+ .unwrap_or_default(),+ );+ candidates.push(sync::DesiredContentGroup {+ content_id: content_id.clone(),+ content_digest: (*digest).clone(),+ content_size: file.size,+ dataset_slugs: assignment+ .assignments+ .iter()+ .map(|(_, slug)| slug.clone())+ .collect(),+ paths,+ expected_records: 0,+ expected_passages: 0,+ expected_vectors: 0,+ });+ }+ let mut groups = sync::reconcile_groups(&base.groups, candidates)?;+ sync_executor::bind_prepared_counts(&mut groups, &snapshot, &inventory.keys)?;+ let (schema_identity, index_identity) = generation_contract_identities(&plan)?;+ let desired = sync::GenerationManifest {+ generation: base.generation + 1,+ execution: Some(sync::ExecutionIdentity {+ version: sync::EXECUTION_IDENTITY_VERSION,+ root_identity: root_identity.to_owned(),+ url: cfg.url.clone(),+ prefix: cfg.prefix.clone(),+ follow_symlinks: cfg.follow_symlinks,+ chunker_identity,+ embedding_identity_sha256: identity.identity_sha256,+ embedding_backend: identity.backend,+ embedding_dimension: identity.dimensions,+ embedding_semantic_contract: identity.semantic_contract,+ embedding_resumable: identity.resumable,+ graph_enabled: false,+ brain: "disabled".into(),+ detector_identity: DETECTOR_DISABLED_IDENTITY.into(),+ schema_identity,+ index_identity,+ source_policy: sync::SourceExecutionPolicy::DurableSnapshot {+ reference: format!("sync-snapshots/{tx_id}"),+ snapshot_digest: snapshot.snapshot_digest,+ },+ }),+ plan,+ groups,+ };+ let pending = sync::PendingSync::new(tx_id, &base, desired)?;+ journal.sync_begin(&pending)+}+#[derive(Debug, PartialEq, Eq)]struct CliErrorRoute {exit_code: i32,@@ -1392,6 +1657,117 @@ fn invocation_report_timestamps((started.to_rfc3339(), summary_generated_at.to_rfc3339())}+fn pin_pending_embedding_identity(+ es: &Es,+ journal: &mut state::Journal,+ pending: &sync::PendingSync,+) -> Result<()> {+ if !pending+ .desired+ .plan+ .datasets+ .iter()+ .any(|dataset| dataset.semantic_field.is_some())+ {+ return Ok(());+ }+ let expected = pending+ .desired+ .execution+ .as_ref()+ .context("pending semantic generation has no execution identity")?;+ let current = es.embedding_execution_identity().context(+ "pending semantic generation cannot verify the server embedding execution identity",+ )?;+ anyhow::ensure!(+ current.resumable,+ "pending semantic generation cannot resume because the current embedding backend is not \+ resumable: {}; restore the original backend or rebuild with --fresh and a new --prefix",+ current+ .non_resumable_reason+ .as_deref()+ .unwrap_or("the server did not provide a stable execution identity")+ );+ anyhow::ensure!(+ current.identity_sha256 == expected.embedding_identity_sha256+ && current.backend == expected.embedding_backend+ && current.dimensions == expected.embedding_dimension+ && current.semantic_contract == expected.embedding_semantic_contract+ && current.resumable == expected.embedding_resumable,+ "pending semantic generation was prepared for a different embedding execution identity; \+ no remote mutation was attempted. Restore the original embedding backend, or discard the \+ pending generation and rebuild with --fresh and a new --prefix"+ );+ journal.pin_embedding_identity(+ ¤t.identity_sha256,+ current.resumable,+ current.non_resumable_reason.as_deref(),+ )+}++fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -> Result<Value> {+ let committed = journal+ .committed_manifest+ .as_ref()+ .context("generated run finished without committed generation authority")?;+ let execution = committed+ .execution+ .as_ref()+ .context("generated run finished without execution identity")?;+ let generation = committed.generation;+ let dataset_count = committed.plan.datasets.len();+ let sync::SourceExecutionPolicy::DurableSnapshot { reference, .. } = &execution.source_policy+ else {+ anyhow::bail!("generated run does not reference a durable snapshot");+ };+ let run_id = reference+ .strip_prefix("sync-snapshots/")+ .context("generated run snapshot reference is not state-relative")?+ .to_owned();+ let response = es.search(+ catalog::CATALOG_INDEX,+ &json!({+ "size": 2,+ "query": {"bool": {"filter": [+ {"term": {"run_id": &run_id}},+ {"term": {"doc_kind": "run"}}+ ]}}+ }),+ )?;+ let hits = response+ .pointer("/hits/hits")+ .and_then(Value::as_array)+ .context("generated run summary query has no hits")?;+ anyhow::ensure!(+ hits.len() == 1,+ "generated run summary query returned {} documents; expected exactly one",+ hits.len()+ );+ let summary = hits[0]+ .get("_source")+ .cloned()+ .context("generated run summary hit has no _source")?;+ anyhow::ensure!(+ summary.get("generation").and_then(Value::as_u64) == Some(generation),+ "generated run summary generation disagrees with committed authority"+ );+ journal.finish(&summary)?;+ if cfg.json {+ println!("{summary}");+ } else if !cfg.quiet {+ println!(+ "generation {} committed — {} datasets, {} records live",+ generation,+ dataset_count,+ summary+ .get("records_total")+ .and_then(Value::as_u64)+ .unwrap_or(0)+ );+ }+ Ok(summary)+}+fn run_index(cfg: IndexCfg) -> Result<i32> {run_index_report(cfg).map(|(code, _)| code)}@@ -1473,6 +1849,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// must never classify a path snapshot taken while another owner was// publishing or replacing the durable plan.let preflight = state::Journal::preflight(&state_dir, &root_str, &cfg.url, &cfg.prefix)?;+ let genesis_recovery = preflight+ .committed_manifest+ .as_ref()+ .is_some_and(|manifest| manifest.generation == 0 && manifest.groups.is_empty());// A durable sync_begin owns the desired generation. Never rediscover and// replan from a mutable source tree while that transaction is pending.// Operation handlers are deliberately not enabled by this foundation@@ -1487,6 +1867,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {cfg.bulk_timeout_secs,cfg.fresh,)?;+ sync_executor::gc_snapshots(&state_dir, &journal)?;+ let pending = journal+ .pending_sync+ .as_ref()+ .context("preflight reported a pending generation but authoritative replay did not")?+ .clone();+ pin_pending_embedding_identity(&es, &mut journal, &pending)?;pr.phase("replay", 0, 0);let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;@@ -1494,8 +1881,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// belongs to that surface, so `--progress none` stays silent and// `--progress json` stays one parseable stream (#241).pr.note("autoindex: resumed and committed pending corpus generation from durable source");- pr.finish(true, 0, "completed", &[]);- return Ok((0, None));+ let summary = finish_generated_run(&es, &mut journal, &cfg)?;+ return Ok((0, Some(summary)));}// Totals are unknown until the walk returns, so this phase honestly// reports `pct=unknown` and proves liveness with the clock alone.@@ -1524,33 +1911,191 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// real work, and before #241 it was minutes with no output at all.pr.phase("hash", discovered_files.len() as u64, discovered_bytes);let mut inventory = content::resolve_reporting(discovered_files, &|bytes| pr.item_done(bytes))?;- if let Some(prior_plan) = preflight.plan.as_ref() {- let comparison_keys = if cfg.fresh {- inventory.keys.clone()+ if cfg.no_graph && preflight.committed_manifest.is_some() && !genesis_recovery {+ let mut journal = state::Journal::open_after_preflight(+ preflight,+ &root_str,+ &cfg.url,+ &cfg.prefix,+ cfg.bulk_timeout_secs,+ cfg.fresh,+ )?;+ sync_executor::gc_snapshots(&state_dir, &journal)?;+ let base = journal+ .committed_manifest+ .as_ref()+ .context("generated journal lost its committed manifest")?+ .clone();+ // Phase A's scan runs in the run's own pool and reports every file it+ // touches, exactly as the legacy route does: `--workers` has to bound+ // the CPU-bound phase for the knob to mean anything (#240 §2), and no+ // file may drop out of the progress denominator (#241). Run-local PDF+ // artifact reuse is disabled here on purpose — this route publishes+ // from its own durable snapshot and never reaches the legacy phase B,+ // so a retained artifact could never be replayed (#248).+ let reconcile_budget = extract::pdf::ExtractionSpoolBudget::new(0, 0);+ let reconcile_ctx = PhaseAContext {+ state_dir: &state_dir,+ budget: &reconcile_budget,+ capacity_warning: None,+ progress: &pr,+ };+ pr.phase(+ "scan",+ inventory.files.len() as u64,+ inventory.files.iter().map(|file| file.size).sum(),+ );+ let scans: Vec<FileScan> = crate::pool::install(|| {+ use rayon::prelude::*;+ inventory+ .files+ .par_iter()+ .zip(inventory.digests.par_iter())+ .map(|(file, digest)| {+ let _in_flight = pr.file(&file.rel, file.size);+ scan_file(+ &file.path,+ file.size,+ digest,+ &reconcile_ctx,+ cfg.sample,+ cfg.max_file_gb,+ )+ })+ .collect()+ });+ let plan = reconcile_plan::reconcile_plan(&inventory, &base.plan, scans)?;+ if serde_json::to_value(&plan)? == serde_json::to_value(&base.plan)? {+ if let Some(expected) = &base.execution {+ let (schema_identity, index_identity) = generation_contract_identities(&plan)?;+ anyhow::ensure!(+ expected.root_identity == root_str+ && expected.url == cfg.url+ && expected.prefix == cfg.prefix+ && expected.follow_symlinks == cfg.follow_symlinks+ && expected.chunker_identity == prepared_records_identity(&cfg)?+ && !expected.graph_enabled+ && expected.brain == "disabled"+ && expected.detector_identity == DETECTOR_DISABLED_IDENTITY+ && expected.schema_identity == schema_identity+ && expected.index_identity == index_identity,+ "autoindex execution configuration changed since the committed generation; \+ rebuild with --fresh and a new --prefix"+ );+ if plan+ .datasets+ .iter()+ .any(|dataset| dataset.semantic_field.is_some())+ {+ let current = es.embedding_execution_identity()?;+ anyhow::ensure!(+ current.resumable+ && current.identity_sha256 == expected.embedding_identity_sha256+ && current.backend == expected.embedding_backend+ && current.dimensions == expected.embedding_dimension+ && current.semantic_contract == expected.embedding_semantic_contract,+ "embedding execution identity changed since this autoindex journal was \+ created; refusing to mix vector spaces. No remote mutation was attempted. \+ Restore the original identity or rebuild with --fresh and a new --prefix"+ );+ }+ }+ let summary = finish_generated_run(&es, &mut journal, &cfg)?;+ return Ok((0, Some(summary)));+ }+ begin_non_graph_generation(+ &es,+ &mut journal,+ &state_dir,+ &cfg,+ &root_str,+ &inventory,+ plan,+ )?;+ let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);+ sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;+ let summary = finish_generated_run(&es, &mut journal, &cfg)?;+ return Ok((0, Some(summary)));+ }+ if cfg.no_graph && preflight.plan.is_some() && !genesis_recovery {+ let replacement_state = state_dir.with_extension("generation-v1");+ let replacement_prefix = format!("{}-generation-v1", cfg.prefix);+ let reasons = if preflight.legacy_migration_reasons.is_empty() {+ "legacy journal has no complete generated-manifest authority".to_owned()} else {- // Ordinary resume preserves planned-key identity for supported- // same-path replacement and legacy plans.- select_resume_plan_keys(- &inventory.files,- &inventory.keys,- prior_plan,- &state_dir.join("journal.ndjson"),- )?- .into_iter()- .zip(inventory.keys.iter())- .map(|(planned, current)| planned.unwrap_or_else(|| current.clone()))- .collect()+ preflight.legacy_migration_reasons.join("; ")};- let delta =- UnsupportedInventoryDelta::between(&inventory.files, &comparison_keys, prior_plan);- if cfg.fresh {- // Even an apparently unchanged content-key set can have alias,- // path, graph, catalog, or partial-publication history that only- // the old plan can reconcile. Route 1 cannot safely discard it.- return Err(delta.into_fresh_error());+ let mut rebuild_argv = vec![+ "xerj".to_owned(),+ "autoindex".to_owned(),+ cfg.root.to_string_lossy().into_owned(),+ "--no-graph".to_owned(),+ "--url".to_owned(),+ cfg.url.clone(),+ "--state-dir".to_owned(),+ replacement_state.to_string_lossy().into_owned(),+ "--prefix".to_owned(),+ replacement_prefix,+ "--workers".to_owned(),+ cfg.workers.to_string(),+ "--pdf-workers".to_owned(),+ cfg.pdf_workers.to_string(),+ "--pdf-timeout-secs".to_owned(),+ cfg.pdf_timeout_secs.to_string(),+ "--bulk-mb".to_owned(),+ cfg.bulk_mb.to_string(),+ "--bulk-timeout-secs".to_owned(),+ cfg.bulk_timeout_secs.to_string(),+ "--snapshot-max-gb".to_owned(),+ (cfg.snapshot_max_bytes >> 30).to_string(),+ "--max-file-gb".to_owned(),+ cfg.max_file_gb.to_string(),+ "--sample".to_owned(),+ cfg.sample.to_string(),+ ];+ if cfg.no_semantic {+ rebuild_argv.push("--no-semantic".to_owned());}- if !delta.is_empty() {- return Err(delta.into_error());+ if cfg.follow_symlinks {+ rebuild_argv.push("--follow-symlinks".to_owned());+ }+ anyhow::bail!(+ "this state directory contains a legacy nonempty plan that cannot become generation \+ authority: {reasons}. Start an independent rebuild using this argv JSON (no shell \+ quoting required): {:?}. Keep XERJ_API_KEY set when the endpoint requires \+ authentication",+ rebuild_argv+ );+ }+ if !genesis_recovery {+ if let Some(prior_plan) = preflight.plan.as_ref() {+ let comparison_keys = if cfg.fresh {+ inventory.keys.clone()+ } else {+ // Ordinary resume preserves planned-key identity for supported+ // same-path replacement and legacy plans.+ select_resume_plan_keys(+ &inventory.files,+ &inventory.keys,+ prior_plan,+ &state_dir.join("journal.ndjson"),+ )?+ .into_iter()+ .zip(inventory.keys.iter())+ .map(|(planned, current)| planned.unwrap_or_else(|| current.clone()))+ .collect()+ };+ let delta =+ UnsupportedInventoryDelta::between(&inventory.files, &comparison_keys, prior_plan);+ if cfg.fresh {+ // Even an apparently unchanged content-key set can have alias,+ // path, graph, catalog, or partial-publication history that only+ // the old plan can reconcile. Route 1 cannot safely discard it.+ return Err(delta.into_fresh_error());+ }+ if !delta.is_empty() {+ return Err(delta.into_error());+ }}}let mut journal = state::Journal::open_after_preflight(@@ -1561,7 +2106,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {cfg.bulk_timeout_secs,cfg.fresh,)?;- let resumed_with_plan = journal.plan.is_some();+ sync_executor::gc_snapshots(&state_dir, &journal)?;+ let resumed_with_plan = journal.plan.is_some() && !genesis_recovery;if inventory.files.is_empty() && !resumed_with_plan {println!("no files found under {}", cfg.root.display());pr.finish(true, 0, "no-files", &[]);@@ -1698,10 +2244,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {plan.duplicate_files = inventory.duplicates.clone();}alias_paths_to_replace.extend(inventory.duplicates.iter().map(|alias| alias.rel.clone()));- let files = inventory.files;- let keys = inventory.keys;- let digests = inventory.digests;- let duplicate_files = inventory.duplicates;+ let files = inventory.files.clone();+ let keys = inventory.keys.clone();+ let digests = inventory.digests.clone();+ let duplicate_files = inventory.duplicates.clone();let paths_discovered = files.len() + duplicate_files.len();let planned_bytes: u64 = files.iter().map(|f| f.size).sum();if !duplicate_files.is_empty() {@@ -1731,20 +2277,32 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// open, phase-B workers are what hold bulk buffers. Reserving for the// larger of the two can only make this optional accelerator hand capacity// back — never take headroom the run still needs.- let (pdf_spool_budget, pdf_spool_capacity_warning) =+ //+ // `--no-graph` is the exception, and it is a decision this rebase had to+ // make between two landed changes. #248's artifact is a phase A→B+ // accelerator, but every `--no-graph` route below returns from the+ // generated path — which publishes from its own sealed snapshot and never+ // reaches the legacy phase B. A retained artifact could therefore never be+ // replayed, so admission is disabled instead of spooling gigabytes nothing+ // will read. Nothing observable changes: the generated run document does+ // not carry `pdf_extraction_reuse`.+ let (pdf_spool_budget, pdf_spool_capacity_warning) = if cfg.no_graph {+ (extract::pdf::ExtractionSpoolBudget::new(0, 0), None)+ } else {extract::pdf::ExtractionSpoolBudget::for_state_dir(&state_dir,cfg.workers.max(scan_threads),cfg.pdf_workers,cfg.bulk_mb,- );+ )+ };let phase_a_context = PhaseAContext {state_dir: &state_dir,budget: &pdf_spool_budget,capacity_warning: pdf_spool_capacity_warning.as_deref(),progress: &pr,};- let mut plan: Plan = if let Some(p) = journal.plan.clone() {+ let mut plan: Plan = if let Some(p) = journal.plan.clone().filter(|_| !genesis_recovery) {p} else {pr.phase("scan", files.len() as u64, planned_bytes);@@ -1830,6 +2388,22 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {return Ok((0, None));}+ if cfg.no_graph && !resumed_with_plan {+ begin_non_graph_generation(+ &es,+ &mut journal,+ &state_dir,+ &cfg,+ &root_str,+ &inventory,+ plan,+ )?;+ let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);+ sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;+ let summary = finish_generated_run(&es, &mut journal, &cfg)?;+ return Ok((0, Some(summary)));+ }+if plan.datasets.iter()@@ -3585,6 +4159,9 @@ fn run_status(cfg: StatusCfg) -> Result<i32> {let mut done = 0u64;let mut records = 0u64;let mut finished = false;+ let mut generation: Option<u64> = None;+ let mut generation_files: Option<u64> = None;+ let mut generation_records: Option<u64> = None;let mut graph_line: Option<String> = None;if let Ok(f) = std::fs::File::open(&jp) {use std::io::BufRead;@@ -3600,10 +4177,17 @@ fn run_status(cfg: StatusCfg) -> Result<i32> {}Some("finish") => {finished = true;+ generation = v.pointer("/summary/generation").and_then(Value::as_u64);+ generation_files =+ v.pointer("/summary/files_indexed").and_then(Value::as_u64);+ generation_records =+ v.pointer("/summary/records_total").and_then(Value::as_u64);// Latest finish wins — the summary embeds the run// doc, whose `graph` block is the edge count of// record for this journal.- if let Some(g) = v.pointer("/summary/graph") {+ if let Some(g) =+ v.pointer("/summary/graph").filter(|graph| !graph.is_null())+ {graph_line = Some(format!("graph: {} edges written to {} (brain {})",g.get("edges_written").and_then(Value::as_u64).unwrap_or(0),@@ -3618,12 +4202,15 @@ fn run_status(cfg: StatusCfg) -> Result<i32> {}}println!(- "journal {} — root {} — {} files done, {} records, {}",+ "journal {} — root {} — {} files done, {} records, {}{}",jp.display(),root,- done,- records,- if finished { "FINISHED" } else { "in progress" }+ generation_files.unwrap_or(done),+ generation_records.unwrap_or(records),+ if finished { "FINISHED" } else { "in progress" },+ generation+ .map(|generation| format!(" (generation {generation})"))+ .unwrap_or_default());if let Some(line) = graph_line {println!(" {line}");@@ -4043,6 +4630,7 @@ mod map_metadata_tests {FileAssignment {rel: "report.dat".into(),path_id: "path:report.dat".into(),+ is_symlink: Some(false),family: "json".into(),gzip: false,content_digest: Some("digest".into()),@@ -4174,5 +4762,57 @@ mod map_metadata_tests {}}+#[cfg(test)]+mod generation_contract_identity_tests {+ use super::*;++ fn dataset(slug: &str, index: &str, family: &str) -> PlanDataset {+ PlanDataset {+ slug: slug.into(),+ index: index.into(),+ family: family.into(),+ group: None,+ specs: Vec::new(),+ time_field: None,+ semantic_field: None,+ sampled_records: 0,+ file_count: 0,+ }+ }++ #[test]+ fn contract_identities_are_order_independent_and_change_with_contracts() {+ let mut first = Plan {+ datasets: vec![+ dataset("b", "prefix-b", "csv"),+ dataset("a", "prefix-a", "json"),+ ],+ ..Plan::default()+ };+ let expected = generation_contract_identities(&first).unwrap();+ first.datasets.reverse();+ assert_eq!(generation_contract_identities(&first).unwrap(), expected);++ first.datasets[0].family = "text".into();+ let schema_changed = generation_contract_identities(&first).unwrap();+ assert_ne!(schema_changed.0, expected.0);++ first.datasets[0].family = "json".into();+ first.datasets[0].index = "different-a".into();+ let index_changed = generation_contract_identities(&first).unwrap();+ assert_eq!(index_changed.0, expected.0);+ assert_ne!(index_changed.1, expected.1);+ }++ #[test]+ fn internal_contract_versions_are_explicit() {+ assert_eq!(PREPARED_RECORDS_IDENTITY, "prepared-records-v1");+ assert_eq!(DOCUMENT_IDS_IDENTITY, "document-ids-v1");+ assert_eq!(DETECTOR_DISABLED_IDENTITY, "disabled");+ }+}+#[cfg(test)]mod failure_resume_http_tests;+#[cfg(test)]+mod incremental_reconcile_http_tests;diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rsnew file mode 100644index 000000000..a8026438b--- /dev/null+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs@@ -0,0 +1,587 @@+//! Deterministic projection of a changed inventory onto a frozen typed plan.+//!+//! Fresh-run clustering is intentionally not reused here: cluster formation+//! and slug election depend on corpus order and membership. Incremental runs+//! must preserve the committed dataset/schema identity and fail closed when a+//! file would require a new dataset or mapping.++use crate::content::Inventory;+use crate::infer::{FieldAcc, FieldSpec};+use crate::state::{FileAssignment, JunkFile, Plan, PlanDataset};+use crate::FileScan;+use anyhow::{Context, Result};+use std::cmp::Ordering;+use std::collections::{HashMap, HashSet};++/// Project the current, byte-verified inventory onto committed dataset schemas.+///+/// Assignment precedence is stable content identity, then stable native path+/// identity, then deterministic schema matching. Dataset definitions are+/// copied byte-for-byte except for their derived `file_count`.+pub(crate) fn reconcile_plan(+ inventory: &Inventory,+ previous: &Plan,+ scans: Vec<FileScan>,+) -> Result<Plan> {+ anyhow::ensure!(+ inventory.files.len() == inventory.keys.len()+ && inventory.files.len() == inventory.digests.len()+ && inventory.files.len() == scans.len(),+ "inventory and scan cardinalities disagree"+ );++ let datasets: HashMap<&str, &PlanDataset> = previous+ .datasets+ .iter()+ .map(|dataset| (dataset.slug.as_str(), dataset))+ .collect();+ anyhow::ensure!(+ datasets.len() == previous.datasets.len(),+ "frozen plan has duplicate dataset slugs"+ );++ let mut prior_path_owner: HashMap<&str, (&str, &FileAssignment)> = HashMap::new();+ for (content_id, assignment) in &previous.files {+ anyhow::ensure!(+ !assignment.path_id.is_empty(),+ "frozen assignment {} has no native path identity",+ assignment.rel+ );+ anyhow::ensure!(+ prior_path_owner+ .insert(&assignment.path_id, (content_id, assignment))+ .is_none(),+ "frozen plan assigns one native path identity more than once"+ );+ }+ for alias in &previous.duplicate_files {+ let assignment = previous+ .files+ .get(&alias.file_key)+ .with_context(|| format!("frozen alias {} has no canonical assignment", alias.rel))?;+ anyhow::ensure!(+ !alias.path_id.is_empty(),+ "frozen alias {} has no native path identity",+ alias.rel+ );+ anyhow::ensure!(+ prior_path_owner+ .insert(&alias.path_id, (&alias.file_key, assignment))+ .is_none(),+ "frozen plan assigns one native path identity more than once"+ );+ }++ let mut files = HashMap::new();+ let mut junk_files = Vec::new();+ for (((file, content_id), content_digest), scan) in inventory+ .files+ .iter()+ .zip(&inventory.keys)+ .zip(&inventory.digests)+ .zip(scans)+ {+ if let Some((status, reason)) = scan.junk {+ junk_files.push(JunkFile {+ file_key: content_id.clone(),+ rel: file.rel.clone(),+ format: crate::format_str(scan.sniffed.as_ref()),+ status,+ reason,+ bytes: file.size,+ });+ continue;+ }+ let sniffed = scan+ .sniffed+ .as_ref()+ .with_context(|| format!("{} has sketches but no detected family", file.rel))?;+ anyhow::ensure!(+ !scan.sketches.is_empty(),+ "{} has no dataset sketches and was not classified as junk",+ file.rel+ );++ let content_owner = previous.files.get(content_id);+ let path_owner = prior_path_owner.get(file.rel_id.as_str()).copied();+ if let (Some(content), Some((path_content_id, path))) = (content_owner, path_owner) {+ anyhow::ensure!(+ std::ptr::eq(content, path)+ || previous+ .files+ .get(path_content_id)+ .is_some_and(|candidate| std::ptr::eq(candidate, content)),+ "{} has conflicting committed content and path owners",+ file.rel+ );+ }+ let retained = content_owner.or_else(|| path_owner.map(|(_, assignment)| assignment));++ let mut assignments = Vec::with_capacity(scan.sketches.len());+ for (group, fields, _records) in &scan.sketches {+ let slug = if let Some(owner) = retained {+ retained_slug(owner, group, sniffed.family.as_str(), fields, &datasets)+ .with_context(|| format!("project retained file {}", file.rel))?+ } else {+ classify_new(group, sniffed.family.as_str(), fields, &previous.datasets)+ .with_context(|| format!("project new file {}", file.rel))?+ };+ assignments.push((group.clone(), slug));+ }+ assignments.sort();+ assignments.dedup();+ anyhow::ensure!(+ assignments.len() == scan.sketches.len(),+ "{} has duplicate dataset groups",+ file.rel+ );+ let replaced = files.insert(+ content_id.clone(),+ FileAssignment {+ rel: file.rel.clone(),+ path_id: file.rel_id.clone(),+ is_symlink: Some(file.is_symlink),+ family: sniffed.family.as_str().to_owned(),+ gzip: sniffed.gzip,+ content_digest: Some(content_digest.clone()),+ assignments,+ },+ );+ anyhow::ensure!(+ replaced.is_none(),+ "byte-verified inventory contains duplicate content identity {content_id}"+ );+ }++ let live_content: HashSet<&str> = files.keys().map(String::as_str).collect();+ let mut duplicate_files = inventory+ .duplicates+ .iter()+ .filter(|alias| live_content.contains(alias.file_key.as_str()))+ .cloned()+ .collect::<Vec<_>>();+ duplicate_files.sort_by(|left, right| {+ left.file_key+ .cmp(&right.file_key)+ .then_with(|| left.path_id.cmp(&right.path_id))+ .then_with(|| left.rel.cmp(&right.rel))+ });+ junk_files.sort_by(|left, right| {+ left.file_key+ .cmp(&right.file_key)+ .then_with(|| left.rel.cmp(&right.rel))+ });++ let mut frozen_datasets = previous.datasets.clone();+ for dataset in &mut frozen_datasets {+ dataset.file_count = files+ .values()+ .filter(|assignment| {+ assignment+ .assignments+ .iter()+ .any(|(_, slug)| slug == &dataset.slug)+ })+ .count();+ }++ Ok(Plan {+ datasets: frozen_datasets,+ files,+ junk_files,+ duplicate_files,+ alias_paths_indexed: previous.alias_paths_indexed,+ })+}++fn retained_slug(+ owner: &FileAssignment,+ group: &Option<String>,+ family: &str,+ fields: &HashMap<String, FieldAcc>,+ datasets: &HashMap<&str, &PlanDataset>,+) -> Result<String> {+ anyhow::ensure!(+ owner.family == family,+ "detected family changed from {} to {family}",+ owner.family+ );+ let matches = owner+ .assignments+ .iter()+ .filter(|(assigned_group, _)| assigned_group == group)+ .collect::<Vec<_>>();+ anyhow::ensure!(+ matches.len() == 1,+ "frozen assignment has {} matches for group {:?}",+ matches.len(),+ group+ );+ let slug = &matches[0].1;+ let dataset = datasets+ .get(slug.as_str())+ .with_context(|| format!("frozen assignment references absent dataset {slug}"))?;+ ensure_compatible(fields, dataset)?;+ Ok(slug.clone())+}++fn classify_new(+ group: &Option<String>,+ family: &str,+ fields: &HashMap<String, FieldAcc>,+ datasets: &[PlanDataset],+) -> Result<String> {+ let mut candidates = Vec::new();+ for dataset in datasets+ .iter()+ .filter(|dataset| dataset.family == family && &dataset.group == group)+ {+ if ensure_compatible(fields, dataset).is_err() {+ continue;+ }+ let (intersection, union) = field_overlap(fields, &dataset.specs);+ let threshold_met = if group.is_some() {+ intersection * 2 >= union // 0.5+ } else {+ intersection * 10 >= union * 7 // 0.7+ };+ if threshold_met {+ candidates.push((dataset, intersection, union));+ }+ }+ candidates.sort_by(|(left, li, lu), (right, ri, ru)| {+ ratio_cmp(*ri, *ru, *li, *lu).then_with(|| left.slug.cmp(&right.slug))+ });+ candidates+ .first()+ .map(|(dataset, _, _)| dataset.slug.clone())+ .with_context(|| {+ format!(+ "no frozen dataset accepts family {family}, group {:?}, and fields {:?}; \+ this requires unsupported dataset/schema evolution (use a new prefix)",+ group,+ sorted_field_names(fields)+ )+ })+}++fn field_overlap(fields: &HashMap<String, FieldAcc>, specs: &[FieldSpec]) -> (usize, usize) {+ // Keep null-only and object-shaped fields in the schema fingerprint too.+ // FieldAcc deliberately does not count those as scalar observations, but+ // allowing an unknown one through would let the server create a dynamic+ // mapping outside the frozen plan.+ let observed: HashSet<&str> = fields.keys().map(String::as_str).collect();+ let frozen: HashSet<&str> = specs.iter().map(|spec| spec.name.as_str()).collect();+ (+ observed.intersection(&frozen).count(),+ observed.union(&frozen).count().max(1),+ )+}++fn ratio_cmp(left_n: usize, left_d: usize, right_n: usize, right_d: usize) -> Ordering {+ (left_n * right_d).cmp(&(right_n * left_d))+}++fn ensure_compatible(fields: &HashMap<String, FieldAcc>, dataset: &PlanDataset) -> Result<()> {+ let specs: HashMap<&str, &FieldSpec> = dataset+ .specs+ .iter()+ .map(|spec| (spec.name.as_str(), spec))+ .collect();+ for (name, acc) in fields {+ let spec = specs.get(name.as_str()).with_context(|| {+ format!(+ "field {name} is absent from frozen dataset {}",+ dataset.slug+ )+ })?;+ anyhow::ensure!(+ compatible_with_spec(acc, spec),+ "field {name} is incompatible with frozen {} mapping in dataset {}",+ spec.es_type,+ dataset.slug+ );+ }+ Ok(())+}++fn compatible_with_spec(acc: &FieldAcc, spec: &FieldSpec) -> bool {+ if acc.n == 0 {+ return true;+ }+ let at_least_95 = |accepted: u64| accepted.saturating_mul(100) >= acc.n.saturating_mul(95);+ match spec.es_type.as_str() {+ "boolean" => at_least_95(acc.bool_ok),+ "long" => at_least_95(acc.long_ok),+ "double" => at_least_95(acc.double_ok),+ "date" => {+ let string_dates = acc.date_hits.values().copied().sum::<u64>();+ let numeric_dates = match spec.date_enc.as_deref() {+ Some("epoch_millis" | "epoch_seconds") => acc.long_ok,+ _ => 0,+ };+ at_least_95(string_dates.saturating_add(numeric_dates).min(acc.n))+ }+ "keyword" | "text" | "semantic_text" => true,+ _ => false,+ }+}++fn sorted_field_names(fields: &HashMap<String, FieldAcc>) -> Vec<&str> {+ let mut names = fields.keys().map(String::as_str).collect::<Vec<_>>();+ names.sort_unstable();+ names+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::sniff::{Family, Sniffed};+ use crate::state::{DuplicateFile, PlanDataset};+ use crate::walk::FileEntry;+ use serde_json::json;+ use std::path::PathBuf;++ fn acc(values: &[serde_json::Value]) -> FieldAcc {+ let mut acc = FieldAcc::default();+ for value in values {+ acc.add(value);+ }+ acc+ }++ fn spec(name: &str, es_type: &str) -> FieldSpec {+ FieldSpec {+ name: name.into(),+ es_type: es_type.into(),+ date_enc: None,+ semantic: None,+ cardinality_est: 0,+ cardinality_overflow: false,+ null_ratio: 0.0,+ avg_len: 0.0,+ coverage: 1.0,+ examples: vec![],+ notes: vec![],+ date_min: None,+ date_max: None,+ date_evidence: vec![],+ }+ }++ fn dataset(slug: &str, specs: Vec<FieldSpec>) -> PlanDataset {+ PlanDataset {+ slug: slug.into(),+ index: format!("ax-{slug}"),+ family: "jsonl".into(),+ group: None,+ specs,+ time_field: None,+ semantic_field: None,+ sampled_records: 1,+ file_count: 1,+ }+ }++ fn file(rel: &str, path_id: &str) -> FileEntry {+ FileEntry {+ path: PathBuf::from(rel),+ rel: rel.into(),+ rel_id: path_id.into(),+ is_symlink: false,+ size: 10,+ }+ }++ fn scan(fields: HashMap<String, FieldAcc>) -> FileScan {+ FileScan {+ sniffed: Some(Sniffed {+ family: Family::Jsonl,+ gzip: false,+ binary_kind: None,+ csv: None,+ encoding: "utf-8",+ }),+ sketches: vec![(None, fields, 1)],+ junk: None,+ }+ }++ fn inventory(rel: &str, path_id: &str, key: &str) -> Inventory {+ Inventory {+ files: vec![file(rel, path_id)],+ keys: vec![key.into()],+ digests: vec![format!("digest-{key}")],+ duplicates: vec![],+ }+ }++ fn assignment(rel: &str, path_id: &str, slug: &str) -> FileAssignment {+ FileAssignment {+ rel: rel.into(),+ path_id: path_id.into(),+ is_symlink: Some(false),+ family: "jsonl".into(),+ gzip: false,+ content_digest: Some("old-digest".into()),+ assignments: vec![(None, slug.into())],+ }+ }++ #[test]+ fn same_path_replacement_retains_dataset_and_never_re_elects_slug() {+ let mut previous = Plan {+ datasets: vec![+ dataset("z-owner", vec![spec("id", "long"), spec("body", "text")]),+ dataset("a-other", vec![spec("id", "long"), spec("body", "text")]),+ ],+ ..Plan::default()+ };+ previous+ .files+ .insert("old".into(), assignment("a.jsonl", "unix:61", "z-owner"));+ let fields = HashMap::from([+ ("id".into(), acc(&[json!(7)])),+ ("body".into(), acc(&[json!("hello world")])),+ ]);+ let plan = reconcile_plan(+ &inventory("a.jsonl", "unix:61", "new"),+ &previous,+ vec![scan(fields)],+ )+ .unwrap();+ assert_eq!(+ plan.files["new"].assignments,+ vec![(None, "z-owner".into())]+ );+ }++ #[test]+ fn new_file_uses_highest_jaccard_then_stable_slug_tie_break() {+ let previous = Plan {+ datasets: vec![+ dataset("z-tie", vec![spec("id", "long"), spec("body", "text")]),+ dataset("a-tie", vec![spec("id", "long"), spec("body", "text")]),+ dataset(+ "lower",+ vec![+ spec("id", "long"),+ spec("body", "text"),+ spec("extra", "keyword"),+ ],+ ),+ ],+ ..Plan::default()+ };+ let fields = HashMap::from([+ ("id".into(), acc(&[json!(7)])),+ ("body".into(), acc(&[json!("hello")])),+ ]);+ let plan = reconcile_plan(+ &inventory("new.jsonl", "unix:6e", "new"),+ &previous,+ vec![scan(fields)],+ )+ .unwrap();+ assert_eq!(plan.files["new"].assignments, vec![(None, "a-tie".into())]);+ assert_eq!(plan.datasets[0].file_count, 0);+ assert_eq!(plan.datasets[1].file_count, 1);+ }++ #[test]+ fn new_field_and_incompatible_type_fail_before_plan_changes() {+ let previous = Plan {+ datasets: vec![dataset("rows", vec![spec("id", "long")])],+ ..Plan::default()+ };+ let unknown = HashMap::from([+ ("id".into(), acc(&[json!(7)])),+ ("surprise".into(), acc(&[json!("new mapping")])),+ ]);+ let error = reconcile_plan(+ &inventory("new.jsonl", "unix:6e", "new"),+ &previous,+ vec![scan(unknown)],+ )+ .unwrap_err();+ assert!(format!("{error:#}").contains("unsupported dataset/schema evolution"));++ // Object-shaped values are not counted as scalar FieldAcc evidence,+ // but their field name must still be rejected rather than dynamically+ // mapped by the server.+ let object_field = HashMap::from([+ ("id".into(), acc(&[json!(7)])),+ ("object".into(), acc(&[json!({"nested": true})])),+ ]);+ let error = reconcile_plan(+ &inventory("new.jsonl", "unix:6e", "new"),+ &previous,+ vec![scan(object_field)],+ )+ .unwrap_err();+ assert!(format!("{error:#}").contains("unsupported dataset/schema evolution"));++ let wrong_type = HashMap::from([("id".into(), acc(&[json!("not-a-number")]))]);+ let error = reconcile_plan(+ &inventory("new.jsonl", "unix:6e", "new"),+ &previous,+ vec![scan(wrong_type)],+ )+ .unwrap_err();+ assert!(format!("{error:#}").contains("unsupported dataset/schema evolution"));+ }++ #[test]+ fn rename_by_content_retains_assignment_and_rebuilds_aliases_and_counts() {+ let mut previous = Plan {+ datasets: vec![dataset("rows", vec![spec("id", "long")])],+ ..Plan::default()+ };+ previous+ .files+ .insert("same".into(), assignment("old.jsonl", "unix:6f", "rows"));+ let mut current = inventory("renamed.jsonl", "unix:72", "same");+ current.duplicates.push(DuplicateFile {+ file_key: "same".into(),+ rel: "alias.jsonl".into(),+ path_id: "unix:61".into(),+ is_symlink: Some(false),+ duplicate_of: "renamed.jsonl".into(),+ bytes: 10,+ });+ let fields = HashMap::from([("id".into(), acc(&[json!(7)]))]);+ let plan = reconcile_plan(¤t, &previous, vec![scan(fields)]).unwrap();+ assert_eq!(plan.files["same"].rel, "renamed.jsonl");+ assert_eq!(plan.datasets[0].file_count, 1);+ assert_eq!(plan.duplicate_files.len(), 1);+ }++ #[test]+ fn duplicate_content_identity_and_conflicting_owners_fail_closed() {+ let mut previous = Plan {+ datasets: vec![dataset("rows", vec![spec("id", "long")])],+ ..Plan::default()+ };+ previous+ .files+ .insert("content-a".into(), assignment("a.jsonl", "unix:61", "rows"));+ previous+ .files+ .insert("content-b".into(), assignment("b.jsonl", "unix:62", "rows"));+ let fields = || HashMap::from([("id".into(), acc(&[json!(7)]))]);++ let conflict = inventory("a.jsonl", "unix:62", "content-a");+ let error = reconcile_plan(&conflict, &previous, vec![scan(fields())]).unwrap_err();+ assert!(format!("{error:#}").contains("conflicting committed content and path owners"));++ let duplicate = Inventory {+ files: vec![file("c.jsonl", "unix:63"), file("d.jsonl", "unix:64")],+ keys: vec!["same-key".into(), "same-key".into()],+ digests: vec!["digest".into(), "digest".into()],+ duplicates: vec![],+ };+ let error = reconcile_plan(&duplicate, &previous, vec![scan(fields()), scan(fields())])+ .unwrap_err();+ assert!(format!("{error:#}").contains("duplicate content identity"));+ }+}diff --git a/engine/crates/xerj-autoindex/src/sniff.rs b/engine/crates/xerj-autoindex/src/sniff.rsindex a0d8c5485..4802cd260 100644--- a/engine/crates/xerj-autoindex/src/sniff.rs+++ b/engine/crates/xerj-autoindex/src/sniff.rs@@ -85,15 +85,30 @@ fn read_prefix(path: &Path, gzip: bool, n: usize) -> Result<Vec<u8>> {}pub fn sniff(path: &Path) -> Result<Sniffed> {- let head = read_prefix(path, false, 8)?;+ sniff_with_name(path, path)+}++/// Classify bytes from `content_path` while retaining the logical filename+/// signals (currently source-code extensions) from `logical_path`.+///+/// Durable preparation uses this to classify an immutable snapshot blob+/// without losing the original name merely because the blob itself is named+/// by an ordinal.+pub fn sniff_with_name(content_path: &Path, logical_path: &Path) -> Result<Sniffed> {+ let head = read_prefix(content_path, false, 8)?;let gzip = head.len() >= 2 && head[0] == 0x1f && head[1] == 0x8b;- let prefix = read_prefix(path, gzip, 8192)?;- let mut s = sniff_bytes(&prefix, path, gzip)?;+ let prefix = read_prefix(content_path, gzip, 8192)?;+ let mut s = sniff_bytes(&prefix, content_path, logical_path, gzip)?;s.gzip = gzip;Ok(s)}-fn sniff_bytes(prefix: &[u8], path: &Path, gzip: bool) -> Result<Sniffed> {+fn sniff_bytes(+ prefix: &[u8],+ content_path: &Path,+ logical_path: &Path,+ gzip: bool,+) -> Result<Sniffed> {let mk = |family: Family| Sniffed {family,gzip: false,@@ -117,7 +132,7 @@ fn sniff_bytes(prefix: &[u8], path: &Path, gzip: bool) -> Result<Sniffed> {if prefix.starts_with(b"PK\x03\x04") {// zip container: DOCX iff it holds word/document.xmlif !gzip {- if let Ok(f) = std::fs::File::open(path) {+ if let Ok(f) = std::fs::File::open(content_path) {if let Ok(mut z) = zip::ZipArchive::new(f) {let is_docx = (0..z.len()).any(|i| {z.by_index_raw(i)@@ -172,7 +187,7 @@ fn sniff_bytes(prefix: &[u8], path: &Path, gzip: bool) -> Result<Sniffed> {// reach here after the binary guards above, so a text `.py`/`.rs`/`.go`/…// routes to the tree-sitter AST extractor (crate::extract::code). Extension// is the right signal — code vs prose is not reliably content-sniffable.- if let Some(ext) = path.extension().and_then(|e| e.to_str()) {+ if let Some(ext) = logical_path.extension().and_then(|e| e.to_str()) {if crate::extract::code::is_code_ext(ext) {let mut s = mk(Family::Code);s.encoding = encoding;diff --git a/engine/crates/xerj-autoindex/src/state.rs b/engine/crates/xerj-autoindex/src/state.rsindex ddb559fa2..16a13c522 100644--- a/engine/crates/xerj-autoindex/src/state.rs+++ b/engine/crates/xerj-autoindex/src/state.rs@@ -12,9 +12,10 @@ use std::io::{BufRead, Write};use std::path::{Path, PathBuf};#[cfg(test)]-static FILE_DONE_IO_FAILPOINT: std::sync::atomic::AtomicU8 = std::sync::atomic::AtomicU8::new(0);+static FILE_DONE_IO_FAILPOINT: std::sync::Mutex<Option<(u8, PathBuf)>> =+ std::sync::Mutex::new(None);#[cfg(test)]-static SYNC_IO_FAILPOINT: std::sync::atomic::AtomicU8 = std::sync::atomic::AtomicU8::new(0);+static SYNC_IO_FAILPOINT: std::sync::Mutex<Option<(u8, PathBuf)>> = std::sync::Mutex::new(None);#[cfg(test)]pub(crate) static FILE_DONE_IO_FAILPOINT_TEST_LOCK: std::sync::Mutex<()> =std::sync::Mutex::new(());@@ -22,13 +23,13 @@ pub(crate) static FILE_DONE_IO_FAILPOINT_TEST_LOCK: std::sync::Mutex<()> =pub(crate) static SYNC_IO_FAILPOINT_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());#[cfg(test)]-pub(crate) fn fail_next_file_done_io(boundary: u8) {- FILE_DONE_IO_FAILPOINT.store(boundary, std::sync::atomic::Ordering::SeqCst);+pub(crate) fn fail_next_file_done_io(boundary: u8, journal_path: &Path) {+ *FILE_DONE_IO_FAILPOINT.lock().unwrap() = Some((boundary, journal_path.to_path_buf()));}#[cfg(test)]-fn fail_next_sync_io(boundary: u8) {- SYNC_IO_FAILPOINT.store(boundary, std::sync::atomic::Ordering::SeqCst);+fn fail_next_sync_io(boundary: u8, journal_path: &Path) {+ *SYNC_IO_FAILPOINT.lock().unwrap() = Some((boundary, journal_path.to_path_buf()));}#[derive(Debug, Clone, Serialize, Deserialize)]@@ -186,6 +187,8 @@ mod tests {"{tail}");}+ }+#[test]fn preflight_holds_the_same_exclusive_lock_through_authoritative_open() {let dir = tempfile::tempdir().unwrap();@@ -420,7 +423,7 @@ fn apply_legacy_plan(}#[cfg(test)]-fn consume_io_failpoint(what: &str, boundary: u8) -> bool {+fn consume_io_failpoint(what: &str, boundary: u8, journal_path: &Path) -> bool {let failpoint = if what == "file_done" {&FILE_DONE_IO_FAILPOINT} else if what.starts_with("sync_") {@@ -428,14 +431,18 @@ fn consume_io_failpoint(what: &str, boundary: u8) -> bool {} else {return false;};- failpoint- .compare_exchange(- boundary,- 0,- std::sync::atomic::Ordering::SeqCst,- std::sync::atomic::Ordering::SeqCst,- )- .is_ok()+ let mut armed = failpoint.lock().unwrap();+ if armed+ .as_ref()+ .is_some_and(|(expected_boundary, expected_path)| {+ *expected_boundary == boundary && expected_path == journal_path+ })+ {+ *armed = None;+ true+ } else {+ false+ }}type PreflightReplay = (@@ -531,11 +538,14 @@ fn read_plan_for_preflight(}Some(kind) if crate::sync::is_sync_record_kind(kind) => {crate::sync::replay_record(&value, &mut committed_manifest, &mut pending_sync)?;- if kind == "sync_commit" {+ if matches!(kind, "sync_bootstrap" | "sync_commit") {plan = committed_manifest.as_ref().map(|manifest| manifest.plan.clone());}+ if kind == "sync_bootstrap" {+ legacy_migration_reasons.clear();+ }}Some(kind) if kind.starts_with("sync_") => {anyhow::bail!("unknown durable sync record kind {kind}");@@ -776,11 +786,14 @@ impl Journal {&mut committed_manifest,&mut pending_sync,)?;- if kind == "sync_commit" {+ if matches!(kind, "sync_bootstrap" | "sync_commit") {plan = committed_manifest.as_ref().map(|manifest| manifest.plan.clone());}+ if kind == "sync_bootstrap" {+ legacy_migration_reasons.clear();+ }}Some(kind) if kind.starts_with("sync_") => {anyhow::bail!(@@ -954,11 +967,11 @@ impl Journal {line.push('\n');let start = self.file.metadata()?.len();#[cfg(test)]- if consume_io_failpoint(what, 1) {+ if consume_io_failpoint(what, 1, &self.path) {anyhow::bail!("injected {what} append failure before any bytes");}#[cfg(test)]- let partial_write = consume_io_failpoint(what, 3);+ let partial_write = consume_io_failpoint(what, 3, &self.path);#[cfg(not(test))]let partial_write = false;let write_result = if partial_write {@@ -977,7 +990,7 @@ impl Journal {return self.rollback_transaction(start, what, "append", error);}#[cfg(test)]- let injected_sync_failure = consume_io_failpoint(what, 2);+ let injected_sync_failure = consume_io_failpoint(what, 2, &self.path);#[cfg(not(test))]let injected_sync_failure = false;let sync_result = if injected_sync_failure {@@ -1057,6 +1070,32 @@ impl Journal {Ok(())}+ pub fn sync_bootstrap_genesis(&mut self) -> Result<()> {+ anyhow::ensure!(+ self.pending_sync.is_none()+ && self.committed_manifest.is_none()+ && self.plan.is_none()+ && self.done.is_empty()+ && self.pending_replacements.is_empty(),+ "generation-zero bootstrap is only valid for an empty new journal"+ );+ let manifest = crate::sync::CommittedManifest::genesis()?;+ let mut committed = None;+ crate::sync::replay_record(+ &serde_json::json!({"kind": "sync_bootstrap", "manifest": manifest}),+ &mut committed,+ &mut None,+ )?;+ self.append_transaction(+ &serde_json::json!({"kind": "sync_bootstrap", "manifest": manifest}),+ "sync_bootstrap",+ )?;+ self.plan = Some(Plan::default());+ self.committed_manifest = committed;+ self.legacy_migration_reasons.clear();+ Ok(())+ }+pub fn sync_operation_state(&mut self,operation_id: &str,@@ -1221,14 +1260,28 @@ mod sync_journal_tests {sampled_records: 1,file_count: 0,});- journal.write_plan(&plan).unwrap();+ journal.sync_bootstrap_genesis().unwrap();journal}fn pending(journal: &Journal) -> PendingSync {let base = journal.committed_manifest.as_ref().unwrap();let mut plan = base.plan.clone();- plan.datasets[0].file_count = 1;+ if plan.datasets.is_empty() {+ plan.datasets.push(PlanDataset {+ slug: "reports".into(),+ index: "prefix-reports".into(),+ family: "pdf".into(),+ group: None,+ specs: Vec::new(),+ time_field: None,+ semantic_field: None,+ sampled_records: 1,+ file_count: 1,+ });+ } else {+ plan.datasets[0].file_count = 1;+ }plan.files.insert("content-a".into(),FileAssignment {@@ -1253,13 +1306,14 @@ mod sync_journal_tests {prefix: "prefix".into(),follow_symlinks: false,chunker_identity: "chunker-v1".into(),+ embedding_identity_sha256: "a".repeat(64),embedding_backend: "lexical".into(),- embedding_model: "feature-hash".into(),- embedding_tokenizer: "builtin".into(),embedding_dimension: 384,+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),+ embedding_resumable: true,graph_enabled: false,- brain: "none".into(),- detector_identity: "detectors-v1".into(),+ brain: "disabled".into(),+ detector_identity: "disabled".into(),schema_identity: "schema-v1".into(),index_identity: "index-v1".into(),source_policy: SourceExecutionPolicy::AbortOnSourceChange {@@ -1311,6 +1365,29 @@ mod sync_journal_tests {assert_eq!(reopened.pending_sync.as_ref().unwrap().tx_id, "sync-1");}+ #[test]+ fn new_journal_can_bootstrap_empty_generation_authority_once() {+ let dir = tempfile::tempdir().unwrap();+ let mut journal = Journal::open(dir.path(), "root", "url", "prefix", 300, false).unwrap();+ journal.sync_bootstrap_genesis().unwrap();+ let committed = journal.committed_manifest.as_ref().unwrap();+ assert_eq!(committed.generation, 0);+ assert!(committed.groups.is_empty());+ assert_eq!(+ serde_json::to_value(journal.plan.as_ref().unwrap()).unwrap(),+ serde_json::to_value(Plan::default()).unwrap()+ );+ assert!(journal.sync_bootstrap_genesis().is_err());+ drop(journal);++ let reopened = Journal::open(dir.path(), "root", "url", "prefix", 300, false).unwrap();+ assert_eq!(reopened.committed_manifest.unwrap().generation, 0);+ assert_eq!(+ serde_json::to_value(reopened.plan.unwrap()).unwrap(),+ serde_json::to_value(Plan::default()).unwrap()+ );+ }+#[test]fn replay_rejects_sync_begin_followed_by_legacy_plan_without_changing_authority() {let _guard = SYNC_IO_FAILPOINT_TEST_LOCK.lock().unwrap();@@ -1535,7 +1612,7 @@ mod sync_journal_tests {let mut journal = journal_with_plan(dir.path());let desired = pending(&journal);let before = std::fs::read(dir.path().join("journal.ndjson")).unwrap();- fail_next_sync_io(boundary);+ fail_next_sync_io(boundary, journal.path());assert!(journal.sync_begin(&desired).is_err());assert!(journal.pending_sync.is_none());drop(journal);@@ -1569,7 +1646,7 @@ mod sync_journal_tests {journal.sync_validated().unwrap();}let before = std::fs::read(dir.path().join("journal.ndjson")).unwrap();- fail_next_sync_io(boundary);+ fail_next_sync_io(boundary, journal.path());let result = match stage {"operation" => {journal.sync_operation_state("upsert:group-a", SyncOperationState::Started)@@ -1809,7 +1886,7 @@ mod compatibility_tests {let mut journal =Journal::open(dir.path(), "root", "http://engine", "ax", 300, false).unwrap();journal.file_replace_start("file-key", "new").unwrap();- fail_next_file_done_io(boundary);+ fail_next_file_done_io(boundary, journal.path());let error = journal.file_done(&FileDone {file_key: "file-key".into(),@@ -1847,7 +1924,7 @@ mod compatibility_tests {journal.file_replace_start("file-c", "generation-c").unwrap();- fail_next_file_done_io(3);+ fail_next_file_done_io(3, journal.path());let error = journal.file_done(&FileDone {file_key: "file-c".into(),diff --git a/engine/crates/xerj-autoindex/src/sync.rs b/engine/crates/xerj-autoindex/src/sync.rsindex c60f4c151..9a20f4352 100644--- a/engine/crates/xerj-autoindex/src/sync.rs+++ b/engine/crates/xerj-autoindex/src/sync.rs@@ -10,7 +10,7 @@ use serde::{Deserialize, Serialize};use serde_json::Value;use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};-pub const EXECUTION_IDENTITY_VERSION: u32 = 1;+pub const EXECUTION_IDENTITY_VERSION: u32 = 2;#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]pub struct ManifestPath {@@ -66,10 +66,13 @@ pub struct ExecutionIdentity {pub prefix: String,pub follow_symlinks: bool,pub chunker_identity: String,+ /// Opaque, server-authenticated identity of the exact embedding execution+ /// space. Autoindex must never synthesize model/tokenizer labels.+ pub embedding_identity_sha256: String,pub embedding_backend: String,- pub embedding_model: String,- pub embedding_tokenizer: String,pub embedding_dimension: usize,+ pub embedding_semantic_contract: String,+ pub embedding_resumable: bool,pub graph_enabled: bool,pub brain: String,pub detector_identity: String,@@ -105,6 +108,27 @@ pub enum LegacyBootstrap {}impl CommittedManifest {+ pub fn genesis() -> Result<Self> {+ match Self::bootstrap_legacy(Plan::default())? {+ LegacyBootstrap::Ready(manifest) => Ok(*manifest),+ LegacyBootstrap::MigrationRequired { .. } => {+ anyhow::bail!("empty generation-zero manifest unexpectedly requires migration")+ }+ }+ }++ fn validate_genesis(&self) -> Result<()> {+ anyhow::ensure!(+ self.generation == 0+ && self.manifest_digest == Self::genesis()?.manifest_digest+ && serde_json::to_value(&self.plan)? == serde_json::to_value(Plan::default())?+ && self.groups.is_empty()+ && self.execution.is_none(),+ "generation-zero bootstrap manifest is not exact genesis"+ );+ Ok(())+ }+pub fn bootstrap_legacy(plan: Plan) -> Result<LegacyBootstrap> {let mut reasons = Vec::new();for (file_key, file) in &plan.files {@@ -355,6 +379,13 @@ fn validate_supported_manifest_delta(// the only configuration transition this vocabulary can represent. A// durable snapshot reference and digest are generation inputs, not engine// configuration, so they must advance with each desired generation.+ if base.generation == 0 && base.execution.is_none() && base.groups.is_empty() {+ anyhow::ensure!(+ base.manifest_digest == CommittedManifest::genesis()?.manifest_digest,+ "generation-one migration base is not exact genesis"+ );+ return Ok(());+ }if base.generation > 0 || base.execution.is_some() {let same_execution_configuration = |left: &ExecutionIdentity, right: &ExecutionIdentity| {let mut left = left.clone();@@ -385,11 +416,10 @@ fn validate_supported_manifest_delta(base.plan.alias_paths_indexed == desired.plan.alias_paths_indexed,"plan flags changed without a supported operation");- anyhow::ensure!(- canonical_json_bytes(&base.plan.junk_files)?- == canonical_json_bytes(&desired.plan.junk_files)?,- "junk plan changed without a supported operation"- );+ // Junk is a local inference result, not remotely published generation+ // membership. Adding/removing an unsupported file may advance the+ // manifest for truthful map/status output, but it intentionally derives+ // no remote data or catalog operation.let normalize_datasets = |datasets: &[crate::state::PlanDataset]| {let mut datasets = datasets.to_vec();for dataset in &mut datasets {@@ -632,16 +662,25 @@ fn validate_manifest(manifest: &GenerationManifest, legacy: bool) -> Result<()>execution.version == EXECUTION_IDENTITY_VERSION,"unsupported execution identity version");+ anyhow::ensure!(+ !execution.graph_enabled+ && execution.brain == "disabled"+ && execution.detector_identity == "disabled",+ "incremental generation manifests require graph-disabled execution identity"+ );for (name, value) in [("root_identity", execution.root_identity.as_str()),("url", execution.url.as_str()),("prefix", execution.prefix.as_str()),("chunker_identity", execution.chunker_identity.as_str()),+ (+ "embedding_identity_sha256",+ execution.embedding_identity_sha256.as_str(),+ ),("embedding_backend", execution.embedding_backend.as_str()),- ("embedding_model", execution.embedding_model.as_str()),(- "embedding_tokenizer",- execution.embedding_tokenizer.as_str(),+ "embedding_semantic_contract",+ execution.embedding_semantic_contract.as_str(),),("detector_identity", execution.detector_identity.as_str()),("schema_identity", execution.schema_identity.as_str()),@@ -649,6 +688,22 @@ fn validate_manifest(manifest: &GenerationManifest, legacy: bool) -> Result<()>] {anyhow::ensure!(!value.is_empty(), "execution identity {name} is empty");}+ anyhow::ensure!(+ execution.embedding_identity_sha256.len() == 64+ && execution+ .embedding_identity_sha256+ .bytes()+ .all(|byte| byte.is_ascii_hexdigit() && !byte.is_ascii_uppercase()),+ "embedding execution identity must be a lowercase SHA-256 digest"+ );+ anyhow::ensure!(+ execution.embedding_dimension > 0,+ "embedding execution dimension must be positive"+ );+ anyhow::ensure!(+ execution.embedding_resumable,+ "a durable generation requires a resumable embedding execution identity"+ );match &execution.source_policy {SourceExecutionPolicy::DurableSnapshot {reference,@@ -901,7 +956,12 @@ pub fn apply_begin(pub fn is_sync_record_kind(kind: &str) -> bool {matches!(kind,- "sync_begin" | "sync_operation_state" | "sync_validated" | "sync_commit" | "sync_abort"+ "sync_bootstrap"+ | "sync_begin"+ | "sync_operation_state"+ | "sync_validated"+ | "sync_commit"+ | "sync_abort")}@@ -912,6 +972,26 @@ pub fn replay_record() -> Result<()> {let kind = record_str(record, "kind")?;match kind {+ "sync_bootstrap" => {+ anyhow::ensure!(+ pending.is_none(),+ "sync_bootstrap cannot replace pending sync"+ );+ anyhow::ensure!(+ committed.is_none(),+ "sync_bootstrap cannot replace committed authority"+ );+ let manifest: CommittedManifest = serde_json::from_value(+ record+ .get("manifest")+ .cloned()+ .context("sync_bootstrap has no manifest")?,+ )+ .context("decode durable sync_bootstrap")?;+ manifest.validate_genesis()?;+ *committed = Some(manifest);+ Ok(())+ }"sync_begin" => {let begin: PendingSync =serde_json::from_value(record.clone()).context("decode durable sync_begin")?;@@ -1015,6 +1095,34 @@ mod tests {}}+ #[test]+ fn bootstrap_replay_accepts_only_exact_empty_genesis() {+ let genesis = CommittedManifest::genesis().unwrap();+ let record = serde_json::json!({+ "kind": "sync_bootstrap",+ "manifest": genesis,+ });+ let mut committed = None;+ let mut pending = None;+ replay_record(&record, &mut committed, &mut pending).unwrap();+ assert_eq!(committed.unwrap().generation, 0);++ let mut non_genesis = CommittedManifest::genesis().unwrap();+ non_genesis.generation = 1;+ let record = serde_json::json!({+ "kind": "sync_bootstrap",+ "manifest": non_genesis,+ });+ assert!(replay_record(&record, &mut None, &mut None).is_err());++ let mut committed = Some(CommittedManifest::genesis().unwrap());+ let record = serde_json::json!({+ "kind": "sync_bootstrap",+ "manifest": CommittedManifest::genesis().unwrap(),+ });+ assert!(replay_record(&record, &mut committed, &mut None).is_err());+ }+#[test]fn canonical_prefers_real_then_native_bytes_including_non_utf8_identity() {let groups = reconcile_groups(@@ -1154,13 +1262,14 @@ mod tests {prefix: "prefix".into(),follow_symlinks: false,chunker_identity: "chunker-v1".into(),+ embedding_identity_sha256: "a".repeat(64),embedding_backend: "lexical".into(),- embedding_model: "feature-hash".into(),- embedding_tokenizer: "builtin".into(),embedding_dimension: 384,+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),+ embedding_resumable: true,graph_enabled: false,- brain: "none".into(),- detector_identity: "detectors-v1".into(),+ brain: "disabled".into(),+ detector_identity: "disabled".into(),schema_identity: "schema-v1".into(),index_identity: "index-v1".into(),source_policy: SourceExecutionPolicy::AbortOnSourceChange {@@ -1183,11 +1292,7 @@ mod tests {sampled_records: 0,file_count: 0,});- let LegacyBootstrap::Ready(legacy) =- CommittedManifest::bootstrap_legacy(legacy_plan.clone()).unwrap()- else {- panic!("empty legacy membership is migratable");- };+ let legacy = CommittedManifest::genesis().unwrap();let first = GenerationManifest {generation: 1,execution: Some(identity("inventory-a")),@@ -1204,10 +1309,21 @@ mod tests {assert!(PendingSync::new("changed-execution".into(), &committed, changed_execution).is_err());+ let mut changed_generated_flag = first.clone();+ changed_generated_flag.generation = 2;+ changed_generated_flag.plan.alias_paths_indexed =+ !changed_generated_flag.plan.alias_paths_indexed;+ assert!(PendingSync::new(+ "changed-generated-flag".into(),+ &committed,+ changed_generated_flag+ )+ .is_err());let mut changed_plan = first;+ changed_plan.generation = 2;changed_plan.plan.alias_paths_indexed = true;- assert!(PendingSync::new("changed-plan".into(), &legacy, changed_plan).is_err());+ assert!(PendingSync::new("changed-plan".into(), &committed, changed_plan).is_err());}#[test]diff --git a/engine/crates/xerj-autoindex/src/sync_executor.rs b/engine/crates/xerj-autoindex/src/sync_executor.rsindex 06423deb4..234d00de3 100644--- a/engine/crates/xerj-autoindex/src/sync_executor.rs+++ b/engine/crates/xerj-autoindex/src/sync_executor.rs@@ -13,20 +13,28 @@ use crate::sync::{use anyhow::{Context, Result};use serde::{Deserialize, Serialize};use serde_json::{Map, Value};-use std::collections::HashMap;+use std::collections::{BTreeMap, HashMap};use std::fs::{File, OpenOptions};use std::io::{BufReader, BufWriter, Read, Write};use std::path::Path;-const SNAPSHOT_VERSION: u32 = 1;+const SNAPSHOT_VERSION: u32 = 2;#[cfg(test)]static SNAPSHOT_FAILPOINT: std::sync::atomic::AtomicU8 = std::sync::atomic::AtomicU8::new(0);#[cfg(test)]static SNAPSHOT_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());#[cfg(test)]+static POST_SEAL_SOURCE_REPLACEMENT: std::sync::Mutex<Option<(std::path::PathBuf, Vec<u8>)>> =+ std::sync::Mutex::new(None);+#[cfg(test)]static REPLAY_FAIL_AFTER_APPLY: std::sync::atomic::AtomicBool =std::sync::atomic::AtomicBool::new(false);+#[cfg(test)]+pub(crate) static REPLAY_FAILPOINT_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());+#[cfg(test)]+static GC_FAIL_AFTER_RENAME: std::sync::atomic::AtomicBool =+ std::sync::atomic::AtomicBool::new(false);#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]pub struct SnapshotFile {@@ -52,15 +60,99 @@ pub struct PreparedArtifact {pub struct SourceSnapshot {pub version: u32,pub tx_id: String,+ /// Publication timestamp sealed once and reused by every replay.+ pub started: String,+ pub preparation_contract_digest: String,+ pub footprint: SnapshotFootprint,pub files: Vec<SnapshotFile>,pub snapshot_digest: String,}+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]+pub struct SnapshotFootprint {+ pub source_bytes: u64,+ pub prepared_bytes: u64,+ pub total_bytes: u64,+ pub hard_budget_bytes: u64,+}++/// Logical payload accounting for source and prepared-record bytes.+/// This is deliberately not a filesystem-allocation limit: directory,+/// manifest, and block-allocation overhead are excluded.+struct PayloadBudget {+ used: u64,+ limit: u64,+}++impl PayloadBudget {+ fn charge(&mut self, bytes: usize) -> std::io::Result<()> {+ let bytes = u64::try_from(bytes)+ .map_err(|_| std::io::Error::other("snapshot payload write length overflow"))?;+ let next = self+ .used+ .checked_add(bytes)+ .ok_or_else(|| std::io::Error::other("snapshot payload byte count overflow"))?;+ if next > self.limit {+ return Err(std::io::Error::other(format!(+ "snapshot logical payload write of {bytes} bytes would raise staged payload from \+ {} to {next} bytes, exceeding configured limit of {} bytes; no generation was \+ committed and the partial snapshot is removed automatically",+ self.used, self.limit+ )));+ }+ self.used = next;+ Ok(())+ }+}++struct BudgetWriter<'a, W> {+ inner: W,+ budget: &'a mut PayloadBudget,+}++impl<W: Write> Write for BudgetWriter<'_, W> {+ fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {+ self.budget.charge(buf.len())?;+ match self.inner.write(buf) {+ Ok(written) if written == buf.len() => Ok(written),+ Ok(written) => {+ self.budget.used -= (buf.len() - written) as u64;+ Ok(written)+ }+ Err(error) => {+ self.budget.used -= buf.len() as u64;+ Err(error)+ }+ }+ }++ fn flush(&mut self) -> std::io::Result<()> {+ self.inner.flush()+ }+}++struct StagingCleanup {+ path: std::path::PathBuf,+ armed: bool,+}++impl Drop for StagingCleanup {+ fn drop(&mut self) {+ if self.armed {+ let _ = std::fs::remove_dir_all(&self.path);+ }+ }+}+/// Remote mutations must be convergent: calling `apply` twice for one/// operation after an accepted-but-unrecorded response must produce exactly/// the same live state. `validate` is the final generation-wide barrier.#[cfg_attr(not(test), allow(dead_code))]pub trait SyncOperationBackend {+ fn provision_generation(&mut self, _desired: &GenerationManifest) -> Result<()> {+ Ok(())+ }+fn apply(&mut self,operation: &SyncOperation,@@ -75,6 +167,17 @@ pub trait SyncOperationBackend {desired: &GenerationManifest,snapshot: &SourceSnapshot,) -> Result<()>;++ /// Publish and exactly read back the complete agent-facing catalog for+ /// this generation. This runs before the validation/authority barrier.+ fn publish_generation_catalog(+ &mut self,+ _base: &CommittedManifest,+ _desired: &GenerationManifest,+ _snapshot: &SourceSnapshot,+ ) -> Result<()> {+ Ok(())+ }}/// Production ES-compatible operation backend for graph-disabled generations.@@ -212,64 +315,162 @@ impl<'a> EsSyncBackend<'a> {Ok(total)}- fn reconcile_file_catalog(+ fn catalog_generation(&self, run_id: &str) -> Result<BTreeMap<String, Value>> {+ let mut documents = BTreeMap::new();+ let mut search_after: Option<Value> = None;+ loop {+ let mut body = serde_json::json!({+ "size": 1000,+ "sort": [{"_id": "asc"}],+ "query": {"term": {"run_id": run_id}}+ });+ if let Some(after) = &search_after {+ body["search_after"] = after.clone();+ }+ let response = self.es.search(crate::catalog::CATALOG_INDEX, &body)?;+ let hits = response+ .pointer("/hits/hits")+ .and_then(Value::as_array)+ .context("catalog generation query has no hits")?;+ for hit in hits {+ let id = hit+ .get("_id")+ .and_then(Value::as_str)+ .context("catalog hit has no _id")?+ .to_owned();+ let source = hit+ .get("_source")+ .cloned()+ .context("catalog hit has no _source")?;+ anyhow::ensure!(+ documents.insert(id.clone(), source).is_none(),+ "catalog generation query returned duplicate ID {id}"+ );+ }+ if hits.len() < 1000 {+ break;+ }+ search_after = hits.last().and_then(|hit| hit.get("sort")).cloned();+ anyhow::ensure!(+ search_after.is_some(),+ "full catalog generation page has no continuation sort key"+ );+ }+ Ok(documents)+ }++ fn exact_dataset_catalog_stats(&self,- old: Option<&ManifestGroup>,- new: Option<&ManifestGroup>,desired: &GenerationManifest,- ) -> Result<()> {- for content_id in old- .into_iter()- .map(|group| group.content_id.as_str())- .chain(new.into_iter().map(|group| group.content_id.as_str()))- {- self.es.delete_by_query(- crate::catalog::CATALOG_INDEX,- &serde_json::json!({"term": {"file_key": content_id}}),- )?;- }- let Some(group) = new else {- return Ok(());- };- let assignment = desired- .plan- .files- .get(&group.content_id)- .context("catalog group has no desired file assignment")?;- let format = if assignment.gzip {- format!("{}(gzip)", assignment.family)- } else {- assignment.family.clone()- };- let (id, doc) = crate::catalog::file_doc(- &group.content_id,- &group.canonical.rel,- &format,- "indexed",- None,- group.expected_records,- 0,- group.content_size,- &format!("generation-{}", desired.generation),- );- let mut body = Vec::new();- append_index_action(crate::catalog::CATALOG_INDEX, &id, &doc, &mut body)?;- for alias in &group.aliases {- let (id, doc) = crate::catalog::duplicate_file_doc(- &group.content_id,- &alias.rel,- &alias.path_id,- &group.canonical.rel,- group.content_size,- &format!("generation-{}", desired.generation),+ ) -> Result<BTreeMap<String, crate::generation_catalog::DatasetCatalogStats>> {+ let mut out = BTreeMap::new();+ for dataset in &desired.plan.datasets {+ let groups: Vec<&ManifestGroup> = desired+ .groups+ .iter()+ .filter(|group| group.dataset_slugs.contains(&dataset.slug))+ .collect();+ let content_ids: Vec<&str> = groups+ .iter()+ .map(|group| group.content_id.as_str())+ .collect();+ let mut body = serde_json::json!({+ "size": 0,+ "track_total_hits": true,+ "query": {"terms": {"ax_file": content_ids}}+ });+ if let Some(field) = &dataset.time_field {+ body["aggs"] = serde_json::json!({+ "time_min": {"min": {"field": field}},+ "time_max": {"max": {"field": field}}+ });+ }+ let response = self.es.search(&dataset.index, &body)?;+ let record_count = response+ .pointer("/hits/total/value")+ .and_then(Value::as_u64)+ .context("dataset exact read-back has no total")?;+ let expected = groups.iter().try_fold(0u64, |sum, group| {+ sum.checked_add(group.expected_records)+ .context("dataset expected record count overflow")+ })?;+ anyhow::ensure!(+ record_count == expected,+ "dataset {} exact read-back count {record_count} disagrees with sealed count \+ {expected}",+ dataset.slug+ );+ let mut formats: Vec<String> = desired+ .plan+ .files+ .values()+ .filter(|assignment| {+ assignment+ .assignments+ .iter()+ .any(|(_, slug)| slug == &dataset.slug)+ })+ .map(|assignment| {+ if assignment.gzip {+ format!("{}(gzip)", assignment.family)+ } else {+ assignment.family.clone()+ }+ })+ .collect();+ formats.sort();+ formats.dedup();+ let bytes = groups.iter().try_fold(0u64, |sum, group| {+ sum.checked_add(group.content_size)+ .context("dataset source byte count overflow")+ })?;+ let time = |name: &str| {+ response+ .pointer(&format!("/aggregations/{name}/value_as_string"))+ .and_then(Value::as_str)+ .map(ToOwned::to_owned)+ };+ out.insert(+ dataset.slug.clone(),+ crate::generation_catalog::DatasetCatalogStats {+ record_count,+ junk_records: 0,+ bytes,+ formats,+ time_min: time("time_min"),+ time_max: time("time_max"),+ sample_queries: crate::catalog::build_sample_queries(dataset, &[]),+ notes: dataset+ .group+ .iter()+ .map(|group| format!("source table: {group}"))+ .chain(dataset.specs.iter().flat_map(|spec| {+ spec.notes+ .iter()+ .map(move |note| format!("{}: {note}", spec.name))+ }))+ .collect(),+ },);- append_index_action(crate::catalog::CATALOG_INDEX, &id, &doc, &mut body)?;}- checked_bulk(self.es, body)+ Ok(out)}}impl SyncOperationBackend for EsSyncBackend<'_> {+ fn provision_generation(&mut self, desired: &GenerationManifest) -> Result<()> {+ let execution = desired+ .execution+ .as_ref()+ .context("desired generation has no execution identity")?;+ let (_, index_identity) = crate::generation_contract_identities(&desired.plan)?;+ anyhow::ensure!(+ execution.index_identity == index_identity,+ "desired generation index identity disagrees with its frozen mappings"+ );+ crate::ensure_generation_mappings(self.es, &desired.plan)+ }+fn apply(&mut self,operation: &SyncOperation,@@ -313,7 +514,73 @@ impl SyncOperationBackend for EsSyncBackend<'_> {)?;}}- self.reconcile_file_catalog(old, new, desired)+ Ok(())+ }++ fn publish_generation_catalog(+ &mut self,+ base: &CommittedManifest,+ desired: &GenerationManifest,+ snapshot: &SourceSnapshot,+ ) -> Result<()> {+ for index in desired+ .plan+ .datasets+ .iter()+ .map(|dataset| dataset.index.as_str())+ .collect::<std::collections::BTreeSet<_>>()+ {+ self.es.refresh(index)?;+ }+ self.es.refresh(crate::catalog::CATALOG_INDEX)?;+ let prior_run_id =+ base.execution+ .as_ref()+ .and_then(|execution| match &execution.source_policy {+ SourceExecutionPolicy::DurableSnapshot { reference, .. } => {+ reference.strip_prefix("sync-snapshots/")+ }+ SourceExecutionPolicy::AbortOnSourceChange { .. } => None,+ });+ let prior_documents = prior_run_id+ .map(|run_id| self.catalog_generation(run_id))+ .transpose()?+ .unwrap_or_default();+ let prior_ids = prior_documents.keys().cloned().collect();+ let stats = self.exact_dataset_catalog_stats(desired)?;+ let projection = crate::generation_catalog::project_generation(+ base,+ desired,+ &crate::generation_catalog::GenerationCatalogMetadata {+ generation_id: snapshot.tx_id.clone(),+ started: snapshot.started.clone(),+ },+ &stats,+ &BTreeMap::new(),+ &prior_ids,+ )?;+ let mut body = Vec::new();+ for id in &projection.stale_ids {+ let action =+ serde_json::json!({"delete": {"_index": crate::catalog::CATALOG_INDEX, "_id": id}});+ body.extend_from_slice(serde_json::to_string(&action)?.as_bytes());+ body.push(b'\n');+ }+ for (id, document) in &projection.documents {+ append_index_action(crate::catalog::CATALOG_INDEX, id, document, &mut body)?;+ }+ checked_bulk(self.es, body)?;+ self.es.refresh(crate::catalog::CATALOG_INDEX)?;+ projection.validate_observed(&self.catalog_generation(&snapshot.tx_id)?)?;+ if let Some(prior_run_id) = prior_run_id {+ let remaining = self.catalog_generation(prior_run_id)?;+ anyhow::ensure!(+ remaining.is_empty(),+ "prior catalog generation {prior_run_id} still has {} managed documents",+ remaining.len()+ );+ }+ Ok(())}fn validate(@@ -394,6 +661,7 @@ pub fn replay_pending_operations(pending.validate_against(&base)?;let snapshot = open_snapshot(state_dir, &pending.tx_id)?;verify_snapshot_binding(&pending, &snapshot)?;+ backend.provision_generation(&pending.desired)?;for operation in &pending.operations {let state = journal@@ -411,9 +679,110 @@ pub fn replay_pending_operations(replay_fail_after_apply()?;journal.sync_operation_state(&operation.operation_id, SyncOperationState::Committed)?;}+ backend.publish_generation_catalog(&base, &pending.desired, &snapshot)?;backend.validate(&base, &pending.desired, &snapshot)?;journal.sync_validated()?;- journal.sync_commit()+ journal.sync_commit()?;+ gc_snapshots(state_dir, journal).context(+ "generation committed durably, but snapshot cleanup failed; inspect the attached cause \+ (remove a refused symlink or repair the reported filesystem permission/I/O problem), \+ then retry the same command to continue bounded cleanup without republishing data",+ )+}++const SNAPSHOT_GC_BATCH_SIZE: usize = 4096;++fn protected_snapshot(+ state_dir: &Path,+ execution: &crate::sync::ExecutionIdentity,+) -> Result<String> {+ let SourceExecutionPolicy::DurableSnapshot {+ reference,+ snapshot_digest,+ } = &execution.source_policy+ else {+ anyhow::bail!("generated authority does not reference a durable snapshot");+ };+ let tx_id = reference+ .strip_prefix("sync-snapshots/")+ .context("protected snapshot reference is not state-relative")?;+ validate_tx_id(tx_id)?;+ let snapshot = open_snapshot(state_dir, tx_id)?;+ anyhow::ensure!(+ snapshot.snapshot_digest == *snapshot_digest,+ "protected snapshot digest disagrees with journal authority"+ );+ Ok(tx_id.to_owned())+}++/// Reclaim only snapshots proven unreferenced by replayed journal authority.+/// The caller owns the journal lock for the complete validation/rename/fsync+/// sequence.+pub fn gc_snapshots(state_dir: &Path, journal: &Journal) -> Result<()> {+ let root = state_dir.join("sync-snapshots");+ let mut protected = std::collections::HashSet::new();+ if let Some(execution) = journal+ .committed_manifest+ .as_ref()+ .and_then(|manifest| manifest.execution.as_ref())+ {+ protected.insert(protected_snapshot(state_dir, execution)?);+ }+ if let Some(execution) = journal+ .pending_sync+ .as_ref()+ .and_then(|pending| pending.desired.execution.as_ref())+ {+ protected.insert(protected_snapshot(state_dir, execution)?);+ }+ let entries = match std::fs::read_dir(&root) {+ Ok(entries) => entries,+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()),+ Err(error) => return Err(error.into()),+ };+ let mut entries = entries.collect::<std::io::Result<Vec<_>>>()?;+ entries.sort_by_key(|entry| entry.file_name());+ // Validate the complete directory before the first mutation. In+ // particular, a late symlink must not be discovered after earlier+ // snapshots have already been deleted.+ for entry in &entries {+ anyhow::ensure!(+ !entry.file_type()?.is_symlink(),+ "snapshot cleanup refuses symlink {}",+ entry.path().display()+ );+ }+ let directory = File::open(&root)?;+ for entry in entries.into_iter().take(SNAPSHOT_GC_BATCH_SIZE) {+ let metadata = entry.file_type()?;+ if !metadata.is_dir() {+ continue;+ }+ let name = entry.file_name().to_string_lossy().into_owned();+ if protected.contains(&name) {+ continue;+ }+ let tombstone = if name.ends_with(".gc") {+ entry.path()+ } else {+ let tombstone = root.join(format!(".{name}.gc"));+ anyhow::ensure!(+ !tombstone.exists(),+ "snapshot tombstone already exists: {}",+ tombstone.display()+ );+ std::fs::rename(entry.path(), &tombstone)?;+ directory.sync_all()?;+ #[cfg(test)]+ if GC_FAIL_AFTER_RENAME.swap(false, std::sync::atomic::Ordering::SeqCst) {+ anyhow::bail!("injected snapshot GC crash after durable tombstone rename");+ }+ tombstone+ };+ std::fs::remove_dir_all(&tombstone)?;+ directory.sync_all()?;+ }+ Ok(())}/// Test helper proving a pending generation binds before mutable discovery.@@ -742,7 +1111,14 @@ pub fn create_snapshot(tx_id: &str,inventory: &Inventory,) -> Result<SourceSnapshot> {- create_snapshot_inner(state_dir, tx_id, inventory, None)+ create_snapshot_inner(+ state_dir,+ tx_id,+ inventory,+ None,+ "source-snapshot-v1",+ u64::MAX,+ )}/// Seal deterministic bulk actions together with source bytes. Extraction and@@ -754,8 +1130,17 @@ pub fn create_prepared_snapshot(tx_id: &str,inventory: &Inventory,plan: &Plan,+ preparation_contract_digest: &str,+ hard_budget_bytes: u64,) -> Result<SourceSnapshot> {- create_snapshot_inner(state_dir, tx_id, inventory, Some(plan))+ create_snapshot_inner(+ state_dir,+ tx_id,+ inventory,+ Some(plan),+ preparation_contract_digest,+ hard_budget_bytes,+ )}fn create_snapshot_inner(@@ -763,6 +1148,8 @@ fn create_snapshot_inner(tx_id: &str,inventory: &Inventory,plan: Option<&Plan>,+ preparation_contract_digest: &str,+ hard_budget_bytes: u64,) -> Result<SourceSnapshot> {validate_tx_id(tx_id)?;ensure_inventory_lengths(inventory)?;@@ -770,19 +1157,69 @@ fn create_snapshot_inner(std::fs::create_dir_all(&root)?;let final_dir = root.join(tx_id);if final_dir.exists() {- return open_snapshot(state_dir, tx_id);+ let existing = open_snapshot(state_dir, tx_id)?;+ anyhow::ensure!(+ existing.preparation_contract_digest == preparation_contract_digest,+ "existing final snapshot was prepared under a different contract"+ );+ let requested = inventory+ .keys+ .iter()+ .zip(&inventory.digests)+ .zip(&inventory.files)+ .map(|((content_id, digest), file)| (content_id.as_str(), digest.as_str(), file.size))+ .collect::<Vec<_>>();+ let sealed = existing+ .files+ .iter()+ .map(|file| {+ (+ file.content_id.as_str(),+ file.content_digest.as_str(),+ file.content_size,+ )+ })+ .collect::<Vec<_>>();+ anyhow::ensure!(+ sealed == requested,+ "existing final snapshot inventory differs from this preparation attempt"+ );+ anyhow::ensure!(+ existing.footprint.total_bytes <= hard_budget_bytes,+ "existing final snapshot exceeds the current logical payload limit"+ );+ return Ok(existing);}let staging = root.join(format!(".{tx_id}.partial"));if staging.exists() {std::fs::remove_dir_all(&staging)?;}std::fs::create_dir(&staging)?;+ let mut cleanup = StagingCleanup {+ path: staging.clone(),+ armed: true,+ };let blobs = staging.join("blobs");std::fs::create_dir(&blobs)?;let prepared_dir = staging.join("prepared");if plan.is_some() {std::fs::create_dir(&prepared_dir)?;}+ let source_bytes = inventory.files.iter().try_fold(0u64, |total, file| {+ total+ .checked_add(file.size)+ .context("snapshot source byte overflow")+ })?;+ if source_bytes > hard_budget_bytes {+ anyhow::bail!(+ "snapshot source footprint {source_bytes} bytes exceeds logical payload limit \+ {hard_budget_bytes} bytes before preparation"+ );+ }+ let mut budget = PayloadBudget {+ used: 0,+ limit: hard_budget_bytes,+ };let mut files = Vec::with_capacity(inventory.files.len());for (ordinal, ((source, content_id), content_digest)) in inventory.files@@ -794,10 +1231,23 @@ fn create_snapshot_inner(crate::content::verify(&source.path, source.size, content_digest)?;let relative_blob = format!("blobs/{ordinal:08}");let destination = staging.join(&relative_blob);- copy_synced(&source.path, &destination)?;+ copy_synced(&source.path, &destination, &mut budget)?;crate::content::verify(&destination, source.size, content_digest)?;+ #[cfg(test)]+ apply_post_seal_source_replacement(&source.path)?;let prepared = plan- .map(|plan| prepare_artifact(&staging, ordinal, source, content_id, &destination, plan))+ .map(|plan| {+ prepare_artifact(+ &staging,+ ordinal,+ tx_id,+ source,+ content_id,+ &destination,+ plan,+ &mut budget,+ )+ }).transpose()?;files.push(SnapshotFile {content_id: content_id.clone(),@@ -807,13 +1257,42 @@ fn create_snapshot_inner(prepared,});}+ let prepared_bytes = files.iter().try_fold(0u64, |total, file| {+ total+ .checked_add(file.prepared.as_ref().map_or(0, |artifact| artifact.bytes))+ .context("prepared snapshot byte overflow")+ })?;+ let total_bytes = source_bytes+ .checked_add(prepared_bytes)+ .context("total snapshot byte overflow")?;+ anyhow::ensure!(+ total_bytes == budget.used && total_bytes <= hard_budget_bytes,+ "snapshot logical payload accounting mismatch"+ );+ let footprint = SnapshotFootprint {+ source_bytes,+ prepared_bytes,+ total_bytes,+ hard_budget_bytes,+ };snapshot_failpoint(1)?;let snapshot = SourceSnapshot {version: SNAPSHOT_VERSION,tx_id: tx_id.to_string(),- snapshot_digest: snapshot_digest(tx_id, &files)?,+ preparation_contract_digest: preparation_contract_digest.to_owned(),+ footprint: footprint.clone(),+ started: chrono::Utc::now().to_rfc3339(),+ snapshot_digest: String::new(),files,};+ let mut snapshot = snapshot;+ snapshot.snapshot_digest = snapshot_digest(+ tx_id,+ &snapshot.started,+ preparation_contract_digest,+ &footprint,+ &snapshot.files,+ )?;write_synced_json(&staging.join("manifest.json"), &snapshot)?;sync_dir(&blobs)?;if plan.is_some() {@@ -822,6 +1301,7 @@ fn create_snapshot_inner(sync_dir(&staging)?;snapshot_failpoint(2)?;std::fs::rename(&staging, &final_dir)?;+ cleanup.armed = false;sync_dir(&root)?;snapshot_failpoint(3)?;open_snapshot(state_dir, tx_id)@@ -836,8 +1316,33 @@ pub fn open_snapshot(state_dir: &Path, tx_id: &str) -> Result<SourceSnapshot> {snapshot.version == SNAPSHOT_VERSION && snapshot.tx_id == tx_id,"source snapshot identity mismatch");+ let source_bytes = snapshot.files.iter().try_fold(0u64, |sum, file| {+ sum.checked_add(file.content_size)+ .context("snapshot source footprint overflow")+ })?;+ let prepared_bytes = snapshot.files.iter().try_fold(0u64, |sum, file| {+ sum.checked_add(file.prepared.as_ref().map_or(0, |artifact| artifact.bytes))+ .context("snapshot prepared footprint overflow")+ })?;+ let total_bytes = source_bytes+ .checked_add(prepared_bytes)+ .context("snapshot total footprint overflow")?;+ anyhow::ensure!(+ snapshot.footprint.source_bytes == source_bytes+ && snapshot.footprint.prepared_bytes == prepared_bytes+ && snapshot.footprint.total_bytes == total_bytes+ && total_bytes <= snapshot.footprint.hard_budget_bytes,+ "source snapshot footprint does not match its sealed artifacts"+ );anyhow::ensure!(- snapshot.snapshot_digest == snapshot_digest(tx_id, &snapshot.files)?,+ snapshot.snapshot_digest+ == snapshot_digest(+ tx_id,+ &snapshot.started,+ &snapshot.preparation_contract_digest,+ &snapshot.footprint,+ &snapshot.files,+ )?,"source snapshot manifest digest mismatch");for file in &snapshot.files {@@ -866,13 +1371,16 @@ pub fn open_snapshot(state_dir: &Path, tx_id: &str) -> Result<SourceSnapshot> {Ok(snapshot)}+#[allow(clippy::too_many_arguments)]fn prepare_artifact(staging: &Path,ordinal: usize,+ generation_identity: &str,source: &crate::walk::FileEntry,content_id: &str,snapshot_blob: &Path,plan: &Plan,+ budget: &mut PayloadBudget,) -> Result<PreparedArtifact> {let assignment = plan.files@@ -907,7 +1415,7 @@ fn prepare_artifact(.iter().map(|(group, slug)| (group.clone(), slug.as_str())).collect();- let sniffed = crate::sniff::sniff(&source.path)+ let sniffed = crate::sniff::sniff_with_name(snapshot_blob, &source.path).with_context(|| format!("sniff {} for durable preparation", source.rel))?;let relative_ndjson = format!("prepared/{ordinal:08}.ndjson");let path = staging.join(&relative_ndjson);@@ -915,7 +1423,10 @@ fn prepare_artifact(.create_new(true).write(true).open(&path)?;- let mut writer = BufWriter::new(file);+ let mut writer = BudgetWriter {+ inner: BufWriter::new(file),+ budget,+ };let mut records = 0u64;let mut passages = 0u64;let mut vectors = 0u64;@@ -952,7 +1463,10 @@ fn prepare_artifact(fields.insert("ax_file".into(), Value::String(content_id.to_string()));fields.insert("ax_locator".into(), Value::String(record.locator.clone()));fields.insert("ax_dataset".into(), Value::String(slug.to_string()));- fields.insert("ax_run".into(), Value::String("generation-pending".into()));+ fields.insert(+ "ax_run".into(),+ Value::String(generation_identity.to_owned()),+ );fields.insert("ax_format".into(),Value::String(source.path.extension().map_or_else(@@ -962,11 +1476,10 @@ fn prepare_artifact();let id = crate::ids::doc_id(slug, content_id, &record.locator);let action = serde_json::json!({"index": {"_index": dataset.index, "_id": id}});- if writeln!(writer, "{action}")+ if let Err(error) = writeln!(writer, "{action}").and_then(|_| writeln!(writer, "{}", Value::Object(fields.clone())))- .is_err(){- sink_error = Some(anyhow::anyhow!("write prepared NDJSON"));+ sink_error = Some(anyhow::Error::new(error).context("write prepared NDJSON"));return false;}records += 1;@@ -992,7 +1505,7 @@ fn prepare_artifact(return Err(error);}writer.flush()?;- writer.get_ref().sync_all()?;+ writer.inner.get_ref().sync_all()?;drop(writer);let bytes = std::fs::metadata(&path)?.len();Ok(PreparedArtifact {@@ -1036,14 +1549,27 @@ fn ensure_inventory_lengths(inventory: &Inventory) -> Result<()> {Ok(())}-fn snapshot_digest(tx_id: &str, files: &[SnapshotFile]) -> Result<String> {+fn snapshot_digest(+ tx_id: &str,+ started: &str,+ preparation_contract_digest: &str,+ footprint: &SnapshotFootprint,+ files: &[SnapshotFile],+) -> Result<String> {let mut files = files.to_vec();files.sort_by(|left, right| {left.content_id.cmp(&right.content_id).then_with(|| left.relative_blob.cmp(&right.relative_blob))});- let encoded = serde_json::to_vec(&(SNAPSHOT_VERSION, tx_id, files))?;+ let encoded = serde_json::to_vec(&(+ SNAPSHOT_VERSION,+ tx_id,+ started,+ preparation_contract_digest,+ footprint,+ files,+ ))?;Ok(format!("axs1-{:032x}",xxhash_rust::xxh3::xxh3_128(&encoded)@@ -1062,14 +1588,18 @@ fn validate_tx_id(tx_id: &str) -> Result<()> {Ok(())}-fn copy_synced(source: &Path, destination: &Path) -> Result<()> {+fn copy_synced(source: &Path, destination: &Path, budget: &mut PayloadBudget) -> Result<()> {let mut input = File::open(source)?;- let mut output = OpenOptions::new()+ let output = OpenOptions::new().create_new(true).write(true).open(destination)?;+ let mut output = BudgetWriter {+ inner: output,+ budget,+ };std::io::copy(&mut input, &mut output)?;- output.sync_all()?;+ output.inner.sync_all()?;Ok(())}@@ -1086,6 +1616,29 @@ fn sync_dir(path: &Path) -> Result<()> {Ok(())}+#[cfg(test)]+fn arm_source_mutation_after_seal(path: &Path, replacement: &[u8]) {+ *POST_SEAL_SOURCE_REPLACEMENT.lock().unwrap() =+ Some((path.to_path_buf(), replacement.to_vec()));+}++#[cfg(test)]+fn apply_post_seal_source_replacement(path: &Path) -> Result<()> {+ let replacement = {+ let mut armed = POST_SEAL_SOURCE_REPLACEMENT.lock().unwrap();+ if armed.as_ref().is_some_and(|(expected, _)| expected == path) {+ armed.take().map(|(_, bytes)| bytes)+ } else {+ None+ }+ };+ if let Some(replacement) = replacement {+ std::fs::write(path, replacement)+ .with_context(|| format!("inject post-seal source replacement {}", path.display()))?;+ }+ Ok(())+}+#[cfg(test)]fn fail_next_snapshot(boundary: u8) {SNAPSHOT_FAILPOINT.store(boundary, std::sync::atomic::Ordering::SeqCst);@@ -1183,13 +1736,14 @@ mod tests {prefix: "ax".into(),follow_symlinks: false,chunker_identity: "chunker-v1".into(),+ embedding_identity_sha256: "a".repeat(64),embedding_backend: "lexical".into(),- embedding_model: "feature-hash".into(),- embedding_tokenizer: "none".into(),embedding_dimension: 384,+ embedding_semantic_contract: "semantic_text-derived-vector.v1".into(),+ embedding_resumable: true,graph_enabled: false,- brain: "brain".into(),- detector_identity: "none".into(),+ brain: "disabled".into(),+ detector_identity: "disabled".into(),schema_identity: "schema-v1".into(),index_identity: "index-v1".into(),source_policy: SourceExecutionPolicy::DurableSnapshot {@@ -1225,9 +1779,16 @@ mod tests {struct FakeBackend {live: BTreeMap<String, LiveGroup>,applications: Vec<String>,+ provisions: usize,+ validations: usize,}impl SyncOperationBackend for FakeBackend {+ fn provision_generation(&mut self, _desired: &GenerationManifest) -> Result<()> {+ self.provisions += 1;+ Ok(())+ }+fn apply(&mut self,operation: &SyncOperation,@@ -1269,6 +1830,7 @@ mod tests {desired: &GenerationManifest,_snapshot: &SourceSnapshot,) -> Result<()> {+ self.validations += 1;let expected: BTreeMap<String, LiveGroup> = desired.groups.iter()@@ -1309,16 +1871,6 @@ mod tests {journal.sync_begin(&pending).unwrap();}- fn empty_base_for(plan: &Plan) -> Plan {- let mut base = plan.clone();- base.files.clear();- base.duplicate_files.clear();- for dataset in &mut base.datasets {- dataset.file_count = 0;- }- base- }-#[test]fn inventory_bridge_retains_counts_only_for_identical_content() {let corpus = tempfile::tempdir().unwrap();@@ -1376,8 +1928,15 @@ mod tests {.unwrap();let inventory = inventory(corpus.path());let plan = plan_for(&inventory);- let snapshot =- create_prepared_snapshot(state.path(), "tx-prepared", &inventory, &plan).unwrap();+ let snapshot = create_prepared_snapshot(+ state.path(),+ "tx-prepared",+ &inventory,+ &plan,+ "test-preparation-v1",+ u64::MAX,+ )+ .unwrap();let prepared = snapshot.files[0].prepared.as_ref().unwrap();assert_eq!(prepared.records, 2);assert!(prepared.bytes > 0);@@ -1403,6 +1962,153 @@ mod tests {assert!(open_snapshot(state.path(), "tx-prepared").is_err());}+ #[test]+ fn preparation_sniffs_and_extracts_only_the_sealed_blob_after_live_mutation() {+ let _guard = SNAPSHOT_TEST_LOCK+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ let corpus = tempfile::tempdir().unwrap();+ let state = tempfile::tempdir().unwrap();+ let source = corpus.path().join("records.jsonl");+ let sealed = b"{\"message\":\"sealed alpha\"}\n{\"message\":\"sealed beta\"}\n";+ let replacement = b"id,value\n9,live mutation\n";+ std::fs::write(&source, sealed).unwrap();+ let inventory = inventory(corpus.path());+ let plan = plan_for(&inventory);+ arm_source_mutation_after_seal(&source, replacement);++ let snapshot = create_prepared_snapshot(+ state.path(),+ "tx-post-seal-mutation",+ &inventory,+ &plan,+ "test-preparation-v1",+ u64::MAX,+ )+ .unwrap();++ assert_eq!(std::fs::read(&source).unwrap(), replacement);+ let prepared = snapshot.files[0].prepared.as_ref().unwrap();+ assert_eq!(prepared.records, 2);+ let root = state.path().join("sync-snapshots/tx-post-seal-mutation");+ let ndjson = std::fs::read_to_string(root.join(&prepared.relative_ndjson)).unwrap();+ assert!(ndjson.contains("sealed alpha"), "{ndjson}");+ assert!(ndjson.contains("sealed beta"), "{ndjson}");+ assert!(!ndjson.contains("live mutation"), "{ndjson}");+ for document in ndjson+ .lines()+ .skip(1)+ .step_by(2)+ .map(|line| serde_json::from_str::<Value>(line).unwrap())+ {+ assert_eq!(document["ax_format"], "jsonl");+ assert!(document.get("message").is_some(), "{document}");+ assert!(document.get("id").is_none(), "{document}");+ assert!(document.get("value").is_none(), "{document}");+ }+ assert_eq!(+ std::fs::read(root.join(&snapshot.files[0].relative_blob)).unwrap(),+ sealed+ );+ }++ #[test]+ fn prepared_snapshot_budget_aborts_and_removes_staging() {+ let corpus = tempfile::tempdir().unwrap();+ std::fs::write(corpus.path().join("a.txt"), "budgeted source bytes").unwrap();+ let inventory =+ crate::content::resolve(crate::walk::walk(corpus.path(), false).unwrap()).unwrap();+ let plan = plan_for(&inventory);+ let state = tempfile::tempdir().unwrap();+ let error = create_prepared_snapshot(+ state.path(),+ "tx-budget",+ &inventory,+ &plan,+ "test-preparation-v1",+ inventory.files[0].size + 1,+ )+ .unwrap_err();+ assert!(format!("{error:#}").contains("logical payload"));+ assert!(!state+ .path()+ .join("sync-snapshots/.tx-budget.partial")+ .exists());+ assert!(!state.path().join("sync-snapshots/tx-budget").exists());+ }++ #[test]+ fn payload_writer_refuses_bytes_before_they_reach_disk() {+ let state = tempfile::tempdir().unwrap();+ let path = state.path().join("payload");+ let file = File::create(&path).unwrap();+ let mut budget = PayloadBudget { used: 0, limit: 3 };+ let mut writer = BudgetWriter {+ inner: file,+ budget: &mut budget,+ };+ writer.write_all(b"abc").unwrap();+ assert!(writer.write_all(b"d").is_err());+ writer.flush().unwrap();+ drop(writer);+ assert_eq!(std::fs::metadata(path).unwrap().len(), 3);+ assert_eq!(budget.used, 3);+ }++ #[test]+ fn gc_tombstone_crash_is_retryable_and_symlinks_are_refused() {+ let state = tempfile::tempdir().unwrap();+ let mut journal = Journal::open(state.path(), "root", "url", "prefix", 300, false).unwrap();+ journal.sync_bootstrap_genesis().unwrap();+ let snapshots = state.path().join("sync-snapshots");+ std::fs::create_dir_all(snapshots.join("orphan")).unwrap();+ std::fs::write(snapshots.join("orphan/blob"), b"x").unwrap();+ GC_FAIL_AFTER_RENAME.store(true, std::sync::atomic::Ordering::SeqCst);+ assert!(gc_snapshots(state.path(), &journal).is_err());+ assert!(!snapshots.join("orphan").exists());+ assert!(snapshots.join(".orphan.gc").exists());+ gc_snapshots(state.path(), &journal).unwrap();+ assert!(!snapshots.join(".orphan.gc").exists());++ #[cfg(unix)]+ {+ std::fs::create_dir(snapshots.join("a-orphan")).unwrap();+ std::os::unix::fs::symlink(state.path(), snapshots.join("hostile")).unwrap();+ assert!(gc_snapshots(state.path(), &journal).is_err());+ assert!(snapshots.join("a-orphan").exists());+ assert!(snapshots.join("hostile").exists());+ }+ }++ #[test]+ fn gc_validates_every_protected_snapshot_before_deleting_orphans() {+ let _guard = SNAPSHOT_TEST_LOCK+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ let corpus = tempfile::tempdir().unwrap();+ std::fs::write(corpus.path().join("a.txt"), "protected bytes").unwrap();+ let inventory =+ crate::content::resolve(crate::walk::walk(corpus.path(), false).unwrap()).unwrap();+ let plan = plan_for(&inventory);+ let state = tempfile::tempdir().unwrap();+ let snapshot = create_snapshot(state.path(), "tx-protected", &inventory).unwrap();+ let mut journal = Journal::open(state.path(), "root", "url", "prefix", 300, false).unwrap();+ journal.sync_bootstrap_genesis().unwrap();+ let groups = groups_from_inventory(&inventory, &plan, &[]).unwrap();+ begin(&mut journal, "tx-protected", &snapshot, plan, groups);+ let orphan = state.path().join("sync-snapshots/orphan");+ std::fs::create_dir(&orphan).unwrap();+ std::fs::write(orphan.join("blob"), b"orphan").unwrap();+ let protected_blob = state+ .path()+ .join("sync-snapshots/tx-protected")+ .join(&snapshot.files[0].relative_blob);+ std::fs::write(protected_blob, b"corrupt").unwrap();+ assert!(gc_snapshots(state.path(), &journal).is_err());+ assert!(orphan.exists(), "validation must precede every deletion");+ assert!(state.path().join("sync-snapshots/tx-protected").exists());+ }+#[test]fn prepared_and_metadata_streams_preserve_pairs_under_tiny_budgets() {let input = concat!(@@ -1490,13 +2196,14 @@ mod tests {"prefix": "ax","follow_symlinks": false,"chunker_identity": "chunker",+ "embedding_identity_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","embedding_backend": "lexical",- "embedding_model": "feature-hash",- "embedding_tokenizer": "none","embedding_dimension": 384,+ "embedding_semantic_contract": "semantic_text-derived-vector.v1",+ "embedding_resumable": true,"graph_enabled": false,- "brain": "brain",- "detector_identity": "none",+ "brain": "disabled",+ "detector_identity": "disabled","schema_identity": "schema","index_identity": "index","source_policy": {@@ -1545,7 +2252,7 @@ mod tests {let snapshot = create_snapshot(state.path(), "tx-add", &inventory).unwrap();let mut journal =Journal::open(state.path(), "root", "http://engine", "ax", 300, false).unwrap();- journal.write_plan(&empty_base_for(&plan)).unwrap();+ journal.sync_bootstrap_genesis().unwrap();begin(&mut journal, "tx-add", &snapshot, plan, groups);let mut backend = FakeBackend::default();@@ -1580,7 +2287,7 @@ mod tests {let first_snapshot = create_snapshot(state.path(), "tx-first", &first_inventory).unwrap();let mut journal =Journal::open(state.path(), "root", "http://engine", "ax", 300, false).unwrap();- journal.write_plan(&empty_base_for(&first_plan)).unwrap();+ journal.sync_bootstrap_genesis().unwrap();begin(&mut journal,"tx-first",@@ -1671,4 +2378,39 @@ mod tests {.groups.is_empty());}++ #[test]+ fn serialized_graph_identity_with_zero_operations_fails_before_backend_or_commit() {+ let state = tempfile::tempdir().unwrap();+ let inventory = Inventory {+ files: vec![],+ keys: vec![],+ digests: vec![],+ duplicates: vec![],+ };+ let snapshot = create_snapshot(state.path(), "tx-hostile", &inventory).unwrap();+ let mut journal =+ Journal::open(state.path(), "root", "http://engine", "ax", 300, false).unwrap();+ journal.sync_bootstrap_genesis().unwrap();+ let base = journal.committed_manifest.as_ref().unwrap().clone();+ let desired = desired(1, "tx-hostile", &snapshot, Plan::default(), vec![]);+ let valid = PendingSync::new("tx-hostile".into(), &base, desired).unwrap();+ assert!(valid.operations.is_empty());+ journal.sync_begin(&valid).unwrap();++ let mut encoded = serde_json::to_value(journal.pending_sync.as_ref().unwrap()).unwrap();+ encoded["desired"]["execution"]["graph_enabled"] = Value::Bool(true);+ encoded["desired"]["execution"]["brain"] = Value::String("hostile".into());+ encoded["desired"]["execution"]["detector_identity"] =+ Value::String("hostile-detectors".into());+ journal.pending_sync = Some(serde_json::from_value(encoded).unwrap());++ let mut backend = FakeBackend::default();+ assert!(replay_pending_operations(state.path(), &mut journal, &mut backend).is_err());+ assert_eq!(backend.provisions, 0);+ assert!(backend.applications.is_empty());+ assert_eq!(backend.validations, 0);+ assert_eq!(journal.committed_manifest.as_ref().unwrap().generation, 0);+ assert!(journal.pending_sync.is_some());+ }}diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rsindex 6def96b9b..1d4d5a70b 100644--- a/engine/crates/xerj-server/src/brain.rs+++ b/engine/crates/xerj-server/src/brain.rs@@ -510,6 +510,7 @@ fn index_cfg(cfg: &BrainCfg, brain: &str, api_key: Option<String>) -> IndexCfg {pdf_timeout_secs: 120,bulk_mb: BULK_MB,bulk_timeout_secs: 300,+ snapshot_max_bytes: 64u64 << 30,prefix: "ax".into(),state_dir: None,fresh: cfg.fresh,diff --git a/landing/docs/recipes/zero-config-autoindex.html b/landing/docs/recipes/zero-config-autoindex.htmlindex edc3d0b29..ae18044c0 100644--- a/landing/docs/recipes/zero-config-autoindex.html+++ b/landing/docs/recipes/zero-config-autoindex.html@@ -239,8 +239,7 @@ live indices at http://localhost:9280:ax-docs 1 docsax-exports 800 docsautoindex-catalog 9 docs</pre>- <p><code class="inline">--fresh</code> ignores the journal and rebuilds the plan in place (ids stay idempotent), which is how you pick up files added since the last run.</p>- <p>A rerun that resumes an existing plan indexes changed files and reports added ones as skipped. Deleting a file is refused: nothing here removes the documents it already published, so the rerun stops before touching the destination and names what is gone. Restore the file and rerun, or rebuild — in place by deleting the named indices and the state directory, or isolated under a new <code class="inline">--state-dir</code>, <code class="inline">--prefix</code> and <code class="inline">--brain</code> (or <code class="inline">--no-graph</code>), validated before you switch readers. The shared <code class="inline">autoindex-catalog</code> and the old target require explicit cleanup.</p>+ <p><code class="inline">--fresh</code> starts without resume state only when the selected state directory has no durable plan; it never removes stale destination records. For an independent rebuild, use a new <code class="inline">--state-dir</code>, new <code class="inline">--prefix</code>, and new <code class="inline">--brain</code> when graph detection is enabled (or add <code class="inline">--no-graph</code>). Validate before switching readers. The shared <code class="inline">autoindex-catalog</code> and old target require explicit, validated cleanup. Generated journals with <code class="inline">--no-graph</code> reconcile additions, changes, deletions, renames, and no-op reruns. Legacy journals and graph-enabled generations refuse membership changes before remote mutation.</p><hr>diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 698924812..e64ac3d47 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -140,18 +140,17 @@ destabilize the server.xerj autoindex map [--url U] [--json] [--dataset SLUG]xerj autoindex status [--url U] [--state-dir P]---fresh ignores the journal and rebuilds the plan in place (ids stay-idempotent); it is how you pick up files added since the last run. A rerun-that resumes an existing plan indexes changed files and reports added ones as-skipped. Deleting an indexed file is refused before any destination change,-because nothing here removes the documents it already published: restore the-file and rerun, or rebuild — in place by deleting the named indices and the-state directory, or isolated under a new --state-dir, --prefix and --brain (or---no-graph), validated before switching readers. The shared autoindex-catalog-and the old target require explicit cleanup.+--fresh starts without resume state only when the selected state directory has+no durable plan. It never removes stale destination records. An independent+rebuild requires a new --state-dir, new --prefix, and new --brain when graph+detection is enabled (or --no-graph), followed by validation before switching+readers. The shared autoindex-catalog and old target require explicit,+validated cleanup. Generated journals with --no-graph reconcile additions,+changes, deletions, renames, and no-op reruns. Legacy journals and graph-enabled+generations refuse membership changes before remote mutation.Exit codes: 0 complete · 3 completed-with-junk (junk recorded, never fatal) ·-2 usage · 1 endpoint/journal error or a refused corpus removal.+2 usage · 1 endpoint/journal error or refused unsafe corpus-state transition.Pipeline: walk → sniff → sample → infer → map → extract → bulk → correlate →catalog → verify.