AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): resolve the generation path against main's resource, progress and spool policies
xerj-org71250fb33e0b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 971eee863..38b6d88d8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -79,6 +79,87 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0file. `--json` still carries every entry.+### Added — incremental `--no-graph` corpus reconciliation++- **`xerj autoindex --no-graph` now reconciles a changed folder instead of+ re-deriving it.** A `--no-graph` run commits a durable *corpus generation*: a+ manifest of content groups (stable group identity, content digest and byte+ length, canonical path plus aliases, dataset assignment, validated output+ counts) plus a sealed source snapshot of the prepared records. A later run+ projects the current inventory onto the committed manifest and publishes only+ the difference — additions, content changes, deletions, renames, junk+ transitions and duplicate-alias moves all converge, and a no-op re-run issues+ no data bulk and appends no new generation. A run interrupted mid-generation+ replays from its own sealed snapshot, so the result never depends on whether+ the source tree changed after the crash. Dataset and mapping identity is+ frozen at generation 1: a file that would need a *new* dataset or a mapping+ change is refused before any remote mutation rather than silently widening+ the schema.++ Scope, plainly: only `--no-graph`. Graph-enabled runs are unchanged, and each+ changed generation still copies and prepares the full corpus (O(N) staging) —+ the win is in what gets published and verified, not yet in what gets read.+ `--snapshot-max-gb` (default 64) caps the logical staged payload; it is a+ payload budget, not a disk-space or peak-memory guarantee.++ Original work by Leonid Bugaev (@buger).++- **Junk keeps the contract it always had on the generated path: recorded,+ never fatal.** A file that cannot be read or recognised (`plan.junk_files`)+ and a record that no dataset assignment accepts (`stats.junk`) both used to+ abort a `--no-graph` run outright — the first with+ `plan has no assignment for <file>`, the second with+ `durable preparation of <file> produced N junk records`. A single zero-byte+ file was enough to make a whole folder unindexable. Both are now counted+ instead: the sealed prepared artifact carries its junk-record count, the+ manifest group carries it into the generation, and the catalog's per-file+ `junk` and run-level `junk_records_total` report it instead of a hardcoded+ zero. Such a run exits **3** ("completed with junk"), the same signal the+ legacy path publishes — the generated path previously returned a flat `0`+ and lost it.++- **`--dry-run` is honoured on an already-generated state directory.** It was+ evaluated only after the pending-replay and reconcile branches had already+ published and committed, so on any state directory past generation 1 the flag+ was accepted, silently ignored, and the destination mutated. It is now decided+ before either branch: it prints the plan a real run would act on — the+ projected reconcile plan, or the sealed plan of a pending generation — and+ returns without opening the journal for write, without snapshot GC, and+ without a single bulk request.++### Changed — `autoindex --fresh` is refused on a durable generation++- **`--fresh` no longer silently destroys a committed corpus generation.**+ `--fresh` deletes the resume journal, and the snapshot GC that follows every+ journal open then sees an empty protected set and removes every sealed+ snapshot — so on a generated state directory `--fresh` would discard the+ committed manifest, the pending replay evidence, and the alias/path/+ stale-record knowledge, while cleaning nothing at the destination. It is now+ refused up front, naming the generation that blocks it, and pointing at the+ plain re-run (which reconciles the change) or at an isolated rebuild with a+ new `--state-dir` and `--prefix`.++ **Unchanged for everyone else.** On a legacy (non-generated) journal —+ including every graph-enabled corpus and anything `xerj brain` writes —+ `--fresh` behaves exactly as before: it ignores the journal and restarts,+ which is what `xerj brain`'s documented self-heal for a wiped data directory+ depends on.++### Migration — pre-generation `--no-graph` state directories must be rebuilt++- **A `--no-graph` state directory written before this release cannot be+ adopted as generation zero, and the first `--no-graph` run against it will+ refuse with a followable rebuild command.** This is a deliberate, versioned+ format boundary (`sync::GENERATION_FORMAT_VERSION = 1`), not a validation+ failure: a pre-generation resume plan records no stable group identity, no+ content byte length and no validated per-group output counts, so no amount of+ evidence in it could reconstruct a manifest group. The refusal prints the+ exact `xerj autoindex` argv for an isolated rebuild into a new `--state-dir`+ and `--prefix`; the old target and the shared `autoindex-catalog` are left+ alone and require explicit, validated cleanup once the new target is+ verified. Nothing is migrated in place and no destination data is touched by+ the refusal. Graph-enabled state directories are not affected.+## [1.0.0-rc.14] - 2026-08-10### Changed@@ -386,85 +467,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0prints the funnel on demand, including which repo-level numbers arecontaminated and why.-### Added — incremental `--no-graph` corpus reconciliation--- **`xerj autoindex --no-graph` now reconciles a changed folder instead of- re-deriving it.** A `--no-graph` run commits a durable *corpus generation*: a- manifest of content groups (stable group identity, content digest and byte- length, canonical path plus aliases, dataset assignment, validated output- counts) plus a sealed source snapshot of the prepared records. A later run- projects the current inventory onto the committed manifest and publishes only- the difference — additions, content changes, deletions, renames, junk- transitions and duplicate-alias moves all converge, and a no-op re-run issues- no data bulk and appends no new generation. A run interrupted mid-generation- replays from its own sealed snapshot, so the result never depends on whether- the source tree changed after the crash. Dataset and mapping identity is- frozen at generation 1: a file that would need a *new* dataset or a mapping- change is refused before any remote mutation rather than silently widening- the schema.-- Scope, plainly: only `--no-graph`. Graph-enabled runs are unchanged, and each- changed generation still copies and prepares the full corpus (O(N) staging) —- the win is in what gets published and verified, not yet in what gets read.- `--snapshot-max-gb` (default 64) caps the logical staged payload; it is a- payload budget, not a disk-space or peak-memory guarantee.--- **Junk keeps the contract it always had on the generated path: recorded,- never fatal.** A file that cannot be read or recognised (`plan.junk_files`)- and a record that no dataset assignment accepts (`stats.junk`) both used to- abort a `--no-graph` run outright — the first with- `plan has no assignment for <file>`, the second with- `durable preparation of <file> produced N junk records`. A single zero-byte- file was enough to make a whole folder unindexable. Both are now counted- instead: the sealed prepared artifact carries its junk-record count, the- manifest group carries it into the generation, and the catalog's per-file- `junk` and run-level `junk_records_total` report it instead of a hardcoded- zero. Such a run exits **3** ("completed with junk"), the same signal the- legacy path publishes — the generated path previously returned a flat `0`- and lost it.--- **`--dry-run` is honoured on an already-generated state directory.** It was- evaluated only after the pending-replay and reconcile branches had already- published and committed, so on any state directory past generation 1 the flag- was accepted, silently ignored, and the destination mutated. It is now decided- before either branch: it prints the plan a real run would act on — the- projected reconcile plan, or the sealed plan of a pending generation — and- returns without opening the journal for write, without snapshot GC, and- without a single bulk request.--### Changed — `autoindex --fresh` is refused on a durable generation--- **`--fresh` no longer silently destroys a committed corpus generation.**- `--fresh` deletes the resume journal, and the snapshot GC that follows every- journal open then sees an empty protected set and removes every sealed- snapshot — so on a generated state directory `--fresh` would discard the- committed manifest, the pending replay evidence, and the alias/path/- stale-record knowledge, while cleaning nothing at the destination. It is now- refused up front, naming the generation that blocks it, and pointing at the- plain re-run (which reconciles the change) or at an isolated rebuild with a- new `--state-dir` and `--prefix`.-- **Unchanged for everyone else.** On a legacy (non-generated) journal —- including every graph-enabled corpus and anything `xerj brain` writes —- `--fresh` behaves exactly as before: it ignores the journal and restarts,- which is what `xerj brain`'s documented self-heal for a wiped data directory- depends on.--### Migration — pre-generation `--no-graph` state directories must be rebuilt--- **A `--no-graph` state directory written before this release cannot be- adopted as generation zero, and the first `--no-graph` run against it will- refuse with a followable rebuild command.** This is a deliberate, versioned- format boundary (`sync::GENERATION_FORMAT_VERSION = 1`), not a validation- failure: a pre-generation resume plan records no stable group identity, no- content byte length and no validated per-group output counts, so no amount of- evidence in it could reconstruct a manifest group. The refusal prints the- exact `xerj autoindex` argv for an isolated rebuild into a new `--state-dir`- and `--prefix`; the old target and the shared `autoindex-catalog` are left- alone and require explicit, validated cleanup once the new target is- verified. Nothing is migrated in place and no destination data is touched by- the refusal. Graph-enabled state directories are not affected.-## [1.0.0-rc.13] - 2026-08-08### Security@@ -734,6 +736,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0caveats. They are tracked for the following release rather than rushed intothis one.+### Changed — runtime-field types (can break existing mappings)- **`mappings.runtime` is now validated against ES's runtime-field typediff --git a/demo/usecases/autoindex/RECIPE_DRAFT.md b/demo/usecases/autoindex/RECIPE_DRAFT.mdindex a035aa865..b5b6d881e 100644--- a/demo/usecases/autoindex/RECIPE_DRAFT.md+++ b/demo/usecases/autoindex/RECIPE_DRAFT.md@@ -203,15 +203,22 @@ done in 0.1s — 3 datasets, 5801 records live, 0 junk records, 1 junk/skipped fjust polite re-runs: in the robustness evaluation, a `kill -9` at 18 s into a200 MB file followed by a re-run converged to **byte-identical counts acrossall 9 datasets** — no duplicates. `xerj autoindex status` shows the journal-and live index counts. `--fresh` ignores the resume journal and restarts (ids stay idempotent). It-never removes stale records from the destination, and it is refused once the-state directory holds a durable corpus generation — re-run without it and the-generated `--no-graph` path reconciles additions, changes, deletions, renames-and no-op reruns incrementally. For an independent rebuild, use a new-`--state-dir` and a new `--prefix`, plus a new `--brain` when graph detection-is enabled (or add `--no-graph`). Validate before switching readers; the shared-`autoindex-catalog` and old target require explicit, validated cleanup. A-`--no-graph` state directory written before the generation format cannot be+and live index counts. `--fresh` ignores the resume journal and restarts (ids stay idempotent), which on a+graph-enabled or pre-generation state directory is how a rerun picks up files added+since the last one. It never removes stale records from the destination, and it is+refused once the state directory holds a durable corpus generation — re-run without it+and the generated `--no-graph` path reconciles additions, changes, deletions, renames+and no-op reruns incrementally.++On a graph-enabled or pre-generation journal, a rerun that resumes an existing plan+indexes changed files and reports added ones as skipped. Deleting a file is refused+there: nothing on that path removes the documents it already published, so the rerun+stops before touching the destination and names what is gone — the first ten, then an+`… and N more` tail, with `--json` carrying every entry. Restore the file and rerun, or+rebuild — in place by deleting the named indices and the state directory, or isolated+under a new `--state-dir`, `--prefix` and `--brain` (or `--no-graph`), validated before+you switch readers. The shared `autoindex-catalog` and the old target require explicit+cleanup. A `--no-graph` state directory written before the generation format cannot beadopted in place and must be rebuilt the same way.## Reproduce it yourselfdiff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.mdindex 2a584d9d4..f3c92f1c3 100644--- a/docs/recipes/zero-config-indexing.md+++ b/docs/recipes/zero-config-indexing.md@@ -228,15 +228,22 @@ live indices at http://localhost:9280:autoindex-catalog 9 docs```-`--fresh` ignores the resume journal and restarts (ids stay idempotent). It-never removes stale records from the destination, and it is refused once the-state directory holds a durable corpus generation — re-run without it and the-generated `--no-graph` path reconciles additions, changes, deletions, renames-and no-op reruns incrementally. For an independent rebuild, use a new-`--state-dir` and a new `--prefix`, plus a new `--brain` when graph detection-is enabled (or add `--no-graph`). Validate before switching readers; the shared-`autoindex-catalog` and old target require explicit, validated cleanup. A-`--no-graph` state directory written before the generation format cannot be+`--fresh` ignores the resume journal and restarts (ids stay idempotent), which on a+graph-enabled or pre-generation state directory is how a rerun picks up files added+since the last one. It never removes stale records from the destination, and it is+refused once the state directory holds a durable corpus generation — re-run without it+and the generated `--no-graph` path reconciles additions, changes, deletions, renames+and no-op reruns incrementally.++On a graph-enabled or pre-generation journal, a rerun that resumes an existing plan+indexes changed files and reports added ones as skipped. Deleting a file is refused+there: nothing on that path removes the documents it already published, so the rerun+stops before touching the destination and names what is gone — the first ten, then an+`… and N more` tail, with `--json` carrying every entry. Restore the file and rerun, or+rebuild — in place by deleting the named indices and the state directory, or isolated+under a new `--state-dir`, `--prefix` and `--brain` (or `--no-graph`), validated before+you switch readers. The shared `autoindex-catalog` and the old target require explicit+cleanup. A `--no-graph` state directory written before the generation format cannot beadopted in place and must be rebuilt the same way.## Reproduce it yourselfdiff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex e662e2f01..2ee480f81 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -74,8 +74,10 @@ pub enum Cmd {}const FRESH_HELP: &str =- "ignore an existing resume journal and restart (ids stay idempotent); refused when the \-state directory holds a durable corpus generation, and it never resets destination records";+ "ignore an existing resume journal and restart, rebuild the plan in place\n\+ (ids stay idempotent); it never resets destination records,\n\+ and is refused on a durable corpus generation — see\n\+ RESUME POLICY";const RESUME_POLICY_HELP: &str ="generated --no-graph journals reconcile add, change, delete, rename, and no-op runs; \a --no-graph state directory written before the generation format must be rebuilt into a new \@@ -186,6 +188,14 @@ pub fn print_help() {\n\RESUME POLICY:\n\{resume_policy_help}.\n\+ On a graph-enabled or pre-generation journal the durable plan supports no-op\n\+ resume and same-path content replacement. Files added after that plan was frozen\n\+ are reported as skipped and are not indexed; --fresh rebuilds the plan in place\n\+ and picks them up. Removing an indexed file is refused there before any remote\n\+ mutation: its documents are already live and nothing on that path deletes them.\n\+ Restore the file and rerun, or rebuild — in place by deleting the published\n\+ indices and the state directory, or isolated under a new --state-dir, --prefix\n\+ and --brain (or --no-graph), validated before you switch readers.\n\--fresh is not cleanup and is not destination reconciliation: it never removes\n\stale records from the destination, and it is refused outright once a durable\n\corpus generation exists (re-run without it — the generated path reconciles the\n\@@ -195,7 +205,8 @@ pub fn print_help() {autoindex-catalog and old target only after validation.\n\\n\EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\- 2 usage; 1 endpoint/journal failure or a refused unsafe state transition\n",+ 2 usage; 1 endpoint/journal failure, a refused corpus removal, or a\n\+ refused unsafe state transition\n",fresh_help = FRESH_HELP,resume_policy_help = RESUME_POLICY_HELP);@@ -545,6 +556,13 @@ mod tests {}}+ #[test]+ fn fresh_help_points_at_the_resume_policy_that_bounds_it() {+ assert!(super::FRESH_HELP.contains("rebuild the plan in place"));+ assert!(super::FRESH_HELP.contains("ids stay idempotent"));+ assert!(super::FRESH_HELP.contains("RESUME POLICY"));+ }+#[test]fn graph_is_on_by_default_with_derived_brain() {let cfg = index(&["data"]);diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rsindex d67c161b5..becedafe4 100644--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs@@ -69,6 +69,8 @@ struct MockState {/// what `_count`/`_search` answer with, and the run verifies live data/// counts against the journal (#195). Catalog rows are not data rows.catalog_docs: HashMap<String, Value>,+ /// Every (method, path) the run issued. The refusal tests assert on the+ /// ABSENCE of remote mutations, which no document count can express.requests: Vec<(String, String)>,data_bulk_number: usize,fail_data_bulk: usize,@@ -207,11 +209,6 @@ fn handle(mut stream: TcpStream, state: &Arc<Mutex<MockState>>) {if response_delay_ms > 0 {thread::sleep(std::time::Duration::from_millis(response_delay_ms));}- state- .lock()- .unwrap()- .requests- .push((method.to_owned(), path.to_owned()));let (status, response) = if method == "GET" && path == "/v1/embedding/identity" {let identity = state.lock().unwrap().embedding_identity_sha256.clone();@@ -368,11 +365,25 @@ fn bulk_response(body: &[u8], state: &Arc<Mutex<MockState>>) -> Value {// Graph edge bulks are accepted and discarded: this module counts *data*// bulks to place injected failures, so edge bulks must not shift the// numbering (the graph pipeline has its own e2e suite in detect::e2e).+ //+ // Read the target off whichever action verb the line carries. The brain+ // meta document is written with `create`, not `index`, so keying on+ // `/index/_index` alone let it fall through to the catalog branch and be+ // recorded as fixture drift once this module started indexing with the+ // graph enabled.let is_graph = lines.first().and_then(|line| serde_json::from_slice::<Value>(line).ok())- .and_then(|action| action.pointer("/index/_index").cloned())- .and_then(|index| index.as_str().map(str::to_owned))+ .and_then(|action| {+ ["index", "create", "delete", "update"]+ .iter()+ .find_map(|verb| {+ action+ .pointer(&format!("/{verb}/_index"))+ .and_then(Value::as_str)+ .map(str::to_owned)+ })+ }).is_some_and(|index| index.ends_with("-edges"));if is_graph {return json!({"errors": false, "items": []});@@ -573,6 +584,348 @@ fn dataset_catalog_docs(endpoint: &MockEndpoint) -> Vec<Value> {docs}+fn assert_unsupported_delta_without_remote_mutation(+ endpoint: &MockEndpoint,+ config: IndexCfg,+ expected_added: &[&str],+ expected_vanished: &[&str],+) {+ let request_start = endpoint.state.lock().unwrap().requests.len();+ let journal_path = config.state_dir.as_ref().unwrap().join("journal.ndjson");+ let journal_before = fs::read(&journal_path).unwrap();+ let error = run_index(config).unwrap_err();+ let attempted_requests = endpoint.state.lock().unwrap().requests[request_start..].to_vec();+ assert_eq!(+ attempted_requests,+ [("GET".to_owned(), "/".to_owned())],+ "a refused attempt may perform only the endpoint-readiness GET"+ );+ assert_eq!(+ fs::read(journal_path).unwrap(),+ journal_before,+ "preflight refusal must not append a resume event or rewrite the journal"+ );+ let message = format!("{error:#}");+ assert!(message.contains("made no remote mutations"), "{message}");+ assert!(+ message.contains("no longer exist in the folder"),+ "{message}"+ );+ assert!(+ message.contains("restore the removed file(s) and rerun"),+ "{message}"+ );+ assert!(+ message.contains("rebuild in place by deleting the indices"),+ "{message}"+ );+ assert!(+ message.contains("new --state-dir, a new --prefix"),+ "{message}"+ );+ assert!(message.contains("new --brain"), "{message}");+ for path in expected_added {+ assert!(+ message.contains(path),+ "missing added path {path}: {message}"+ );+ }+ for path in expected_vanished {+ assert!(+ message.contains(path),+ "missing vanished path {path}: {message}"+ );+ }+}++/// The documented headline workflow: point autoindex at a folder, add a file,+/// rerun. The rerun must not fail, must say plainly that the added file was+/// not indexed, and `--fresh` must then absorb it in place.+#[test]+fn a_rerun_after_an_added_file_succeeds_and_fresh_absorbs_it() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("first.csv"), "id,value\n1,first\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();+ // 3 = completed-with-junk: the added file is reported as skipped, not+ // indexed, because the frozen plan cannot absorb it.+ let (code, report) = run_index_report(config.clone()).unwrap();+ assert_eq!(code, 3);+ let report = report.unwrap();+ assert_eq!(report["files_junk"], 1, "{report}");+ // `records_total` is the live server-side count (#195), so it still reports+ // the record the first run published. What must be zero is the run-local+ // counter: this rerun indexed nothing, because the frozen plan cannot+ // absorb a file that appeared after it was written.+ assert_eq!(report["records_submitted_this_run"], 0, "{report}");+ assert_eq!(report["files_submitted_this_run"], 0, "{report}");+ // Two live documents for the one indexed source, not one: this module now+ // indexes graph-enabled (`--no-graph` takes the generated executor), and a+ // graph-enabled run publishes a file-level node document alongside the row.+ // `second.csv` still published nothing — that is what the counters above+ // pin.+ assert_eq!(report["records_total"], 2, "{report}");+ {+ let locked = endpoint.state.lock().unwrap();+ let live: Vec<&str> = locked+ .docs+ .values()+ .filter_map(|doc| doc["ax_path"].as_str())+ .collect();+ assert!(+ live.iter().all(|path| *path == "first.csv"),+ "the added file must not be published by a plan that predates it: {live:?}"+ );+ }++ // --fresh rebuilds the plan in place and picks the new file up.+ config.fresh = true;+ assert_eq!(run_index(config).unwrap(), 0);+ let locked = endpoint.state.lock().unwrap();+ let indexed: std::collections::HashSet<&str> = locked+ .docs+ .values()+ .filter_map(|doc| doc["ax_path"].as_str())+ .collect();+ assert!(indexed.contains("first.csv"), "{indexed:?}");+ assert!(indexed.contains("second.csv"), "{indexed:?}");+}++#[test]+fn completed_plan_rejects_vanished_content_group_before_remote_mutation() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("first.csv"), "id,value\n1,first\n").unwrap();+ fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::remove_file(corpus.path().join("second.csv")).unwrap();+ assert_unsupported_delta_without_remote_mutation(&endpoint, config, &[], &["second.csv"]);+}++#[test]+fn completed_plan_rejects_mixed_membership_delta_in_stable_order() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("b-old.csv"), "id,value\n1,b\n").unwrap();+ fs::write(corpus.path().join("a-old.csv"), "id,value\n2,a\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::remove_file(corpus.path().join("b-old.csv")).unwrap();+ fs::remove_file(corpus.path().join("a-old.csv")).unwrap();+ fs::write(corpus.path().join("z-new.csv"), "id,value\n3,z\n").unwrap();+ fs::write(corpus.path().join("m-new.csv"), "id,value\n4,m\n").unwrap();+ config.json = true;+ let request_start = endpoint.state.lock().unwrap().requests.len();+ let error = run_index(config).unwrap_err();+ assert_eq!(+ endpoint.state.lock().unwrap().requests[request_start..],+ [("GET".to_owned(), "/".to_owned())]+ );++ let typed = error+ .downcast_ref::<UnsupportedInventoryDeltaError>()+ .unwrap();+ let value = typed.to_json();+ assert_eq!(value["schema"], "xerj.autoindex.unsupported_sync_delta.v1");+ assert_eq!(value["status"], "error");+ let added: Vec<&str> = value["added_content_groups"]+ .as_array()+ .unwrap()+ .iter()+ .map(|entry| entry["path"].as_str().unwrap())+ .collect();+ let vanished: Vec<&str> = value["vanished_content_groups"]+ .as_array()+ .unwrap()+ .iter()+ .map(|entry| entry["path"].as_str().unwrap())+ .collect();+ assert_eq!(added, ["m-new.csv", "z-new.csv"]);+ assert_eq!(vanished, ["a-old.csv", "b-old.csv"]);+}++/// An in-place EDIT is a replacement, not a removal. `--fresh` is the route+/// the refusal itself recommends for picking up added and changed files, so+/// the gate must not classify the edited file's superseded content key as a+/// vanished group — the path is still there, and the pipeline republishes it.+#[test]+fn fresh_absorbs_an_edited_file_instead_of_calling_it_a_removal() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("notes.csv"), "id,value\n1,before\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::write(corpus.path().join("notes.csv"), "id,value\n1,after\n").unwrap();+ config.fresh = true;+ assert_eq!(+ run_index(config).unwrap(),+ 0,+ "an edited file is a same-path replacement, not a vanished content group"+ );+ let locked = endpoint.state.lock().unwrap();+ assert!(+ locked+ .docs+ .values()+ .any(|doc| doc["value"].as_str() == Some("after")),+ "the new content is live: {:?}",+ locked.docs+ );+ // Documented, and NOT what this gate is for: `--fresh` rebuilds the plan,+ // it does not reconcile the destination. Document ids derive from the+ // content key (`ids::doc_id`), so the pre-edit record stays live beside+ // the new one — the same outcome `--fresh` has always had. An ordinary+ // rerun is the clean route for an edit: it keeps the planned key and runs+ // a delete-before-replace transaction on it. What must never happen is+ // this run being REFUSED by a message claiming a file that is sitting in+ // the folder has vanished.+ assert!(+ locked+ .docs+ .values()+ .any(|doc| doc["value"].as_str() == Some("before")),+ "known --fresh gap: superseded records are not deleted: {:?}",+ locked.docs+ );+}++/// The clean route for the same edit: an ordinary rerun keeps the planned key+/// and replaces its records, so nothing is stranded. This is the contrast that+/// makes the `--fresh` gap above a documented trade-off rather than a silent+/// one.+#[test]+fn an_ordinary_rerun_replaces_an_edited_file_without_stranding_records() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("notes.csv"), "id,value\n1,before\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::write(corpus.path().join("notes.csv"), "id,value\n1,after\n").unwrap();+ assert_eq!(run_index(config).unwrap(), 0);+ let locked = endpoint.state.lock().unwrap();+ let values: Vec<&str> = locked+ .docs+ .values()+ .filter_map(|doc| doc["value"].as_str())+ .collect();+ assert_eq!(values, ["after"], "the superseded record is replaced");+}++/// The composite workflow the run's own stderr recommends: index, then add one+/// file and edit another, rerun (added file reported and skipped by the frozen+/// plan), then `--fresh` to absorb both.+#[test]+fn fresh_absorbs_an_addition_and_an_edit_after_a_reported_rerun() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("first.csv"), "id,value\n1,before\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::write(corpus.path().join("first.csv"), "id,value\n1,after\n").unwrap();+ fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();+ // 3 = completed-with-junk: the frozen plan cannot absorb the added file,+ // so the rerun reports it as skipped rather than refusing.+ assert_eq!(run_index(config.clone()).unwrap(), 3);++ config.fresh = true;+ assert_eq!(+ run_index(config).unwrap(),+ 0,+ "--fresh is the documented way out of exactly this state"+ );+ let locked = endpoint.state.lock().unwrap();+ let live: std::collections::HashSet<&str> = locked+ .docs+ .values()+ .filter_map(|doc| doc["ax_path"].as_str())+ .collect();+ assert!(live.contains("first.csv"), "{live:?}");+ assert!(live.contains("second.csv"), "{live:?}");+ assert!(+ locked+ .docs+ .values()+ .any(|doc| doc["value"].as_str() == Some("after")),+ "the edited file's new content is live: {:?}",+ locked.docs+ );+}++#[test]+fn fresh_cannot_erase_the_plan_and_bypass_the_removal_gate() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("old.csv"), "id,value\n1,old\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::remove_file(corpus.path().join("old.csv")).unwrap();+ fs::write(corpus.path().join("new.csv"), "id,value\n2,new\n").unwrap();+ config.fresh = true;+ assert_unsupported_delta_without_remote_mutation(&endpoint, config, &["new.csv"], &["old.csv"]);+}++#[test]+fn deleting_one_path_of_a_duplicate_pair_is_not_a_removed_content_group() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ let bytes = "id,value\n1,same\n";+ fs::write(corpus.path().join("a.csv"), bytes).unwrap();+ fs::write(corpus.path().join("b.csv"), bytes).unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ // b.csv becomes canonical under the same content key: the group survives+ // the deletion, so no document is stranded and the rerun is allowed.+ fs::remove_file(corpus.path().join("a.csv")).unwrap();+ let result = run_index(config);+ assert!(+ result.is_ok(),+ "a surviving content group must not trip the removal gate: {result:?}"+ );+ let locked = endpoint.state.lock().unwrap();+ // Row documents only: this module indexes graph-enabled, so the surviving+ // source also carries a file-level node document under the same ax_path.+ let live: Vec<&str> = data_rows(&locked)+ .into_iter()+ .filter_map(|doc| doc["ax_path"].as_str())+ .collect();+ assert_eq!(live, ["b.csv"], "canonical path follows the surviving file");+}+#[test]fn unchanged_planned_junk_is_not_reported_as_added_content() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();@@ -595,6 +948,70 @@ fn unchanged_planned_junk_is_not_reported_as_added_content() {);}+#[test]+fn deleted_planned_junk_is_swept_rather_than_refused() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("rows.csv"), "id,value\n1,kept\n").unwrap();+ fs::write(+ corpus.path().join("opaque.bin"),+ [0_u8, 159, 146, 150, 0, 255],+ )+ .unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 3);++ // A junk file publishes exactly one catalog row and nothing else, and the+ // #238 sweep deletes that row. Nothing is stranded, so the removal gate+ // must not fire — refusing here would block a case the pipeline handles+ // completely. Only a file that published DOCUMENTS refuses a rerun.+ fs::remove_file(corpus.path().join("opaque.bin")).unwrap();+ assert_eq!(run_index(config.clone()).unwrap(), 0);+ {+ let locked = endpoint.state.lock().unwrap();+ assert!(+ locked+ .catalog_docs+ .values()+ .all(|doc| doc["path"].as_str() != Some("opaque.bin")),+ "the deleted junk file's catalog row must be swept, not left immortal"+ );+ assert!(+ locked+ .catalog_docs+ .values()+ .any(|doc| doc["path"].as_str() == Some("rows.csv")),+ "the surviving indexed file keeps its entry"+ );+ }++ // Deleting the INDEXED file is still refused: its documents are live.+ fs::remove_file(corpus.path().join("rows.csv")).unwrap();+ assert_unsupported_delta_without_remote_mutation(&endpoint, config, &[], &["rows.csv"]);+}++#[test]+fn completed_plan_rejects_empty_current_folder_before_remote_mutation() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("only.csv"), "id,value\n1,only\n").unwrap();+ let endpoint = MockEndpoint::start(usize::MAX);+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(run_index(config.clone()).unwrap(), 0);++ fs::remove_file(corpus.path().join("only.csv")).unwrap();+ assert_unsupported_delta_without_remote_mutation(&endpoint, config.clone(), &[], &["only.csv"]);+ // Even `--fresh` must not erase the only durable inventory evidence and+ // then report success while the destination still contains the document.+ config.fresh = true;+ assert_unsupported_delta_without_remote_mutation(&endpoint, config, &[], &["only.csv"]);+}+#[test]fn semantic_resume_rejects_embedding_identity_drift_before_another_bulk() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();@@ -822,7 +1239,10 @@ fn deleted_path_bytes_remain_while_its_documents_remain_live() {dataset_catalog_docs(&endpoint)[0]["bytes"],durable_live_bytes);- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 4);+ // Row documents only: this module indexes graph-enabled now that+ // `--no-graph` takes the generated executor, so each source also publishes+ // one file-level node document.+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 4);fs::remove_file(&removed).unwrap();// Autoindex still does not reconcile a missing canonical path: its indexed@@ -839,7 +1259,10 @@ fn deleted_path_bytes_remain_while_its_documents_remain_live() {&[],&["removed.csv"],);- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 4);+ // Row documents only: this module indexes graph-enabled now that+ // `--no-graph` takes the generated executor, so each source also+ // publishes one file-level node document.+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 4);assert_eq!(dataset_catalog_docs(&endpoint)[0]["bytes"],durable_live_bytes@@ -1645,7 +2068,12 @@ fn deleting_an_entire_duplicate_group_strands_no_pending_replacement() {fs::remove_file(corpus.path().join("dup-a.csv")).unwrap();fs::remove_file(corpus.path().join("dup-b.csv")).unwrap();for _ in 0..2 {- assert_eq!(run_index(config.clone()).unwrap(), 0);+ assert_unsupported_delta_without_remote_mutation(+ &endpoint,+ config.clone(),+ &[],+ &["dup-a.csv"],+ );assert_eq!(event_count(state_dir.path(), "file_replace_start"),starts,@@ -2090,7 +2518,7 @@ printf '%s' '{"schema":1,"extractor":"xerj-autoindex/__XERJ_VERSION__","parser":assert_eq!(run_index(config).unwrap(), 0);assert_eq!(fs::read_to_string(&count).unwrap().lines().count(), 2);assert_eq!(file_done_count(state_dir.path()), 1);- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 1);+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 1);}#[cfg(unix)]@@ -2149,7 +2577,7 @@ printf '%s' '{"schema":1,"extractor":"xerj-autoindex/__XERJ_VERSION__","parser":assert_eq!(code, 0);assert_eq!(fs::read_to_string(&count).unwrap().lines().count(), 2);assert_eq!(file_done_count(state_dir.path()), 1);- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 1);+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 1);let reuse = report.unwrap()["pdf_extraction_reuse"].clone();assert_eq!(reuse["reservations_started"], 0);assert_eq!(reuse["capacity_fallbacks"], 1);@@ -2435,7 +2863,7 @@ printf '%s' '{"schema":1,"extractor":"xerj-autoindex/__XERJ_VERSION__","parser":PDFS,"each unique PDF must be parsed once in Phase A and replayed in Phase B");- assert_eq!(endpoint.state.lock().unwrap().docs.len(), PDFS);+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), PDFS);let reuse = report.unwrap()["pdf_extraction_reuse"].clone();assert_eq!(reuse["reservations_started"], PDFS);assert_eq!(@@ -2498,7 +2926,7 @@ printf '%s' '{"schema":1,"extractor":"xerj-autoindex/__XERJ_VERSION__","parser":let (code, report) = run_index_report(config).unwrap();assert_eq!(code, 0);assert_eq!(fs::read_to_string(&count).unwrap().lines().count(), 2);- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 1);+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 1);assert_eq!(file_done_count(state_dir.path()), 1);assert!(crate::extract::pdf::corrupted_replay_reservation_was_dropped());let reuse = report.unwrap()["pdf_extraction_reuse"].clone();diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rsindex 0e4970e9c..6db75e62a 100644--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs@@ -389,7 +389,9 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {url: url.to_owned(),api_key: None,workers: 1,+ scan_workers: 1,pdf_workers: 1,+ resource_notes: Vec::new(),pdf_timeout_secs: 30,bulk_mb: 1,bulk_timeout_secs: 30,@@ -406,6 +408,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {dry_run: false,json: false,quiet: true,+ progress: crate::progress::ProgressMode::None,+ progress_interval: None,}}@@ -531,6 +535,7 @@ fn initial_generation_and_noop_are_end_to_end_idempotent() {&root.to_string_lossy(),&endpoint.url,"incremental-http",+ false,).unwrap();assert!(@@ -1181,3 +1186,91 @@ fn a_junk_record_is_counted_into_the_generation_and_never_fatal() {assert_eq!(file_doc["junk"], 1);assert_eq!(file_doc["records"], 40);}++/// Where this branch and #241 meet.+///+/// The generated `--no-graph` route returns from inside `run_index_report`,+/// several hundred lines before the legacy `pr.finish` at the bottom of the+/// function. `Ticker::drop` closes an unfinished stream with `ok=false`, so a+/// generated run that merely returned would print an *aborted* terminal line+/// immediately after committing a generation successfully — a silent+/// contradiction between two changes that each looked right on its own.+///+/// This pins all three properties on the generated path: the phase-A scan is+/// narrated (it is a per-file, minutes-long phase here exactly as on the legacy+/// path), the stream closes with a truthful terminal line, and `--progress+/// none` on the same path stays completely silent.+#[test]+fn a_generated_run_narrates_its_scan_and_closes_its_own_stream() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap();+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();+ let _sink_guard = crate::progress::SINK_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ for n in 0..4 {+ fs::write(+ corpus.path().join(format!("rows-{n}.csv")),+ format!("id,value\n{n}0,alpha\n{n}1,beta\n"),+ )+ .unwrap();+ }+ let endpoint = HttpEndpoint::start();++ // Generation 1: the initial commit, through the real progress surface.+ let mut loud = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);+ loud.quiet = false;+ loud.progress = crate::progress::ProgressMode::Plain;+ loud.progress_interval = Some(std::time::Duration::from_secs(1));+ let buffer = Arc::new(Mutex::new(Vec::new()));+ let code = {+ let _sink = crate::progress::install_test_sink(&buffer);+ run_index(loud.clone()).unwrap()+ };+ assert_eq!(code, 0);+ let stream = String::from_utf8(buffer.lock().unwrap().clone()).unwrap();+ let done = stream+ .lines()+ .find(|line| line.starts_with("xerj-done "))+ .unwrap_or_else(|| panic!("a generated run must close its own stream:\n{stream}"));+ assert!(+ done.contains("ok=true") && done.contains("exit=0"),+ "a committed generation is not an aborted run: {done}"+ );+ assert!(done.contains("reason=completed"), "{done}");++ // Generation 2: the reconcile branch, which runs its own phase-A scan.+ fs::write(corpus.path().join("rows-0.csv"), "id,value\n00,CHANGED\n").unwrap();+ let buffer = Arc::new(Mutex::new(Vec::new()));+ let code = {+ let _sink = crate::progress::install_test_sink(&buffer);+ run_index(loud).unwrap()+ };+ assert_eq!(code, 0);+ let stream = String::from_utf8(buffer.lock().unwrap().clone()).unwrap();+ assert!(+ stream+ .lines()+ .any(|line| line.starts_with("xerj-progress ") && line.contains("phase=scan")),+ "the reconcile projection re-scans every file and must say so:\n{stream}"+ );+ let done = stream+ .lines()+ .find(|line| line.starts_with("xerj-done "))+ .unwrap_or_else(|| panic!("a reconciled generation must close its own stream:\n{stream}"));+ assert!(done.contains("ok=true"), "{done}");++ // The other half of the contract on the same path: silence means silence.+ fs::write(corpus.path().join("rows-1.csv"), "id,value\n10,CHANGED\n").unwrap();+ let quiet = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);+ assert_eq!(quiet.progress, crate::progress::ProgressMode::None);+ let buffer = Arc::new(Mutex::new(Vec::new()));+ {+ let _sink = crate::progress::install_test_sink(&buffer);+ assert_eq!(run_index(quiet).unwrap(), 0);+ }+ assert!(+ buffer.lock().unwrap().is_empty(),+ "--progress none asked for nothing: {}",+ String::from_utf8_lossy(&buffer.lock().unwrap())+ );+}diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 414d62e00..205487df3 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -28,6 +28,7 @@ mod sync_executor;pub mod walk;use anyhow::{Context, Result};+use serde::Serialize;use serde_json::{json, Map, Value};use std::collections::HashMap;use std::io::{BufRead, BufReader, Seek, Write};@@ -222,6 +223,33 @@ fn generated_exit_code(summary: &Value) -> i32 {}}+/// Terminal progress line for a run that ends on the generated `--no-graph`+/// path.+///+/// Every exit closes the stream itself (#241). `Ticker::drop` reports+/// `ok=false` for a run that never called `finish`, so a generated run that+/// merely returned would print an aborted line straight after a successful+/// commit — the one place the two landed changes could contradict each other.+/// Exit 3 is success ("completed, some input was unusable"), so it is spelled+/// out in words rather than left as a bare number.+fn finish_generated_progress(pr: &Progress, code: i32, summary: &Value) {+ let count = |field: &str| summary.get(field).and_then(Value::as_u64).unwrap_or(0);+ pr.finish(+ true,+ code,+ if code == 3 {+ "completed-with-junk"+ } else {+ "completed"+ },+ &[+ ("files", count("files_indexed")),+ ("records", count("records_total")),+ ("generation", count("generation")),+ ],+ );+}+fn begin_non_graph_generation(es: &Es,journal: &mut state::Journal,@@ -391,6 +419,13 @@ struct CliErrorRoute {fn route_cli_error(error: &anyhow::Error, json_output: bool) -> CliErrorRoute {if json_output {+ // Two distinct refusals reach the JSON route, and both must keep it.+ // `UnsafeFreshGenerationError` guards durable *generation* state on the+ // `--no-graph` path; `UnsupportedInventoryDeltaError` (#254) guards the+ // legacy graph-enabled path against a rerun that would strand live+ // documents. Neither supersedes the other — they protect different+ // destinations — so routing only one would silently drop the other's+ // machine-readable rendering back to a prose stderr line.if let Some(unsafe_fresh) = error.downcast_ref::<UnsafeFreshGenerationError>() {return CliErrorRoute {exit_code: 1,@@ -398,6 +433,13 @@ fn route_cli_error(error: &anyhow::Error, json_output: bool) -> CliErrorRoute {stderr: None,};}+ if let Some(delta) = error.downcast_ref::<UnsupportedInventoryDeltaError>() {+ return CliErrorRoute {+ exit_code: 1,+ stdout: Some(delta.to_json().to_string()),+ stderr: None,+ };+ }}CliErrorRoute {exit_code: 1,@@ -1538,6 +1580,284 @@ impl std::fmt::Display for UnsafeFreshGenerationError {impl std::error::Error for UnsafeFreshGenerationError {}+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]+struct InventoryDeltaEntry {+ file_key: String,+ path: String,+}++#[derive(Debug, Clone, PartialEq, Eq, Serialize)]+struct UnsupportedInventoryDelta {+ added_content_groups: Vec<InventoryDeltaEntry>,+ vanished_content_groups: Vec<InventoryDeltaEntry>,+}++/// Everything the refusal needs to name the destination it is protecting.+/// Collected at the gate so the message can list the exact indices and state+/// directory an operator has to act on, instead of describing them abstractly.+#[derive(Debug, Clone, PartialEq, Eq)]+struct RefusalTargets {+ state_dir: String,+ data_indices: Vec<String>,+ edges_index: Option<String>,+}++impl RefusalTargets {+ fn describe(cfg: &IndexCfg, state_dir: &Path, plan: &Plan) -> Self {+ let mut data_indices: Vec<String> = plan.datasets.iter().map(|d| d.index.clone()).collect();+ data_indices.sort();+ data_indices.dedup();+ let edges_index = (!cfg.no_graph).then(|| {+ let brain = cfg+ .brain+ .clone()+ .unwrap_or_else(|| derive_brain_name(&cfg.root));+ detect::edges_index_name(&brain)+ });+ Self {+ state_dir: state_dir.display().to_string(),+ data_indices,+ edges_index,+ }+ }++ fn indices_phrase(&self) -> String {+ if self.data_indices.is_empty() {+ "the indices this plan publishes".to_string()+ } else {+ self.data_indices.join(", ")+ }+ }++ fn edges_note(&self) -> String {+ match &self.edges_index {+ Some(edges) => format!(+ " Graph edges taught by the removed file(s) stay live in {edges} until that \+ index is deleted too."+ ),+ None => String::new(),+ }+ }+}++/// The #195 zero-live verification failure, as a type rather than a string.+///+/// The journal claims records were made visible; the destination answers with+/// none. That is always a failure, but not always the *same* failure: for a+/// composing caller like `xerj brain` a resume journal that outlived a wiped+/// data directory has a specific, executable recovery (`--fresh` in place),+/// while a write-blocked or unreachable server does not. Only a typed error+/// lets the caller tell them apart — `anyhow::bail!` forces it to either+/// string-match the prose or reprint it verbatim.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct ZeroLiveVerificationError {+ /// Records the resume journal says were published.+ pub journal_records: u64,+ /// Completed files those records came from.+ pub files_done_journaled: usize,+ /// Dataset indices that answered with zero live documents.+ pub dataset_indices: usize,+}++impl std::fmt::Display for ZeroLiveVerificationError {+ fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(+ formatter,+ "autoindex verification failed: the resume journal records {} \+ record(s) from {} completed file(s), but 0 documents are \+ live across the {} dataset index(es). The indices exist and look healthy but \+ hold nothing — a server-side write rejection (e.g. a disk flood-stage or \+ index write block), deleted indices, or an unreadable server can all cause \+ this. Fix the server-side condition, then rerun with --fresh to re-index \+ from scratch",+ self.journal_records, self.files_done_journaled, self.dataset_indices+ )+ }+}++impl std::error::Error for ZeroLiveVerificationError {}++#[derive(Debug)]+struct UnsupportedInventoryDeltaError {+ delta: UnsupportedInventoryDelta,+ targets: RefusalTargets,+}++impl UnsupportedInventoryDeltaError {+ fn to_json(&self) -> Value {+ json!({+ "schema": "xerj.autoindex.unsupported_sync_delta.v1",+ "status": "error",+ "error": "unsupported_content_group_removal",+ "message": "this attempt made no remote mutations. Files that were indexed under this resume plan no longer exist in the folder, and their documents are still live in the destination; removing files from an indexed folder is not reconciled yet",+ "vanished_content_groups": self.delta.vanished_content_groups,+ // Context, not the reason for the refusal: a rerun over a frozen+ // plan does not index files added after the plan was frozen.+ "added_content_groups": self.delta.added_content_groups,+ "recovery": {+ "restore_removed_files": "put the listed file(s) back and rerun; every other file keeps its resume state",+ "rebuild_in_place": format!(+ "delete the indices this plan publishes ({}) and the state directory {}, then rerun. This re-extracts and re-embeds the whole corpus.{}",+ self.targets.indices_phrase(),+ self.targets.state_dir,+ self.targets.edges_note().trim_start_matches(' ')+ ),+ "rebuild_isolated": "index with a new --state-dir, new --prefix, and (when graph detection is enabled) new --brain; alternatively add --no-graph. Validate the isolated target before switching readers, then clean the old one",+ "fresh_warning": "--fresh re-extracts the current folder in place and does pick up added and changed files, but it never deletes documents already published for removed files, so it is refused here"+ }+ })+ }+}++impl std::fmt::Display for UnsupportedInventoryDeltaError {+ fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ // Capped like the duplicate and unplanned-file listings above; see+ // REFUSAL_LIST_CAP. The machine-readable `--json` rendering keeps the+ // full lists, so nothing is lost — only the prose is bounded.+ let render = |entries: &[InventoryDeltaEntry]| {+ let mut rendered = entries+ .iter()+ .take(REFUSAL_LIST_CAP)+ .map(|entry| format!("{} ({})", entry.path, entry.file_key))+ .collect::<Vec<_>>()+ .join(", ");+ let remaining = entries.len().saturating_sub(REFUSAL_LIST_CAP);+ if remaining > 0 {+ rendered.push_str(&format!(", … and {remaining} more"));+ }+ rendered+ };+ write!(+ formatter,+ "{} file(s) indexed under this resume plan no longer exist in the folder, and their \+ documents are still live in the destination. Removing files from an indexed folder \+ is not reconciled yet, so this attempt made no remote mutations — no documents, \+ aliases, graph edges or catalog entries were written. Removed content groups [{}].",+ self.delta.vanished_content_groups.len(),+ render(&self.delta.vanished_content_groups)+ )?;+ if !self.delta.added_content_groups.is_empty() {+ write!(+ formatter,+ " Also present but not in the frozen resume plan, so not indexed by this run \+ either [{}].",+ render(&self.delta.added_content_groups)+ )?;+ }+ write!(+ formatter,+ " Recovery, cheapest first: (1) restore the removed file(s) and rerun — every other \+ file keeps its resume state; (2) rebuild in place by deleting the indices this plan \+ publishes ({}) and the state directory {}, then rerunning — this re-extracts and \+ re-embeds the whole corpus.{} (3) rebuild isolated with a new --state-dir, a new \+ --prefix and, when graph detection is enabled, a new --brain (or --no-graph), \+ validate it, switch readers, then clean the old target. `--fresh` picks up added \+ and changed files in place but never deletes documents for removed files, so it is \+ refused here too",+ self.targets.indices_phrase(),+ self.targets.state_dir,+ self.targets.edges_note()+ )+ }+}++impl std::error::Error for UnsupportedInventoryDeltaError {}++impl UnsupportedInventoryDelta {+ fn between(files: &[walk::FileEntry], keys: &[String], plan: &Plan) -> Self {+ let current_keys: std::collections::HashSet<&str> = keys+ .iter()+ .filter(|key| !key.is_empty())+ .map(String::as_str)+ .collect();+ let durable_keys: std::collections::HashSet<&str> = plan+ .files+ .keys()+ .map(String::as_str)+ .chain(plan.junk_files.iter().map(|junk| junk.file_key.as_str()))+ .collect();+ // Path identity, not just content identity. An in-place EDIT gives the+ // same file a new content key, so a key-only comparison reads the+ // superseded key as vanished and the new one as added — the SAME path+ // listed as both removed and added. That is a replacement, and the file+ // is still there to be republished, so it must never be refused.+ // Ordinary resume already reaches this conclusion by mapping each file+ // onto its planned key (`select_resume_plan_keys`); doing it here makes+ // `--fresh`, which has no plan to map through, agree — without making+ // the gate more fatal than the open it precedes.+ let current_rels: std::collections::HashSet<&str> =+ files.iter().map(|file| file.rel.as_str()).collect();+ let current_path_ids: std::collections::HashSet<&str> = files+ .iter()+ .map(|file| file.rel_id.as_str())+ .filter(|id| !id.is_empty())+ .collect();+ let path_survives = |assignment: &FileAssignment| {+ current_rels.contains(assignment.rel.as_str())+ || (!assignment.path_id.is_empty()+ && current_path_ids.contains(assignment.path_id.as_str()))+ };++ let mut added_content_groups: Vec<InventoryDeltaEntry> = files+ .iter()+ .zip(keys)+ .filter(|(_, key)| !key.is_empty() && !durable_keys.contains(key.as_str()))+ .map(|(file, key)| InventoryDeltaEntry {+ file_key: key.clone(),+ path: file.rel.clone(),+ })+ .collect();+ let mut vanished_content_groups: Vec<InventoryDeltaEntry> = plan+ .files+ .iter()+ .filter(|(key, assignment)| {+ !current_keys.contains(key.as_str()) && !path_survives(assignment)+ })+ .map(|(key, assignment)| InventoryDeltaEntry {+ file_key: key.clone(),+ path: assignment.rel.clone(),+ })+ .collect();+ // Deliberately NOT extended with `plan.junk_files`. A junk/skipped file+ // published no documents, no aliases and no graph edges — its entire+ // live footprint is one `file:{key}` catalog row, and the stale+ // junk-catalog sweep below (#238) deletes that row before dropping the+ // plan entry. Removing it therefore strands nothing, so refusing the+ // rerun would block a case the pipeline now handles completely. Junk+ // keys stay in `durable_keys` above: an unchanged skipped file is still+ // not an addition.++ let stable_order = |left: &InventoryDeltaEntry, right: &InventoryDeltaEntry| {+ left.path+ .cmp(&right.path)+ .then_with(|| left.file_key.cmp(&right.file_key))+ };+ added_content_groups.sort_by(stable_order);+ vanished_content_groups.sort_by(stable_order);+ Self {+ added_content_groups,+ vanished_content_groups,+ }+ }++ /// A rerun is refused only when a content group the plan published has+ /// vanished from the folder: those documents stay live and searchable with+ /// no source file behind them, and nothing in this pipeline removes them.+ /// Additions are not refused — they are skipped by the frozen plan exactly+ /// as before and `--fresh` rebuilds the plan in place to include them.+ fn refuses(&self) -> bool {+ !self.vanished_content_groups.is_empty()+ }++ fn into_error(self, targets: RefusalTargets) -> anyhow::Error {+ UnsupportedInventoryDeltaError {+ delta: self,+ targets,+ }+ .into()+ }+}+fn alias_keys_to_reindex(previous: &[state::DuplicateFile],current: &[state::DuplicateFile],@@ -1838,7 +2158,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// Acquire state authority before discovery as well as hashing. A waiter// must never classify a path snapshot taken while another owner was// publishing or replacing the durable plan.- let preflight = state::Journal::preflight(&state_dir, &root_str, &cfg.url, &cfg.prefix)?;+ let preflight =+ state::Journal::preflight(&state_dir, &root_str, &cfg.url, &cfg.prefix, cfg.fresh)?;let genesis_recovery = preflight.committed_manifest.as_ref()@@ -1872,6 +2193,18 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {}.into());}+ if let Some(reason) = preflight.unreadable_plan.as_deref() {+ // Only reachable under --fresh; without it the preflight would have+ // returned this as an error. It goes through the progress surface, not+ // a raw eprintln!, so `--progress none` (which `--quiet` selects) stays+ // silent and `--progress json` stays one parseable stream (#241).+ pr.note(&format!(+ "autoindex: --fresh: the durable resume plan in {} could not be read ({reason}); \+ rebuilding it from the current folder. Documents published for files that are no \+ longer present cannot be identified from an unreadable plan and are not deleted.",+ state_dir.join("journal.ndjson").display()+ ));+ }// `--dry-run` has to be decided *before* the generated branches, not after// them. Both of those branches publish and commit, and both return before// control ever reaches the legacy `cfg.dry_run` check further down — so on@@ -1884,11 +2217,16 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {if cfg.dry_run {if let Some(pending) = &preflight.pending_sync {println!("{}", serde_json::to_string_pretty(&pending.desired.plan)?);- eprintln!(+ // stdout is the RESULT, stderr is PROGRESS — so this explanation+ // goes through the progress surface, never a bare `eprintln!`+ // that `--progress none` could not silence and `--progress json`+ // could not parse (#241).+ pr.note(&format!("(dry run — nothing indexed; corpus generation {} is already sealed and pending \replay from its own durable snapshot. Re-run without --dry-run to finish it.)",pending.desired.generation- );+ ));+ pr.finish(true, 0, "dry-run", &[]);return Ok((0, None));}}@@ -1921,7 +2259,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// `--progress json` stays one parseable stream (#241).pr.note("autoindex: resumed and committed pending corpus generation from durable source");let summary = finish_generated_run(&es, &mut journal, &cfg)?;- return Ok((generated_exit_code(&summary), Some(summary)));+ let code = generated_exit_code(&summary);+ finish_generated_progress(&pr, code, &summary);+ return Ok((code, Some(summary)));}// Totals are unknown until the walk returns, so this phase honestly// reports `pct=unknown` and proves liveness with the clock alone.@@ -2036,7 +2376,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {}}let summary = finish_generated_run(&es, &mut journal, &cfg)?;- return Ok((generated_exit_code(&summary), Some(summary)));+ let code = generated_exit_code(&summary);+ finish_generated_progress(&pr, code, &summary);+ return Ok((code, Some(summary)));}begin_non_graph_generation(&es,@@ -2050,7 +2392,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;let summary = finish_generated_run(&es, &mut journal, &cfg)?;- return Ok((generated_exit_code(&summary), Some(summary)));+ let code = generated_exit_code(&summary);+ finish_generated_progress(&pr, code, &summary);+ return Ok((code, Some(summary)));}if cfg.no_graph && preflight.plan.is_some() && !genesis_recovery {let replacement_state = state_dir.with_extension("generation-v1");@@ -2102,6 +2446,32 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {rebuild_argv);}+ if let Some(prior_plan) = preflight.plan.as_ref() {+ let comparison_keys = if cfg.fresh {+ inventory.keys.clone()+ } else {+ // Ordinary resume preserves planned-key identity for supported+ // same-path replacement and legacy plans.+ select_resume_plan_keys(+ &inventory.files,+ &inventory.keys,+ prior_plan,+ &state_dir.join("journal.ndjson"),+ )?+ .into_iter()+ .zip(inventory.keys.iter())+ .map(|(planned, current)| planned.unwrap_or_else(|| current.clone()))+ .collect()+ };+ // `--fresh` is checked here too: discarding the plan does not delete+ // the documents already published for a file that is now gone, so a+ // removal is unsafe in place whether or not the plan is kept.+ let delta =+ UnsupportedInventoryDelta::between(&inventory.files, &comparison_keys, prior_plan);+ if delta.refuses() {+ return Err(delta.into_error(RefusalTargets::describe(&cfg, &state_dir, prior_plan)));+ }+ }let mut journal = state::Journal::open_after_preflight(preflight,&root_str,@@ -2340,6 +2710,22 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {plan};+ // Second gate, after key selection has settled: fail before index+ // creation/mapping, replacement intent, graph invalidation,+ // delete-by-query, bulk publication, refresh, or catalog writes when a+ // canonical content group the plan published is gone from the folder.+ // Continuing would leave those documents searchable with no source file.+ // Additions, same-path replacement and crash repair remain supported.+ //+ // It runs before the #238 junk sweep below on purpose: a refused rerun+ // must compute nothing and mutate nothing.+ if resumed_with_plan {+ let delta = UnsupportedInventoryDelta::between(&files, &keys, &plan);+ if delta.refuses() {+ return Err(delta.into_error(RefusalTargets::describe(&cfg, &state_dir, &plan)));+ }+ }+// ── stale junk-catalog sweep (#238) ──────────────────────────────────//// A junk/skipped file is never indexed and never enters the graph corpus@@ -2391,7 +2777,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;let summary = finish_generated_run(&es, &mut journal, &cfg)?;- return Ok((generated_exit_code(&summary), Some(summary)));+ let code = generated_exit_code(&summary);+ finish_generated_progress(&pr, code, &summary);+ return Ok((code, Some(summary)));}if plan@@ -2528,9 +2916,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {rel: files[i].rel.clone(),format: "unknown".into(),status: "skipped".into(),- reason: "not in the frozen resume plan; no destination change was made. For an \- exact rebuild use a new --state-dir, a new --prefix, and, when graph \- detection is enabled, a new --brain (or use --no-graph)"+ reason: "not in the frozen resume plan, so nothing was published for it. Re-run \+ with --fresh to rebuild the plan in place and include it (ids stay \+ idempotent), or index it under a new --state-dir and --prefix".into(),bytes: files[i].size,});@@ -4289,6 +4677,215 @@ mod unsafe_fresh_generation_tests {}}+#[cfg(test)]+mod inventory_delta_tests {+ use super::*;+ use std::path::PathBuf;++ fn file(path: &str) -> walk::FileEntry {+ walk::FileEntry {+ path: PathBuf::from(path),+ rel: path.to_owned(),+ rel_id: format!("id:{path}"),+ is_symlink: false,+ size: 1,+ }+ }++ fn assignment(path: &str) -> FileAssignment {+ FileAssignment {+ rel: path.to_owned(),+ path_id: format!("id:{path}"),+ family: "csv".into(),+ gzip: false,+ content_digest: Some(format!("digest:{path}")),+ assignments: vec![(None, "rows".into())],+ as_document: false,+ is_symlink: Some(false),+ }+ }++ fn targets() -> RefusalTargets {+ RefusalTargets {+ state_dir: "/state".into(),+ data_indices: vec!["ax-rows".into()],+ edges_index: Some(".xerj-memory-corpus-edges".into()),+ }+ }++ #[test]+ fn only_a_removed_content_group_refuses_a_rerun() {+ let mut plan = Plan::default();+ plan.files.insert("keep".into(), assignment("keep.csv"));+ assert!(+ !UnsupportedInventoryDelta::between(&[file("keep.csv")], &["keep".into()], &plan)+ .refuses()+ );+ // An added file is skipped by the frozen plan, not a refusal: the+ // documented rerun-then---fresh workflow has to keep working.+ let added = UnsupportedInventoryDelta::between(+ &[file("keep.csv"), file("new.csv")],+ &["keep".into(), "new".into()],+ &plan,+ );+ assert_eq!(added.added_content_groups.len(), 1);+ assert!(!added.refuses(), "an addition alone must not fail the run");+ // A removal leaves live documents with no source file behind them.+ assert!(UnsupportedInventoryDelta::between(&[], &[], &plan).refuses());+ }++ #[test]+ fn classifier_sorts_added_and_vanished_groups() {+ let mut plan = Plan::default();+ plan.files.insert("keep".into(), assignment("keep.csv"));+ plan.junk_files.push(JunkFile {+ file_key: "junk".into(),+ rel: "broken.pdf".into(),+ format: "pdf".into(),+ status: "junk".into(),+ reason: "fixture".into(),+ bytes: 1,+ });+ assert!(+ !UnsupportedInventoryDelta::between(+ &[file("keep.csv"), file("broken.pdf")],+ &["keep".into(), "junk".into()],+ &plan,+ )+ .refuses(),+ "unchanged durable junk is neither added nor vanished"+ );++ plan.files.insert("old-z".into(), assignment("z-old.csv"));+ plan.files.insert("old-a".into(), assignment("a-old.csv"));+ let delta = UnsupportedInventoryDelta::between(+ &[+ file("keep.csv"),+ file("broken.pdf"),+ file("z-new.csv"),+ file("m-new.csv"),+ ],+ &["keep".into(), "junk".into(), "new-z".into(), "new-m".into()],+ &plan,+ );+ assert_eq!(+ delta+ .added_content_groups+ .iter()+ .map(|entry| entry.path.as_str())+ .collect::<Vec<_>>(),+ ["m-new.csv", "z-new.csv"]+ );+ assert_eq!(+ delta+ .vanished_content_groups+ .iter()+ .map(|entry| entry.path.as_str())+ .collect::<Vec<_>>(),+ ["a-old.csv", "z-old.csv"]+ );+ }++ #[test]+ fn refusal_names_the_removed_files_and_every_recovery_route() {+ let mut plan = Plan::default();+ plan.files.insert("gone".into(), assignment("gone.csv"));+ let error = UnsupportedInventoryDelta::between(&[], &[], &plan).into_error(targets());+ let message = format!("{error:#}");+ assert!(message.contains("gone.csv"), "{message}");+ assert!(+ message.contains("no longer exist in the folder"),+ "{message}"+ );+ assert!(message.contains("made no remote mutations"), "{message}");+ assert!(message.contains("restore the removed file(s)"), "{message}");+ assert!(message.contains("ax-rows"), "{message}");+ assert!(message.contains("/state"), "{message}");+ assert!(message.contains(".xerj-memory-corpus-edges"), "{message}");+ assert!(message.contains("new --state-dir"), "{message}");+ assert!(message.contains("`--fresh`"), "{message}");+ }++ /// The prose refusal is bounded by REFUSAL_LIST_CAP, not by the corpus: an+ /// unmounted bind mount under an indexed root vanishes every group at once,+ /// and rendering 82k entries into one String is megabytes of stderr in the+ /// one path whose job is to be read. `--json` still carries all of them.+ #[test]+ fn refusal_prose_caps_its_listings_while_json_keeps_every_entry() {+ let mut plan = Plan::default();+ let total = REFUSAL_LIST_CAP * 3;+ for i in 0..total {+ plan.files.insert(+ format!("gone{i:03}"),+ assignment(&format!("gone{i:03}.csv")),+ );+ }+ let error = UnsupportedInventoryDelta::between(&[], &[], &plan).into_error(targets());+ let message = format!("{error:#}");+ assert!(message.contains("gone000.csv"), "{message}");+ assert!(+ message.contains(&format!(", … and {} more", total - REFUSAL_LIST_CAP)),+ "{message}"+ );+ // The tail past the cap must not be rendered at all, not merely elided.+ let last = format!("gone{:03}.csv", total - 1);+ assert!(!message.contains(&last), "{message}");+ assert_eq!(+ message.matches(".csv (").count(),+ REFUSAL_LIST_CAP,+ "{message}"+ );++ let stdout = route_cli_error(&error, true)+ .stdout+ .expect("the typed refusal renders as JSON under --json");+ let value: Value = serde_json::from_str(&stdout).unwrap();+ assert_eq!(+ value["vanished_content_groups"].as_array().unwrap().len(),+ total+ );+ }++ #[test]+ fn refusal_prose_below_the_cap_has_no_more_tail() {+ let mut plan = Plan::default();+ plan.files.insert("gone".into(), assignment("gone.csv"));+ let error = UnsupportedInventoryDelta::between(&[], &[], &plan).into_error(targets());+ let message = format!("{error:#}");+ assert!(!message.contains("… and "), "{message}");+ }++ #[test]+ fn cli_error_routing_separates_typed_json_from_unrelated_human_errors() {+ let typed = UnsupportedInventoryDelta {+ added_content_groups: Vec::new(),+ vanished_content_groups: vec![InventoryDeltaEntry {+ file_key: "key".into(),+ path: "gone.csv".into(),+ }],+ }+ .into_error(targets());+ let route = route_cli_error(&typed, true);+ assert_eq!(route.exit_code, 1);+ assert!(route.stderr.is_none());+ let stdout = route.stdout.unwrap();+ let value: Value = serde_json::from_str(&stdout).unwrap();+ assert_eq!(value["schema"], "xerj.autoindex.unsupported_sync_delta.v1");+ assert_eq!(value["error"], "unsupported_content_group_removal");+ assert_eq!(value["vanished_content_groups"][0]["path"], "gone.csv");+ assert!(value["recovery"]["rebuild_in_place"]+ .as_str()+ .unwrap()+ .contains("ax-rows"));++ let unrelated = anyhow::anyhow!("endpoint unavailable");+ let route = route_cli_error(&unrelated, true);+ assert_eq!(route.exit_code, 1);+ assert!(route.stdout.is_none());+ assert_eq!(route.stderr.as_deref(), Some("error: endpoint unavailable"));+ }+}+#[cfg(test)]mod duplicate_integration_tests {use super::*;diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rsindex eee92c704..8cfd9150f 100644--- a/engine/crates/xerj-autoindex/src/reconcile_plan.rs+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs@@ -468,6 +468,11 @@ mod tests {records: 1,}],junk: None,+ // Reconciliation re-samples from the source tree and publishes+ // through the sealed snapshot, so it never carries a run-local+ // PDF artifact (#248 is a phase A→B accelerator).+ pdf_spool: None,+ pdf_spool_fallbacks: Vec::new(),}}diff --git a/engine/crates/xerj-autoindex/src/state.rs b/engine/crates/xerj-autoindex/src/state.rsindex a089c101c..5bd21e261 100644--- a/engine/crates/xerj-autoindex/src/state.rs+++ b/engine/crates/xerj-autoindex/src/state.rs@@ -202,21 +202,6 @@ mod tests {}}- #[test]- fn preflight_holds_the_same_exclusive_lock_through_authoritative_open() {- let dir = tempfile::tempdir().unwrap();- let preflight = Journal::preflight(dir.path(), "root", "url", "prefix").unwrap();- let error = Journal::open(dir.path(), "root", "url", "prefix", 300, false)- .err()- .expect("a second opener must not pass the held preflight lock");- assert!(format!("{error:#}").contains("already in use"));-- let journal =- Journal::open_after_preflight(preflight, "root", "url", "prefix", 300, false).unwrap();- drop(journal);- assert!(Journal::open(dir.path(), "root", "url", "prefix", 300, false).is_ok());- }-/// The preflight must never be more fatal than the open it precedes./// `open_after_preflight` deletes the journal unread when `--fresh` is set,/// so a preflight that hard-errors while parsing that journal would make@@ -534,10 +519,14 @@ fn read_plan_for_preflight("journal corruption at byte {record_start} in {}: malformed \newline-terminated record. Refusing to discard later records. Restore \the journal from a backup, or truncate it to exactly {record_start} \- bytes to keep every completion recorded before the corruption. For an \- isolated rebuild use a new --state-dir, new --prefix, and new --brain \- when graph detection is enabled (or --no-graph); explicitly validate \- and clean the shared catalog and old target",+ bytes to keep every completion recorded before the corruption. Deleting \+ the whole journal (or rerunning with --fresh) also recovers, but \+ re-extracts and re-embeds the entire corpus — and it discards any \+ durable generation records in this journal along with it, which is \+ why --fresh is reachable here at all. For an isolated rebuild use a \+ new --state-dir, new --prefix, and new --brain when graph detection \+ is enabled (or --no-graph); explicitly validate and clean the shared \+ catalog and old target",path.display())})?;@@ -608,14 +597,25 @@ pub struct JournalPreflight {pub pending_sync: Option<crate::sync::PendingSync>,pub legacy_migration_reasons: Vec<String>,pub journal_exists: bool,+ /// Set only under `--fresh`, when the durable plan could not be read and+ /// the run is about to discard the journal anyway. The caller owns the+ /// decision to print it (autoindex honours `--quiet`).+ pub unreadable_plan: Option<String>,}impl Journal {+ /// `fresh` is not a formality here. `open_after_preflight` removes the+ /// journal before replaying it when `--fresh` is set, so nothing inside a+ /// journal can be fatal to that path — and a preflight that hard-errors on+ /// a corrupt record or a root/url/prefix mismatch would make `--fresh`+ /// unreachable in exactly the cases whose error text recommends it. The+ /// preflight must therefore never be more fatal than the open it precedes.pub fn preflight(state_dir: &Path,root: &str,url: &str,prefix: &str,+ fresh: bool,) -> Result<JournalPreflight> {std::fs::create_dir_all(state_dir).with_context(|| format!("create state dir {}", state_dir.display()))?;@@ -635,8 +635,26 @@ impl Journal {})?;let journal_path = state_dir.join("journal.ndjson");let journal_exists = journal_path.exists();- let (plan, committed_manifest, pending_sync, legacy_migration_reasons) =- read_plan_for_preflight(&journal_path, root, url, prefix)?;+ let (plan, committed_manifest, pending_sync, legacy_migration_reasons, unreadable_plan) =+ match read_plan_for_preflight(&journal_path, root, url, prefix) {+ Ok((plan, committed, pending, reasons)) => {+ (plan, committed, pending, reasons, None)+ }+ // The journal is about to be deleted unread. Losing the plan here+ // costs the removed-file gate its comparison basis, which is+ // exactly what a full in-place rebuild accepts; the alternative is+ // a state directory that no supported flag can recover.+ //+ // Every generated record is discarded with it, so the durable+ // generation this run might otherwise have refused to overwrite is+ // reported as absent rather than half-parsed. That is what keeps+ // `--fresh` genuinely reachable here: the caller's generation+ // refusal reads `committed_manifest`/`pending_sync`, and firing it+ // off a partially-replayed journal would contradict the recovery+ // this very error recommends.+ Err(error) if fresh => (None, None, None, Vec::new(), Some(format!("{error:#}"))),+ Err(error) => return Err(error),+ };Ok(JournalPreflight {state_dir: state_dir.to_owned(),state_lock,@@ -645,6 +663,7 @@ impl Journal {pending_sync,legacy_migration_reasons,journal_exists,+ unreadable_plan,})}@@ -656,7 +675,7 @@ impl Journal {bulk_timeout_secs: u64,fresh: bool,) -> Result<Journal> {- let preflight = Self::preflight(state_dir, root, url, prefix)?;+ let preflight = Self::preflight(state_dir, root, url, prefix, fresh)?;Self::open_after_preflight(preflight, root, url, prefix, bulk_timeout_secs, fresh)}@@ -732,10 +751,9 @@ impl Journal {anyhow::bail!("journal at {} was created for root={jr} url={ju} prefix={jp}; \current run has root={root} url={url} prefix={prefix}. \- Refusing to discard history under the same destination. For an \- isolated rebuild use a new --state-dir, new --prefix, and new \- --brain when graph detection is enabled (or --no-graph); \- explicitly validate and clean the shared catalog and old target.",+ Use --state-dir for separate state, or --fresh to rebuild the \+ plan in place — note that --fresh never deletes documents \+ already published under the other root, url or prefix.",jpath.display());}@@ -1452,7 +1470,7 @@ mod sync_journal_tests {.err().unwrap();assert!(format!("{error:#}").contains("while a sync is pending"));- let preflight = Journal::preflight(dir.path(), "root", "url", "prefix")+ let preflight = Journal::preflight(dir.path(), "root", "url", "prefix", false).err().unwrap();assert!(format!("{preflight:#}").contains("while a sync is pending"));@@ -1530,7 +1548,7 @@ mod sync_journal_tests {let mut journal = journal_with_plan(dir.path());journal.sync_begin(&pending(&journal)).unwrap();drop(journal);- let preflight = Journal::preflight(dir.path(), "root", "url", "prefix").unwrap();+ let preflight = Journal::preflight(dir.path(), "root", "url", "prefix", false).unwrap();assert_eq!(preflight.committed_manifest.as_ref().unwrap().generation, 0);assert_eq!(preflight.pending_sync.as_ref().unwrap().tx_id, "sync-1");drop(preflight);@@ -1623,6 +1641,7 @@ mod sync_journal_tests {records: 1,junk: 0,bytes: 10,+ dropped_by_dataset: std::collections::HashMap::new(),generation: None,}).unwrap();diff --git a/engine/crates/xerj-autoindex/src/sync_executor.rs b/engine/crates/xerj-autoindex/src/sync_executor.rsindex 2ee8f6543..1a621a83c 100644--- a/engine/crates/xerj-autoindex/src/sync_executor.rs+++ b/engine/crates/xerj-autoindex/src/sync_executor.rs@@ -1721,7 +1721,7 @@ mod tests {use std::collections::BTreeMap;fn inventory(root: &Path) -> Inventory {- crate::content::resolve(walk::walk(root, false).unwrap()).unwrap()+ crate::content::resolve_reporting(walk::walk(root, false).unwrap(), &|_| {}).unwrap()}fn plan_for(inventory: &Inventory) -> Plan {@@ -2045,8 +2045,11 @@ mod tests {fn prepared_snapshot_budget_aborts_and_removes_staging() {let corpus = tempfile::tempdir().unwrap();std::fs::write(corpus.path().join("a.txt"), "budgeted source bytes").unwrap();- let inventory =- crate::content::resolve(crate::walk::walk(corpus.path(), false).unwrap()).unwrap();+ let inventory = crate::content::resolve_reporting(+ crate::walk::walk(corpus.path(), false).unwrap(),+ &|_| {},+ )+ .unwrap();let plan = plan_for(&inventory);let state = tempfile::tempdir().unwrap();let error = create_prepared_snapshot(@@ -2116,8 +2119,11 @@ mod tests {.unwrap_or_else(|poisoned| poisoned.into_inner());let corpus = tempfile::tempdir().unwrap();std::fs::write(corpus.path().join("a.txt"), "protected bytes").unwrap();- let inventory =- crate::content::resolve(crate::walk::walk(corpus.path(), false).unwrap()).unwrap();+ let inventory = crate::content::resolve_reporting(+ crate::walk::walk(corpus.path(), false).unwrap(),+ &|_| {},+ )+ .unwrap();let plan = plan_for(&inventory);let state = tempfile::tempdir().unwrap();let snapshot = create_snapshot(state.path(), "tx-protected", &inventory).unwrap();diff --git a/landing/docs/recipes/zero-config-autoindex.html b/landing/docs/recipes/zero-config-autoindex.htmlindex 625485b98..9ccca2fe9 100644--- a/landing/docs/recipes/zero-config-autoindex.html+++ b/landing/docs/recipes/zero-config-autoindex.html@@ -239,7 +239,8 @@ live indices at http://localhost:9280:ax-docs 1 docsax-exports 800 docsautoindex-catalog 9 docs</pre>- <p><code class="inline">--fresh</code> ignores the resume journal and restarts (ids stay idempotent). It never removes stale records from the destination, and it is refused once the state directory holds a durable corpus generation — re-run without it and the generated <code class="inline">--no-graph</code> path reconciles additions, changes, deletions, renames and no-op reruns incrementally. For an independent rebuild, use a new <code class="inline">--state-dir</code> and a new <code class="inline">--prefix</code>, plus a new <code class="inline">--brain</code> when graph detection is enabled (or add <code class="inline">--no-graph</code>). Validate before switching readers; the shared <code class="inline">autoindex-catalog</code> and old target require explicit, validated cleanup. A <code class="inline">--no-graph</code> state directory written before the generation format cannot be adopted in place and must be rebuilt the same way.</p>+ <p><code class="inline">--fresh</code> ignores the resume journal and restarts (ids stay idempotent), which on a graph-enabled or pre-generation state directory is how a rerun picks up files added since the last one. It never removes stale records from the destination, and it is refused once the state directory holds a durable corpus generation — re-run without it and the generated <code class="inline">--no-graph</code> path reconciles additions, changes, deletions, renames and no-op reruns incrementally.</p>+ <p>On a graph-enabled or pre-generation journal, a rerun that resumes an existing plan indexes changed files and reports added ones as skipped. Deleting a file is refused there: nothing on that path removes the documents it already published, so the rerun stops before touching the destination and names what is gone — the first ten, then an <code class="inline">… and N more</code> tail, with <code class="inline">--json</code> carrying every entry. Restore the file and rerun, or rebuild — in place by deleting the named indices and the state directory, or isolated under a new <code class="inline">--state-dir</code>, <code class="inline">--prefix</code> and <code class="inline">--brain</code> (or <code class="inline">--no-graph</code>), validated before you switch readers. The shared <code class="inline">autoindex-catalog</code> and the old target require explicit cleanup. A <code class="inline">--no-graph</code> state directory written before the generation format cannot be adopted in place and must be rebuilt the same way.</p><hr>diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 6145ed2eb..282110ffb 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -140,19 +140,28 @@ destabilize the server.xerj autoindex map [--url U] [--json] [--dataset SLUG]xerj autoindex status [--url U] [--state-dir P]---fresh ignores the resume journal and restarts (ids stay idempotent). It never-removes stale destination records, and it is refused once the state directory-holds a durable corpus generation — re-run without it and the generated---no-graph path reconciles additions, changes, deletions, renames and no-op-reruns incrementally. An independent rebuild requires a new --state-dir and a-new --prefix, plus a new --brain when graph detection is enabled (or---no-graph), followed by validation before switching readers. The shared-autoindex-catalog and old target require explicit, validated cleanup. A+--fresh ignores the resume journal and restarts (ids stay idempotent); on a+graph-enabled or pre-generation state directory that is how you pick up files+added since the last run. It never removes stale destination records, and it is+refused once the state directory holds a durable corpus generation — re-run+without it and the generated --no-graph path reconciles additions, changes,+deletions, renames and no-op reruns incrementally.++On a graph-enabled or pre-generation journal, a rerun that resumes an existing+plan indexes changed files and reports added ones as skipped. Deleting an+indexed file is refused there before any destination change, because nothing on+that path removes the documents it already published: restore the file and+rerun, or rebuild — in place by deleting the named indices and the state+directory, or isolated under a new --state-dir, --prefix and --brain (or+--no-graph), validated before switching readers. The refusal names the first ten+removed files with an "… and N more" tail; --json carries every entry. The+shared autoindex-catalog and the old target require explicit cleanup. A--no-graph state directory written before the generation format cannot beadopted in place and must be rebuilt the same way.Exit codes: 0 complete · 3 completed-with-junk (junk recorded, never fatal) ·-2 usage · 1 endpoint/journal error or refused unsafe corpus-state transition.+2 usage · 1 endpoint/journal error, a refused corpus removal, or a refused+unsafe corpus-state transition.Pipeline: walk → sniff → sample → infer → map → extract → bulk → correlate →catalog → verify.