Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(autoindex): resolve the generation path against main's resource, progress and spool policies

xerj-org71250fb33e0b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 971eee863..38b6d88d8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -79,6 +79,87 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
file. `--json` still carries every entry.
+### Added — incremental `--no-graph` corpus reconciliation
+
+- **`xerj autoindex --no-graph` now reconciles a changed folder instead of
+ re-deriving it.** A `--no-graph` run commits a durable *corpus generation*: a
+ manifest of content groups (stable group identity, content digest and byte
+ length, canonical path plus aliases, dataset assignment, validated output
+ counts) plus a sealed source snapshot of the prepared records. A later run
+ projects the current inventory onto the committed manifest and publishes only
+ the difference — additions, content changes, deletions, renames, junk
+ transitions and duplicate-alias moves all converge, and a no-op re-run issues
+ no data bulk and appends no new generation. A run interrupted mid-generation
+ replays from its own sealed snapshot, so the result never depends on whether
+ the source tree changed after the crash. Dataset and mapping identity is
+ frozen at generation 1: a file that would need a *new* dataset or a mapping
+ change is refused before any remote mutation rather than silently widening
+ the schema.
+
+ Scope, plainly: only `--no-graph`. Graph-enabled runs are unchanged, and each
+ changed generation still copies and prepares the full corpus (O(N) staging) —
+ the win is in what gets published and verified, not yet in what gets read.
+ `--snapshot-max-gb` (default 64) caps the logical staged payload; it is a
+ payload budget, not a disk-space or peak-memory guarantee.
+
+ Original work by Leonid Bugaev (@buger).
+
+- **Junk keeps the contract it always had on the generated path: recorded,
+ never fatal.** A file that cannot be read or recognised (`plan.junk_files`)
+ and a record that no dataset assignment accepts (`stats.junk`) both used to
+ abort a `--no-graph` run outright — the first with
+ `plan has no assignment for <file>`, the second with
+ `durable preparation of <file> produced N junk records`. A single zero-byte
+ file was enough to make a whole folder unindexable. Both are now counted
+ instead: the sealed prepared artifact carries its junk-record count, the
+ manifest group carries it into the generation, and the catalog's per-file
+ `junk` and run-level `junk_records_total` report it instead of a hardcoded
+ zero. Such a run exits **3** ("completed with junk"), the same signal the
+ legacy path publishes — the generated path previously returned a flat `0`
+ and lost it.
+
+- **`--dry-run` is honoured on an already-generated state directory.** It was
+ evaluated only after the pending-replay and reconcile branches had already
+ published and committed, so on any state directory past generation 1 the flag
+ was accepted, silently ignored, and the destination mutated. It is now decided
+ before either branch: it prints the plan a real run would act on — the
+ projected reconcile plan, or the sealed plan of a pending generation — and
+ returns without opening the journal for write, without snapshot GC, and
+ without a single bulk request.
+
+### Changed — `autoindex --fresh` is refused on a durable generation
+
+- **`--fresh` no longer silently destroys a committed corpus generation.**
+ `--fresh` deletes the resume journal, and the snapshot GC that follows every
+ journal open then sees an empty protected set and removes every sealed
+ snapshot — so on a generated state directory `--fresh` would discard the
+ committed manifest, the pending replay evidence, and the alias/path/
+ stale-record knowledge, while cleaning nothing at the destination. It is now
+ refused up front, naming the generation that blocks it, and pointing at the
+ plain re-run (which reconciles the change) or at an isolated rebuild with a
+ new `--state-dir` and `--prefix`.
+
+ **Unchanged for everyone else.** On a legacy (non-generated) journal —
+ including every graph-enabled corpus and anything `xerj brain` writes —
+ `--fresh` behaves exactly as before: it ignores the journal and restarts,
+ which is what `xerj brain`'s documented self-heal for a wiped data directory
+ depends on.
+
+### Migration — pre-generation `--no-graph` state directories must be rebuilt
+
+- **A `--no-graph` state directory written before this release cannot be
+ adopted as generation zero, and the first `--no-graph` run against it will
+ refuse with a followable rebuild command.** This is a deliberate, versioned
+ format boundary (`sync::GENERATION_FORMAT_VERSION = 1`), not a validation
+ failure: a pre-generation resume plan records no stable group identity, no
+ content byte length and no validated per-group output counts, so no amount of
+ evidence in it could reconstruct a manifest group. The refusal prints the
+ exact `xerj autoindex` argv for an isolated rebuild into a new `--state-dir`
+ and `--prefix`; the old target and the shared `autoindex-catalog` are left
+ alone and require explicit, validated cleanup once the new target is
+ verified. Nothing is migrated in place and no destination data is touched by
+ the refusal. Graph-enabled state directories are not affected.
+
## [1.0.0-rc.14] - 2026-08-10
### Changed
@@ -386,85 +467,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
prints the funnel on demand, including which repo-level numbers are
contaminated and why.
-### Added — incremental `--no-graph` corpus reconciliation
-
-- **`xerj autoindex --no-graph` now reconciles a changed folder instead of
- re-deriving it.** A `--no-graph` run commits a durable *corpus generation*: a
- manifest of content groups (stable group identity, content digest and byte
- length, canonical path plus aliases, dataset assignment, validated output
- counts) plus a sealed source snapshot of the prepared records. A later run
- projects the current inventory onto the committed manifest and publishes only
- the difference — additions, content changes, deletions, renames, junk
- transitions and duplicate-alias moves all converge, and a no-op re-run issues
- no data bulk and appends no new generation. A run interrupted mid-generation
- replays from its own sealed snapshot, so the result never depends on whether
- the source tree changed after the crash. Dataset and mapping identity is
- frozen at generation 1: a file that would need a *new* dataset or a mapping
- change is refused before any remote mutation rather than silently widening
- the schema.
-
- Scope, plainly: only `--no-graph`. Graph-enabled runs are unchanged, and each
- changed generation still copies and prepares the full corpus (O(N) staging) —
- the win is in what gets published and verified, not yet in what gets read.
- `--snapshot-max-gb` (default 64) caps the logical staged payload; it is a
- payload budget, not a disk-space or peak-memory guarantee.
-
-- **Junk keeps the contract it always had on the generated path: recorded,
- never fatal.** A file that cannot be read or recognised (`plan.junk_files`)
- and a record that no dataset assignment accepts (`stats.junk`) both used to
- abort a `--no-graph` run outright — the first with
- `plan has no assignment for <file>`, the second with
- `durable preparation of <file> produced N junk records`. A single zero-byte
- file was enough to make a whole folder unindexable. Both are now counted
- instead: the sealed prepared artifact carries its junk-record count, the
- manifest group carries it into the generation, and the catalog's per-file
- `junk` and run-level `junk_records_total` report it instead of a hardcoded
- zero. Such a run exits **3** ("completed with junk"), the same signal the
- legacy path publishes — the generated path previously returned a flat `0`
- and lost it.
-
-- **`--dry-run` is honoured on an already-generated state directory.** It was
- evaluated only after the pending-replay and reconcile branches had already
- published and committed, so on any state directory past generation 1 the flag
- was accepted, silently ignored, and the destination mutated. It is now decided
- before either branch: it prints the plan a real run would act on — the
- projected reconcile plan, or the sealed plan of a pending generation — and
- returns without opening the journal for write, without snapshot GC, and
- without a single bulk request.
-
-### Changed — `autoindex --fresh` is refused on a durable generation
-
-- **`--fresh` no longer silently destroys a committed corpus generation.**
- `--fresh` deletes the resume journal, and the snapshot GC that follows every
- journal open then sees an empty protected set and removes every sealed
- snapshot — so on a generated state directory `--fresh` would discard the
- committed manifest, the pending replay evidence, and the alias/path/
- stale-record knowledge, while cleaning nothing at the destination. It is now
- refused up front, naming the generation that blocks it, and pointing at the
- plain re-run (which reconciles the change) or at an isolated rebuild with a
- new `--state-dir` and `--prefix`.
-
- **Unchanged for everyone else.** On a legacy (non-generated) journal —
- including every graph-enabled corpus and anything `xerj brain` writes —
- `--fresh` behaves exactly as before: it ignores the journal and restarts,
- which is what `xerj brain`'s documented self-heal for a wiped data directory
- depends on.
-
-### Migration — pre-generation `--no-graph` state directories must be rebuilt
-
-- **A `--no-graph` state directory written before this release cannot be
- adopted as generation zero, and the first `--no-graph` run against it will
- refuse with a followable rebuild command.** This is a deliberate, versioned
- format boundary (`sync::GENERATION_FORMAT_VERSION = 1`), not a validation
- failure: a pre-generation resume plan records no stable group identity, no
- content byte length and no validated per-group output counts, so no amount of
- evidence in it could reconstruct a manifest group. The refusal prints the
- exact `xerj autoindex` argv for an isolated rebuild into a new `--state-dir`
- and `--prefix`; the old target and the shared `autoindex-catalog` are left
- alone and require explicit, validated cleanup once the new target is
- verified. Nothing is migrated in place and no destination data is touched by
- the refusal. Graph-enabled state directories are not affected.
-
## [1.0.0-rc.13] - 2026-08-08
### Security
@@ -734,6 +736,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
caveats. They are tracked for the following release rather than rushed into
this one.
+
### Changed — runtime-field types (can break existing mappings)
- **`mappings.runtime` is now validated against ES's runtime-field type
diff --git a/demo/usecases/autoindex/RECIPE_DRAFT.md b/demo/usecases/autoindex/RECIPE_DRAFT.md
index a035aa865..b5b6d881e 100644
--- a/demo/usecases/autoindex/RECIPE_DRAFT.md
+++ b/demo/usecases/autoindex/RECIPE_DRAFT.md
@@ -203,15 +203,22 @@ done in 0.1s — 3 datasets, 5801 records live, 0 junk records, 1 junk/skipped f
just polite re-runs: in the robustness evaluation, a `kill -9` at 18 s into a
200 MB file followed by a re-run converged to **byte-identical counts across
all 9 datasets** — no duplicates. `xerj autoindex status` shows the journal
-and live index counts. `--fresh` ignores the resume journal and restarts (ids stay idempotent). It
-never removes stale records from the destination, and it is refused once the
-state directory holds a durable corpus generation — re-run without it and the
-generated `--no-graph` path reconciles additions, changes, deletions, renames
-and no-op reruns incrementally. For an independent rebuild, use a new
-`--state-dir` and a new `--prefix`, plus a new `--brain` when graph detection
-is enabled (or add `--no-graph`). Validate before switching readers; the shared
-`autoindex-catalog` and old target require explicit, validated cleanup. A
-`--no-graph` state directory written before the generation format cannot be
+and live index counts. `--fresh` ignores the resume journal and restarts (ids stay idempotent), which on a
+graph-enabled or pre-generation state directory is how a rerun picks up files added
+since the last one. It never removes stale records from the destination, and it is
+refused once the state directory holds a durable corpus generation — re-run without it
+and the generated `--no-graph` path reconciles additions, changes, deletions, renames
+and no-op reruns incrementally.
+
+On a graph-enabled or pre-generation journal, a rerun that resumes an existing plan
+indexes changed files and reports added ones as skipped. Deleting a file is refused
+there: nothing on that path removes the documents it already published, so the rerun
+stops before touching the destination and names what is gone — the first ten, then an
+`… and N more` tail, with `--json` carrying every entry. Restore the file and rerun, or
+rebuild — in place by deleting the named indices and the state directory, or isolated
+under a new `--state-dir`, `--prefix` and `--brain` (or `--no-graph`), validated before
+you switch readers. The shared `autoindex-catalog` and the old target require explicit
+cleanup. A `--no-graph` state directory written before the generation format cannot be
adopted in place and must be rebuilt the same way.
## Reproduce it yourself
diff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.md
index 2a584d9d4..f3c92f1c3 100644
--- a/docs/recipes/zero-config-indexing.md
+++ b/docs/recipes/zero-config-indexing.md
@@ -228,15 +228,22 @@ live indices at http://localhost:9280:
autoindex-catalog 9 docs
```
-`--fresh` ignores the resume journal and restarts (ids stay idempotent). It
-never removes stale records from the destination, and it is refused once the
-state directory holds a durable corpus generation — re-run without it and the
-generated `--no-graph` path reconciles additions, changes, deletions, renames
-and no-op reruns incrementally. For an independent rebuild, use a new
-`--state-dir` and a new `--prefix`, plus a new `--brain` when graph detection
-is enabled (or add `--no-graph`). Validate before switching readers; the shared
-`autoindex-catalog` and old target require explicit, validated cleanup. A
-`--no-graph` state directory written before the generation format cannot be
+`--fresh` ignores the resume journal and restarts (ids stay idempotent), which on a
+graph-enabled or pre-generation state directory is how a rerun picks up files added
+since the last one. It never removes stale records from the destination, and it is
+refused once the state directory holds a durable corpus generation — re-run without it
+and the generated `--no-graph` path reconciles additions, changes, deletions, renames
+and no-op reruns incrementally.
+
+On a graph-enabled or pre-generation journal, a rerun that resumes an existing plan
+indexes changed files and reports added ones as skipped. Deleting a file is refused
+there: nothing on that path removes the documents it already published, so the rerun
+stops before touching the destination and names what is gone — the first ten, then an
+`… and N more` tail, with `--json` carrying every entry. Restore the file and rerun, or
+rebuild — in place by deleting the named indices and the state directory, or isolated
+under a new `--state-dir`, `--prefix` and `--brain` (or `--no-graph`), validated before
+you switch readers. The shared `autoindex-catalog` and the old target require explicit
+cleanup. A `--no-graph` state directory written before the generation format cannot be
adopted in place and must be rebuilt the same way.
## Reproduce it yourself
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index e662e2f01..2ee480f81 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -74,8 +74,10 @@ pub enum Cmd {
}
const FRESH_HELP: &str =
- "ignore an existing resume journal and restart (ids stay idempotent); refused when the \
-state directory holds a durable corpus generation, and it never resets destination records";
+ "ignore an existing resume journal and restart, rebuild the plan in place\n\
+ (ids stay idempotent); it never resets destination records,\n\
+ and is refused on a durable corpus generation — see\n\
+ RESUME POLICY";
const RESUME_POLICY_HELP: &str =
"generated --no-graph journals reconcile add, change, delete, rename, and no-op runs; \
a --no-graph state directory written before the generation format must be rebuilt into a new \
@@ -186,6 +188,14 @@ pub fn print_help() {
\n\
RESUME POLICY:\n\
{resume_policy_help}.\n\
+ On a graph-enabled or pre-generation journal the durable plan supports no-op\n\
+ resume and same-path content replacement. Files added after that plan was frozen\n\
+ are reported as skipped and are not indexed; --fresh rebuilds the plan in place\n\
+ and picks them up. Removing an indexed file is refused there before any remote\n\
+ mutation: its documents are already live and nothing on that path deletes them.\n\
+ Restore the file and rerun, or rebuild — in place by deleting the published\n\
+ indices and the state directory, or isolated under a new --state-dir, --prefix\n\
+ and --brain (or --no-graph), validated before you switch readers.\n\
--fresh is not cleanup and is not destination reconciliation: it never removes\n\
stale records from the destination, and it is refused outright once a durable\n\
corpus generation exists (re-run without it — the generated path reconciles the\n\
@@ -195,7 +205,8 @@ pub fn print_help() {
autoindex-catalog and old target only after validation.\n\
\n\
EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\
- 2 usage; 1 endpoint/journal failure or a refused unsafe state transition\n",
+ 2 usage; 1 endpoint/journal failure, a refused corpus removal, or a\n\
+ refused unsafe state transition\n",
fresh_help = FRESH_HELP,
resume_policy_help = RESUME_POLICY_HELP
);
@@ -545,6 +556,13 @@ mod tests {
}
}
+ #[test]
+ fn fresh_help_points_at_the_resume_policy_that_bounds_it() {
+ assert!(super::FRESH_HELP.contains("rebuild the plan in place"));
+ assert!(super::FRESH_HELP.contains("ids stay idempotent"));
+ assert!(super::FRESH_HELP.contains("RESUME POLICY"));
+ }
+
#[test]
fn graph_is_on_by_default_with_derived_brain() {
let cfg = index(&["data"]);
diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
index d67c161b5..becedafe4 100644
--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
@@ -69,6 +69,8 @@ struct MockState {
/// what `_count`/`_search` answer with, and the run verifies live data
/// counts against the journal (#195). Catalog rows are not data rows.
catalog_docs: HashMap<String, Value>,
+ /// Every (method, path) the run issued. The refusal tests assert on the
+ /// ABSENCE of remote mutations, which no document count can express.
requests: Vec<(String, String)>,
data_bulk_number: usize,
fail_data_bulk: usize,
@@ -207,11 +209,6 @@ fn handle(mut stream: TcpStream, state: &Arc<Mutex<MockState>>) {
if response_delay_ms > 0 {
thread::sleep(std::time::Duration::from_millis(response_delay_ms));
}
- state
- .lock()
- .unwrap()
- .requests
- .push((method.to_owned(), path.to_owned()));
let (status, response) = if method == "GET" && path == "/v1/embedding/identity" {
let identity = state.lock().unwrap().embedding_identity_sha256.clone();
@@ -368,11 +365,25 @@ fn bulk_response(body: &[u8], state: &Arc<Mutex<MockState>>) -> Value {
// Graph edge bulks are accepted and discarded: this module counts *data*
// bulks to place injected failures, so edge bulks must not shift the
// numbering (the graph pipeline has its own e2e suite in detect::e2e).
+ //
+ // Read the target off whichever action verb the line carries. The brain
+ // meta document is written with `create`, not `index`, so keying on
+ // `/index/_index` alone let it fall through to the catalog branch and be
+ // recorded as fixture drift once this module started indexing with the
+ // graph enabled.
let is_graph = lines
.first()
.and_then(|line| serde_json::from_slice::<Value>(line).ok())
- .and_then(|action| action.pointer("/index/_index").cloned())
- .and_then(|index| index.as_str().map(str::to_owned))
+ .and_then(|action| {
+ ["index", "create", "delete", "update"]
+ .iter()
+ .find_map(|verb| {
+ action
+ .pointer(&format!("/{verb}/_index"))
+ .and_then(Value::as_str)
+ .map(str::to_owned)
+ })
+ })
.is_some_and(|index| index.ends_with("-edges"));
if is_graph {
return json!({"errors": false, "items": []});
@@ -573,6 +584,348 @@ fn dataset_catalog_docs(endpoint: &MockEndpoint) -> Vec<Value> {
docs
}
+fn assert_unsupported_delta_without_remote_mutation(
+ endpoint: &MockEndpoint,
+ config: IndexCfg,
+ expected_added: &[&str],
+ expected_vanished: &[&str],
+) {
+ let request_start = endpoint.state.lock().unwrap().requests.len();
+ let journal_path = config.state_dir.as_ref().unwrap().join("journal.ndjson");
+ let journal_before = fs::read(&journal_path).unwrap();
+ let error = run_index(config).unwrap_err();
+ let attempted_requests = endpoint.state.lock().unwrap().requests[request_start..].to_vec();
+ assert_eq!(
+ attempted_requests,
+ [("GET".to_owned(), "/".to_owned())],
+ "a refused attempt may perform only the endpoint-readiness GET"
+ );
+ assert_eq!(
+ fs::read(journal_path).unwrap(),
+ journal_before,
+ "preflight refusal must not append a resume event or rewrite the journal"
+ );
+ let message = format!("{error:#}");
+ assert!(message.contains("made no remote mutations"), "{message}");
+ assert!(
+ message.contains("no longer exist in the folder"),
+ "{message}"
+ );
+ assert!(
+ message.contains("restore the removed file(s) and rerun"),
+ "{message}"
+ );
+ assert!(
+ message.contains("rebuild in place by deleting the indices"),
+ "{message}"
+ );
+ assert!(
+ message.contains("new --state-dir, a new --prefix"),
+ "{message}"
+ );
+ assert!(message.contains("new --brain"), "{message}");
+ for path in expected_added {
+ assert!(
+ message.contains(path),
+ "missing added path {path}: {message}"
+ );
+ }
+ for path in expected_vanished {
+ assert!(
+ message.contains(path),
+ "missing vanished path {path}: {message}"
+ );
+ }
+}
+
+/// The documented headline workflow: point autoindex at a folder, add a file,
+/// rerun. The rerun must not fail, must say plainly that the added file was
+/// not indexed, and `--fresh` must then absorb it in place.
+#[test]
+fn a_rerun_after_an_added_file_succeeds_and_fresh_absorbs_it() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("first.csv"), "id,value\n1,first\n").unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();
+ // 3 = completed-with-junk: the added file is reported as skipped, not
+ // indexed, because the frozen plan cannot absorb it.
+ let (code, report) = run_index_report(config.clone()).unwrap();
+ assert_eq!(code, 3);
+ let report = report.unwrap();
+ assert_eq!(report["files_junk"], 1, "{report}");
+ // `records_total` is the live server-side count (#195), so it still reports
+ // the record the first run published. What must be zero is the run-local
+ // counter: this rerun indexed nothing, because the frozen plan cannot
+ // absorb a file that appeared after it was written.
+ assert_eq!(report["records_submitted_this_run"], 0, "{report}");
+ assert_eq!(report["files_submitted_this_run"], 0, "{report}");
+ // Two live documents for the one indexed source, not one: this module now
+ // indexes graph-enabled (`--no-graph` takes the generated executor), and a
+ // graph-enabled run publishes a file-level node document alongside the row.
+ // `second.csv` still published nothing — that is what the counters above
+ // pin.
+ assert_eq!(report["records_total"], 2, "{report}");
+ {
+ let locked = endpoint.state.lock().unwrap();
+ let live: Vec<&str> = locked
+ .docs
+ .values()
+ .filter_map(|doc| doc["ax_path"].as_str())
+ .collect();
+ assert!(
+ live.iter().all(|path| *path == "first.csv"),
+ "the added file must not be published by a plan that predates it: {live:?}"
+ );
+ }
+
+ // --fresh rebuilds the plan in place and picks the new file up.
+ config.fresh = true;
+ assert_eq!(run_index(config).unwrap(), 0);
+ let locked = endpoint.state.lock().unwrap();
+ let indexed: std::collections::HashSet<&str> = locked
+ .docs
+ .values()
+ .filter_map(|doc| doc["ax_path"].as_str())
+ .collect();
+ assert!(indexed.contains("first.csv"), "{indexed:?}");
+ assert!(indexed.contains("second.csv"), "{indexed:?}");
+}
+
+#[test]
+fn completed_plan_rejects_vanished_content_group_before_remote_mutation() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("first.csv"), "id,value\n1,first\n").unwrap();
+ fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::remove_file(corpus.path().join("second.csv")).unwrap();
+ assert_unsupported_delta_without_remote_mutation(&endpoint, config, &[], &["second.csv"]);
+}
+
+#[test]
+fn completed_plan_rejects_mixed_membership_delta_in_stable_order() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("b-old.csv"), "id,value\n1,b\n").unwrap();
+ fs::write(corpus.path().join("a-old.csv"), "id,value\n2,a\n").unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::remove_file(corpus.path().join("b-old.csv")).unwrap();
+ fs::remove_file(corpus.path().join("a-old.csv")).unwrap();
+ fs::write(corpus.path().join("z-new.csv"), "id,value\n3,z\n").unwrap();
+ fs::write(corpus.path().join("m-new.csv"), "id,value\n4,m\n").unwrap();
+ config.json = true;
+ let request_start = endpoint.state.lock().unwrap().requests.len();
+ let error = run_index(config).unwrap_err();
+ assert_eq!(
+ endpoint.state.lock().unwrap().requests[request_start..],
+ [("GET".to_owned(), "/".to_owned())]
+ );
+
+ let typed = error
+ .downcast_ref::<UnsupportedInventoryDeltaError>()
+ .unwrap();
+ let value = typed.to_json();
+ assert_eq!(value["schema"], "xerj.autoindex.unsupported_sync_delta.v1");
+ assert_eq!(value["status"], "error");
+ let added: Vec<&str> = value["added_content_groups"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .map(|entry| entry["path"].as_str().unwrap())
+ .collect();
+ let vanished: Vec<&str> = value["vanished_content_groups"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .map(|entry| entry["path"].as_str().unwrap())
+ .collect();
+ assert_eq!(added, ["m-new.csv", "z-new.csv"]);
+ assert_eq!(vanished, ["a-old.csv", "b-old.csv"]);
+}
+
+/// An in-place EDIT is a replacement, not a removal. `--fresh` is the route
+/// the refusal itself recommends for picking up added and changed files, so
+/// the gate must not classify the edited file's superseded content key as a
+/// vanished group — the path is still there, and the pipeline republishes it.
+#[test]
+fn fresh_absorbs_an_edited_file_instead_of_calling_it_a_removal() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("notes.csv"), "id,value\n1,before\n").unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::write(corpus.path().join("notes.csv"), "id,value\n1,after\n").unwrap();
+ config.fresh = true;
+ assert_eq!(
+ run_index(config).unwrap(),
+ 0,
+ "an edited file is a same-path replacement, not a vanished content group"
+ );
+ let locked = endpoint.state.lock().unwrap();
+ assert!(
+ locked
+ .docs
+ .values()
+ .any(|doc| doc["value"].as_str() == Some("after")),
+ "the new content is live: {:?}",
+ locked.docs
+ );
+ // Documented, and NOT what this gate is for: `--fresh` rebuilds the plan,
+ // it does not reconcile the destination. Document ids derive from the
+ // content key (`ids::doc_id`), so the pre-edit record stays live beside
+ // the new one — the same outcome `--fresh` has always had. An ordinary
+ // rerun is the clean route for an edit: it keeps the planned key and runs
+ // a delete-before-replace transaction on it. What must never happen is
+ // this run being REFUSED by a message claiming a file that is sitting in
+ // the folder has vanished.
+ assert!(
+ locked
+ .docs
+ .values()
+ .any(|doc| doc["value"].as_str() == Some("before")),
+ "known --fresh gap: superseded records are not deleted: {:?}",
+ locked.docs
+ );
+}
+
+/// The clean route for the same edit: an ordinary rerun keeps the planned key
+/// and replaces its records, so nothing is stranded. This is the contrast that
+/// makes the `--fresh` gap above a documented trade-off rather than a silent
+/// one.
+#[test]
+fn an_ordinary_rerun_replaces_an_edited_file_without_stranding_records() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("notes.csv"), "id,value\n1,before\n").unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::write(corpus.path().join("notes.csv"), "id,value\n1,after\n").unwrap();
+ assert_eq!(run_index(config).unwrap(), 0);
+ let locked = endpoint.state.lock().unwrap();
+ let values: Vec<&str> = locked
+ .docs
+ .values()
+ .filter_map(|doc| doc["value"].as_str())
+ .collect();
+ assert_eq!(values, ["after"], "the superseded record is replaced");
+}
+
+/// The composite workflow the run's own stderr recommends: index, then add one
+/// file and edit another, rerun (added file reported and skipped by the frozen
+/// plan), then `--fresh` to absorb both.
+#[test]
+fn fresh_absorbs_an_addition_and_an_edit_after_a_reported_rerun() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("first.csv"), "id,value\n1,before\n").unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::write(corpus.path().join("first.csv"), "id,value\n1,after\n").unwrap();
+ fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();
+ // 3 = completed-with-junk: the frozen plan cannot absorb the added file,
+ // so the rerun reports it as skipped rather than refusing.
+ assert_eq!(run_index(config.clone()).unwrap(), 3);
+
+ config.fresh = true;
+ assert_eq!(
+ run_index(config).unwrap(),
+ 0,
+ "--fresh is the documented way out of exactly this state"
+ );
+ let locked = endpoint.state.lock().unwrap();
+ let live: std::collections::HashSet<&str> = locked
+ .docs
+ .values()
+ .filter_map(|doc| doc["ax_path"].as_str())
+ .collect();
+ assert!(live.contains("first.csv"), "{live:?}");
+ assert!(live.contains("second.csv"), "{live:?}");
+ assert!(
+ locked
+ .docs
+ .values()
+ .any(|doc| doc["value"].as_str() == Some("after")),
+ "the edited file's new content is live: {:?}",
+ locked.docs
+ );
+}
+
+#[test]
+fn fresh_cannot_erase_the_plan_and_bypass_the_removal_gate() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("old.csv"), "id,value\n1,old\n").unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::remove_file(corpus.path().join("old.csv")).unwrap();
+ fs::write(corpus.path().join("new.csv"), "id,value\n2,new\n").unwrap();
+ config.fresh = true;
+ assert_unsupported_delta_without_remote_mutation(&endpoint, config, &["new.csv"], &["old.csv"]);
+}
+
+#[test]
+fn deleting_one_path_of_a_duplicate_pair_is_not_a_removed_content_group() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ let bytes = "id,value\n1,same\n";
+ fs::write(corpus.path().join("a.csv"), bytes).unwrap();
+ fs::write(corpus.path().join("b.csv"), bytes).unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ // b.csv becomes canonical under the same content key: the group survives
+ // the deletion, so no document is stranded and the rerun is allowed.
+ fs::remove_file(corpus.path().join("a.csv")).unwrap();
+ let result = run_index(config);
+ assert!(
+ result.is_ok(),
+ "a surviving content group must not trip the removal gate: {result:?}"
+ );
+ let locked = endpoint.state.lock().unwrap();
+ // Row documents only: this module indexes graph-enabled, so the surviving
+ // source also carries a file-level node document under the same ax_path.
+ let live: Vec<&str> = data_rows(&locked)
+ .into_iter()
+ .filter_map(|doc| doc["ax_path"].as_str())
+ .collect();
+ assert_eq!(live, ["b.csv"], "canonical path follows the surviving file");
+}
+
#[test]
fn unchanged_planned_junk_is_not_reported_as_added_content() {
let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
@@ -595,6 +948,70 @@ fn unchanged_planned_junk_is_not_reported_as_added_content() {
);
}
+#[test]
+fn deleted_planned_junk_is_swept_rather_than_refused() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("rows.csv"), "id,value\n1,kept\n").unwrap();
+ fs::write(
+ corpus.path().join("opaque.bin"),
+ [0_u8, 159, 146, 150, 0, 255],
+ )
+ .unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 3);
+
+ // A junk file publishes exactly one catalog row and nothing else, and the
+ // #238 sweep deletes that row. Nothing is stranded, so the removal gate
+ // must not fire — refusing here would block a case the pipeline handles
+ // completely. Only a file that published DOCUMENTS refuses a rerun.
+ fs::remove_file(corpus.path().join("opaque.bin")).unwrap();
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+ {
+ let locked = endpoint.state.lock().unwrap();
+ assert!(
+ locked
+ .catalog_docs
+ .values()
+ .all(|doc| doc["path"].as_str() != Some("opaque.bin")),
+ "the deleted junk file's catalog row must be swept, not left immortal"
+ );
+ assert!(
+ locked
+ .catalog_docs
+ .values()
+ .any(|doc| doc["path"].as_str() == Some("rows.csv")),
+ "the surviving indexed file keeps its entry"
+ );
+ }
+
+ // Deleting the INDEXED file is still refused: its documents are live.
+ fs::remove_file(corpus.path().join("rows.csv")).unwrap();
+ assert_unsupported_delta_without_remote_mutation(&endpoint, config, &[], &["rows.csv"]);
+}
+
+#[test]
+fn completed_plan_rejects_empty_current_folder_before_remote_mutation() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("only.csv"), "id,value\n1,only\n").unwrap();
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(run_index(config.clone()).unwrap(), 0);
+
+ fs::remove_file(corpus.path().join("only.csv")).unwrap();
+ assert_unsupported_delta_without_remote_mutation(&endpoint, config.clone(), &[], &["only.csv"]);
+ // Even `--fresh` must not erase the only durable inventory evidence and
+ // then report success while the destination still contains the document.
+ config.fresh = true;
+ assert_unsupported_delta_without_remote_mutation(&endpoint, config, &[], &["only.csv"]);
+}
+
#[test]
fn semantic_resume_rejects_embedding_identity_drift_before_another_bulk() {
let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
@@ -822,7 +1239,10 @@ fn deleted_path_bytes_remain_while_its_documents_remain_live() {
dataset_catalog_docs(&endpoint)[0]["bytes"],
durable_live_bytes
);
- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 4);
+ // Row documents only: this module indexes graph-enabled now that
+ // `--no-graph` takes the generated executor, so each source also publishes
+ // one file-level node document.
+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 4);
fs::remove_file(&removed).unwrap();
// Autoindex still does not reconcile a missing canonical path: its indexed
@@ -839,7 +1259,10 @@ fn deleted_path_bytes_remain_while_its_documents_remain_live() {
&[],
&["removed.csv"],
);
- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 4);
+ // Row documents only: this module indexes graph-enabled now that
+ // `--no-graph` takes the generated executor, so each source also
+ // publishes one file-level node document.
+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 4);
assert_eq!(
dataset_catalog_docs(&endpoint)[0]["bytes"],
durable_live_bytes
@@ -1645,7 +2068,12 @@ fn deleting_an_entire_duplicate_group_strands_no_pending_replacement() {
fs::remove_file(corpus.path().join("dup-a.csv")).unwrap();
fs::remove_file(corpus.path().join("dup-b.csv")).unwrap();
for _ in 0..2 {
- assert_eq!(run_index(config.clone()).unwrap(), 0);
+ assert_unsupported_delta_without_remote_mutation(
+ &endpoint,
+ config.clone(),
+ &[],
+ &["dup-a.csv"],
+ );
assert_eq!(
event_count(state_dir.path(), "file_replace_start"),
starts,
@@ -2090,7 +2518,7 @@ printf '%s' '{"schema":1,"extractor":"xerj-autoindex/__XERJ_VERSION__","parser":
assert_eq!(run_index(config).unwrap(), 0);
assert_eq!(fs::read_to_string(&count).unwrap().lines().count(), 2);
assert_eq!(file_done_count(state_dir.path()), 1);
- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 1);
+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 1);
}
#[cfg(unix)]
@@ -2149,7 +2577,7 @@ printf '%s' '{"schema":1,"extractor":"xerj-autoindex/__XERJ_VERSION__","parser":
assert_eq!(code, 0);
assert_eq!(fs::read_to_string(&count).unwrap().lines().count(), 2);
assert_eq!(file_done_count(state_dir.path()), 1);
- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 1);
+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 1);
let reuse = report.unwrap()["pdf_extraction_reuse"].clone();
assert_eq!(reuse["reservations_started"], 0);
assert_eq!(reuse["capacity_fallbacks"], 1);
@@ -2435,7 +2863,7 @@ printf '%s' '{"schema":1,"extractor":"xerj-autoindex/__XERJ_VERSION__","parser":
PDFS,
"each unique PDF must be parsed once in Phase A and replayed in Phase B"
);
- assert_eq!(endpoint.state.lock().unwrap().docs.len(), PDFS);
+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), PDFS);
let reuse = report.unwrap()["pdf_extraction_reuse"].clone();
assert_eq!(reuse["reservations_started"], PDFS);
assert_eq!(
@@ -2498,7 +2926,7 @@ printf '%s' '{"schema":1,"extractor":"xerj-autoindex/__XERJ_VERSION__","parser":
let (code, report) = run_index_report(config).unwrap();
assert_eq!(code, 0);
assert_eq!(fs::read_to_string(&count).unwrap().lines().count(), 2);
- assert_eq!(endpoint.state.lock().unwrap().docs.len(), 1);
+ assert_eq!(data_rows(&endpoint.state.lock().unwrap()).len(), 1);
assert_eq!(file_done_count(state_dir.path()), 1);
assert!(crate::extract::pdf::corrupted_replay_reservation_was_dropped());
let reuse = report.unwrap()["pdf_extraction_reuse"].clone();
diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
index 0e4970e9c..6db75e62a 100644
--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
@@ -389,7 +389,9 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {
url: url.to_owned(),
api_key: None,
workers: 1,
+ scan_workers: 1,
pdf_workers: 1,
+ resource_notes: Vec::new(),
pdf_timeout_secs: 30,
bulk_mb: 1,
bulk_timeout_secs: 30,
@@ -406,6 +408,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {
dry_run: false,
json: false,
quiet: true,
+ progress: crate::progress::ProgressMode::None,
+ progress_interval: None,
}
}
@@ -531,6 +535,7 @@ fn initial_generation_and_noop_are_end_to_end_idempotent() {
&root.to_string_lossy(),
&endpoint.url,
"incremental-http",
+ false,
)
.unwrap();
assert!(
@@ -1181,3 +1186,91 @@ fn a_junk_record_is_counted_into_the_generation_and_never_fatal() {
assert_eq!(file_doc["junk"], 1);
assert_eq!(file_doc["records"], 40);
}
+
+/// Where this branch and #241 meet.
+///
+/// The generated `--no-graph` route returns from inside `run_index_report`,
+/// several hundred lines before the legacy `pr.finish` at the bottom of the
+/// function. `Ticker::drop` closes an unfinished stream with `ok=false`, so a
+/// generated run that merely returned would print an *aborted* terminal line
+/// immediately after committing a generation successfully — a silent
+/// contradiction between two changes that each looked right on its own.
+///
+/// This pins all three properties on the generated path: the phase-A scan is
+/// narrated (it is a per-file, minutes-long phase here exactly as on the legacy
+/// path), the stream closes with a truthful terminal line, and `--progress
+/// none` on the same path stays completely silent.
+#[test]
+fn a_generated_run_narrates_its_scan_and_closes_its_own_stream() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap();
+ let _replay_guard = sync_executor::REPLAY_FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _sink_guard = crate::progress::SINK_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ for n in 0..4 {
+ fs::write(
+ corpus.path().join(format!("rows-{n}.csv")),
+ format!("id,value\n{n}0,alpha\n{n}1,beta\n"),
+ )
+ .unwrap();
+ }
+ let endpoint = HttpEndpoint::start();
+
+ // Generation 1: the initial commit, through the real progress surface.
+ let mut loud = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);
+ loud.quiet = false;
+ loud.progress = crate::progress::ProgressMode::Plain;
+ loud.progress_interval = Some(std::time::Duration::from_secs(1));
+ let buffer = Arc::new(Mutex::new(Vec::new()));
+ let code = {
+ let _sink = crate::progress::install_test_sink(&buffer);
+ run_index(loud.clone()).unwrap()
+ };
+ assert_eq!(code, 0);
+ let stream = String::from_utf8(buffer.lock().unwrap().clone()).unwrap();
+ let done = stream
+ .lines()
+ .find(|line| line.starts_with("xerj-done "))
+ .unwrap_or_else(|| panic!("a generated run must close its own stream:\n{stream}"));
+ assert!(
+ done.contains("ok=true") && done.contains("exit=0"),
+ "a committed generation is not an aborted run: {done}"
+ );
+ assert!(done.contains("reason=completed"), "{done}");
+
+ // Generation 2: the reconcile branch, which runs its own phase-A scan.
+ fs::write(corpus.path().join("rows-0.csv"), "id,value\n00,CHANGED\n").unwrap();
+ let buffer = Arc::new(Mutex::new(Vec::new()));
+ let code = {
+ let _sink = crate::progress::install_test_sink(&buffer);
+ run_index(loud).unwrap()
+ };
+ assert_eq!(code, 0);
+ let stream = String::from_utf8(buffer.lock().unwrap().clone()).unwrap();
+ assert!(
+ stream
+ .lines()
+ .any(|line| line.starts_with("xerj-progress ") && line.contains("phase=scan")),
+ "the reconcile projection re-scans every file and must say so:\n{stream}"
+ );
+ let done = stream
+ .lines()
+ .find(|line| line.starts_with("xerj-done "))
+ .unwrap_or_else(|| panic!("a reconciled generation must close its own stream:\n{stream}"));
+ assert!(done.contains("ok=true"), "{done}");
+
+ // The other half of the contract on the same path: silence means silence.
+ fs::write(corpus.path().join("rows-1.csv"), "id,value\n10,CHANGED\n").unwrap();
+ let quiet = cfg(corpus.path(), state_dir.path(), &endpoint.url, false);
+ assert_eq!(quiet.progress, crate::progress::ProgressMode::None);
+ let buffer = Arc::new(Mutex::new(Vec::new()));
+ {
+ let _sink = crate::progress::install_test_sink(&buffer);
+ assert_eq!(run_index(quiet).unwrap(), 0);
+ }
+ assert!(
+ buffer.lock().unwrap().is_empty(),
+ "--progress none asked for nothing: {}",
+ String::from_utf8_lossy(&buffer.lock().unwrap())
+ );
+}
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index 414d62e00..205487df3 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -28,6 +28,7 @@ mod sync_executor;
pub mod walk;
use anyhow::{Context, Result};
+use serde::Serialize;
use serde_json::{json, Map, Value};
use std::collections::HashMap;
use std::io::{BufRead, BufReader, Seek, Write};
@@ -222,6 +223,33 @@ fn generated_exit_code(summary: &Value) -> i32 {
}
}
+/// Terminal progress line for a run that ends on the generated `--no-graph`
+/// path.
+///
+/// Every exit closes the stream itself (#241). `Ticker::drop` reports
+/// `ok=false` for a run that never called `finish`, so a generated run that
+/// merely returned would print an aborted line straight after a successful
+/// commit — the one place the two landed changes could contradict each other.
+/// Exit 3 is success ("completed, some input was unusable"), so it is spelled
+/// out in words rather than left as a bare number.
+fn finish_generated_progress(pr: &Progress, code: i32, summary: &Value) {
+ let count = |field: &str| summary.get(field).and_then(Value::as_u64).unwrap_or(0);
+ pr.finish(
+ true,
+ code,
+ if code == 3 {
+ "completed-with-junk"
+ } else {
+ "completed"
+ },
+ &[
+ ("files", count("files_indexed")),
+ ("records", count("records_total")),
+ ("generation", count("generation")),
+ ],
+ );
+}
+
fn begin_non_graph_generation(
es: &Es,
journal: &mut state::Journal,
@@ -391,6 +419,13 @@ struct CliErrorRoute {
fn route_cli_error(error: &anyhow::Error, json_output: bool) -> CliErrorRoute {
if json_output {
+ // Two distinct refusals reach the JSON route, and both must keep it.
+ // `UnsafeFreshGenerationError` guards durable *generation* state on the
+ // `--no-graph` path; `UnsupportedInventoryDeltaError` (#254) guards the
+ // legacy graph-enabled path against a rerun that would strand live
+ // documents. Neither supersedes the other — they protect different
+ // destinations — so routing only one would silently drop the other's
+ // machine-readable rendering back to a prose stderr line.
if let Some(unsafe_fresh) = error.downcast_ref::<UnsafeFreshGenerationError>() {
return CliErrorRoute {
exit_code: 1,
@@ -398,6 +433,13 @@ fn route_cli_error(error: &anyhow::Error, json_output: bool) -> CliErrorRoute {
stderr: None,
};
}
+ if let Some(delta) = error.downcast_ref::<UnsupportedInventoryDeltaError>() {
+ return CliErrorRoute {
+ exit_code: 1,
+ stdout: Some(delta.to_json().to_string()),
+ stderr: None,
+ };
+ }
}
CliErrorRoute {
exit_code: 1,
@@ -1538,6 +1580,284 @@ impl std::fmt::Display for UnsafeFreshGenerationError {
impl std::error::Error for UnsafeFreshGenerationError {}
+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
+struct InventoryDeltaEntry {
+ file_key: String,
+ path: String,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
+struct UnsupportedInventoryDelta {
+ added_content_groups: Vec<InventoryDeltaEntry>,
+ vanished_content_groups: Vec<InventoryDeltaEntry>,
+}
+
+/// Everything the refusal needs to name the destination it is protecting.
+/// Collected at the gate so the message can list the exact indices and state
+/// directory an operator has to act on, instead of describing them abstractly.
+#[derive(Debug, Clone, PartialEq, Eq)]
+struct RefusalTargets {
+ state_dir: String,
+ data_indices: Vec<String>,
+ edges_index: Option<String>,
+}
+
+impl RefusalTargets {
+ fn describe(cfg: &IndexCfg, state_dir: &Path, plan: &Plan) -> Self {
+ let mut data_indices: Vec<String> = plan.datasets.iter().map(|d| d.index.clone()).collect();
+ data_indices.sort();
+ data_indices.dedup();
+ let edges_index = (!cfg.no_graph).then(|| {
+ let brain = cfg
+ .brain
+ .clone()
+ .unwrap_or_else(|| derive_brain_name(&cfg.root));
+ detect::edges_index_name(&brain)
+ });
+ Self {
+ state_dir: state_dir.display().to_string(),
+ data_indices,
+ edges_index,
+ }
+ }
+
+ fn indices_phrase(&self) -> String {
+ if self.data_indices.is_empty() {
+ "the indices this plan publishes".to_string()
+ } else {
+ self.data_indices.join(", ")
+ }
+ }
+
+ fn edges_note(&self) -> String {
+ match &self.edges_index {
+ Some(edges) => format!(
+ " Graph edges taught by the removed file(s) stay live in {edges} until that \
+ index is deleted too."
+ ),
+ None => String::new(),
+ }
+ }
+}
+
+/// The #195 zero-live verification failure, as a type rather than a string.
+///
+/// The journal claims records were made visible; the destination answers with
+/// none. That is always a failure, but not always the *same* failure: for a
+/// composing caller like `xerj brain` a resume journal that outlived a wiped
+/// data directory has a specific, executable recovery (`--fresh` in place),
+/// while a write-blocked or unreachable server does not. Only a typed error
+/// lets the caller tell them apart — `anyhow::bail!` forces it to either
+/// string-match the prose or reprint it verbatim.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ZeroLiveVerificationError {
+ /// Records the resume journal says were published.
+ pub journal_records: u64,
+ /// Completed files those records came from.
+ pub files_done_journaled: usize,
+ /// Dataset indices that answered with zero live documents.
+ pub dataset_indices: usize,
+}
+
+impl std::fmt::Display for ZeroLiveVerificationError {
+ fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(
+ formatter,
+ "autoindex verification failed: the resume journal records {} \
+ record(s) from {} completed file(s), but 0 documents are \
+ live across the {} dataset index(es). The indices exist and look healthy but \
+ hold nothing — a server-side write rejection (e.g. a disk flood-stage or \
+ index write block), deleted indices, or an unreadable server can all cause \
+ this. Fix the server-side condition, then rerun with --fresh to re-index \
+ from scratch",
+ self.journal_records, self.files_done_journaled, self.dataset_indices
+ )
+ }
+}
+
+impl std::error::Error for ZeroLiveVerificationError {}
+
+#[derive(Debug)]
+struct UnsupportedInventoryDeltaError {
+ delta: UnsupportedInventoryDelta,
+ targets: RefusalTargets,
+}
+
+impl UnsupportedInventoryDeltaError {
+ fn to_json(&self) -> Value {
+ json!({
+ "schema": "xerj.autoindex.unsupported_sync_delta.v1",
+ "status": "error",
+ "error": "unsupported_content_group_removal",
+ "message": "this attempt made no remote mutations. Files that were indexed under this resume plan no longer exist in the folder, and their documents are still live in the destination; removing files from an indexed folder is not reconciled yet",
+ "vanished_content_groups": self.delta.vanished_content_groups,
+ // Context, not the reason for the refusal: a rerun over a frozen
+ // plan does not index files added after the plan was frozen.
+ "added_content_groups": self.delta.added_content_groups,
+ "recovery": {
+ "restore_removed_files": "put the listed file(s) back and rerun; every other file keeps its resume state",
+ "rebuild_in_place": format!(
+ "delete the indices this plan publishes ({}) and the state directory {}, then rerun. This re-extracts and re-embeds the whole corpus.{}",
+ self.targets.indices_phrase(),
+ self.targets.state_dir,
+ self.targets.edges_note().trim_start_matches(' ')
+ ),
+ "rebuild_isolated": "index with a new --state-dir, new --prefix, and (when graph detection is enabled) new --brain; alternatively add --no-graph. Validate the isolated target before switching readers, then clean the old one",
+ "fresh_warning": "--fresh re-extracts the current folder in place and does pick up added and changed files, but it never deletes documents already published for removed files, so it is refused here"
+ }
+ })
+ }
+}
+
+impl std::fmt::Display for UnsupportedInventoryDeltaError {
+ fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ // Capped like the duplicate and unplanned-file listings above; see
+ // REFUSAL_LIST_CAP. The machine-readable `--json` rendering keeps the
+ // full lists, so nothing is lost — only the prose is bounded.
+ let render = |entries: &[InventoryDeltaEntry]| {
+ let mut rendered = entries
+ .iter()
+ .take(REFUSAL_LIST_CAP)
+ .map(|entry| format!("{} ({})", entry.path, entry.file_key))
+ .collect::<Vec<_>>()
+ .join(", ");
+ let remaining = entries.len().saturating_sub(REFUSAL_LIST_CAP);
+ if remaining > 0 {
+ rendered.push_str(&format!(", … and {remaining} more"));
+ }
+ rendered
+ };
+ write!(
+ formatter,
+ "{} file(s) indexed under this resume plan no longer exist in the folder, and their \
+ documents are still live in the destination. Removing files from an indexed folder \
+ is not reconciled yet, so this attempt made no remote mutations — no documents, \
+ aliases, graph edges or catalog entries were written. Removed content groups [{}].",
+ self.delta.vanished_content_groups.len(),
+ render(&self.delta.vanished_content_groups)
+ )?;
+ if !self.delta.added_content_groups.is_empty() {
+ write!(
+ formatter,
+ " Also present but not in the frozen resume plan, so not indexed by this run \
+ either [{}].",
+ render(&self.delta.added_content_groups)
+ )?;
+ }
+ write!(
+ formatter,
+ " Recovery, cheapest first: (1) restore the removed file(s) and rerun — every other \
+ file keeps its resume state; (2) rebuild in place by deleting the indices this plan \
+ publishes ({}) and the state directory {}, then rerunning — this re-extracts and \
+ re-embeds the whole corpus.{} (3) rebuild isolated with a new --state-dir, a new \
+ --prefix and, when graph detection is enabled, a new --brain (or --no-graph), \
+ validate it, switch readers, then clean the old target. `--fresh` picks up added \
+ and changed files in place but never deletes documents for removed files, so it is \
+ refused here too",
+ self.targets.indices_phrase(),
+ self.targets.state_dir,
+ self.targets.edges_note()
+ )
+ }
+}
+
+impl std::error::Error for UnsupportedInventoryDeltaError {}
+
+impl UnsupportedInventoryDelta {
+ fn between(files: &[walk::FileEntry], keys: &[String], plan: &Plan) -> Self {
+ let current_keys: std::collections::HashSet<&str> = keys
+ .iter()
+ .filter(|key| !key.is_empty())
+ .map(String::as_str)
+ .collect();
+ let durable_keys: std::collections::HashSet<&str> = plan
+ .files
+ .keys()
+ .map(String::as_str)
+ .chain(plan.junk_files.iter().map(|junk| junk.file_key.as_str()))
+ .collect();
+ // Path identity, not just content identity. An in-place EDIT gives the
+ // same file a new content key, so a key-only comparison reads the
+ // superseded key as vanished and the new one as added — the SAME path
+ // listed as both removed and added. That is a replacement, and the file
+ // is still there to be republished, so it must never be refused.
+ // Ordinary resume already reaches this conclusion by mapping each file
+ // onto its planned key (`select_resume_plan_keys`); doing it here makes
+ // `--fresh`, which has no plan to map through, agree — without making
+ // the gate more fatal than the open it precedes.
+ let current_rels: std::collections::HashSet<&str> =
+ files.iter().map(|file| file.rel.as_str()).collect();
+ let current_path_ids: std::collections::HashSet<&str> = files
+ .iter()
+ .map(|file| file.rel_id.as_str())
+ .filter(|id| !id.is_empty())
+ .collect();
+ let path_survives = |assignment: &FileAssignment| {
+ current_rels.contains(assignment.rel.as_str())
+ || (!assignment.path_id.is_empty()
+ && current_path_ids.contains(assignment.path_id.as_str()))
+ };
+
+ let mut added_content_groups: Vec<InventoryDeltaEntry> = files
+ .iter()
+ .zip(keys)
+ .filter(|(_, key)| !key.is_empty() && !durable_keys.contains(key.as_str()))
+ .map(|(file, key)| InventoryDeltaEntry {
+ file_key: key.clone(),
+ path: file.rel.clone(),
+ })
+ .collect();
+ let mut vanished_content_groups: Vec<InventoryDeltaEntry> = plan
+ .files
+ .iter()
+ .filter(|(key, assignment)| {
+ !current_keys.contains(key.as_str()) && !path_survives(assignment)
+ })
+ .map(|(key, assignment)| InventoryDeltaEntry {
+ file_key: key.clone(),
+ path: assignment.rel.clone(),
+ })
+ .collect();
+ // Deliberately NOT extended with `plan.junk_files`. A junk/skipped file
+ // published no documents, no aliases and no graph edges — its entire
+ // live footprint is one `file:{key}` catalog row, and the stale
+ // junk-catalog sweep below (#238) deletes that row before dropping the
+ // plan entry. Removing it therefore strands nothing, so refusing the
+ // rerun would block a case the pipeline now handles completely. Junk
+ // keys stay in `durable_keys` above: an unchanged skipped file is still
+ // not an addition.
+
+ let stable_order = |left: &InventoryDeltaEntry, right: &InventoryDeltaEntry| {
+ left.path
+ .cmp(&right.path)
+ .then_with(|| left.file_key.cmp(&right.file_key))
+ };
+ added_content_groups.sort_by(stable_order);
+ vanished_content_groups.sort_by(stable_order);
+ Self {
+ added_content_groups,
+ vanished_content_groups,
+ }
+ }
+
+ /// A rerun is refused only when a content group the plan published has
+ /// vanished from the folder: those documents stay live and searchable with
+ /// no source file behind them, and nothing in this pipeline removes them.
+ /// Additions are not refused — they are skipped by the frozen plan exactly
+ /// as before and `--fresh` rebuilds the plan in place to include them.
+ fn refuses(&self) -> bool {
+ !self.vanished_content_groups.is_empty()
+ }
+
+ fn into_error(self, targets: RefusalTargets) -> anyhow::Error {
+ UnsupportedInventoryDeltaError {
+ delta: self,
+ targets,
+ }
+ .into()
+ }
+}
+
fn alias_keys_to_reindex(
previous: &[state::DuplicateFile],
current: &[state::DuplicateFile],
@@ -1838,7 +2158,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// Acquire state authority before discovery as well as hashing. A waiter
// must never classify a path snapshot taken while another owner was
// publishing or replacing the durable plan.
- let preflight = state::Journal::preflight(&state_dir, &root_str, &cfg.url, &cfg.prefix)?;
+ let preflight =
+ state::Journal::preflight(&state_dir, &root_str, &cfg.url, &cfg.prefix, cfg.fresh)?;
let genesis_recovery = preflight
.committed_manifest
.as_ref()
@@ -1872,6 +2193,18 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
}
.into());
}
+ if let Some(reason) = preflight.unreadable_plan.as_deref() {
+ // Only reachable under --fresh; without it the preflight would have
+ // returned this as an error. It goes through the progress surface, not
+ // a raw eprintln!, so `--progress none` (which `--quiet` selects) stays
+ // silent and `--progress json` stays one parseable stream (#241).
+ pr.note(&format!(
+ "autoindex: --fresh: the durable resume plan in {} could not be read ({reason}); \
+ rebuilding it from the current folder. Documents published for files that are no \
+ longer present cannot be identified from an unreadable plan and are not deleted.",
+ state_dir.join("journal.ndjson").display()
+ ));
+ }
// `--dry-run` has to be decided *before* the generated branches, not after
// them. Both of those branches publish and commit, and both return before
// control ever reaches the legacy `cfg.dry_run` check further down — so on
@@ -1884,11 +2217,16 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
if cfg.dry_run {
if let Some(pending) = &preflight.pending_sync {
println!("{}", serde_json::to_string_pretty(&pending.desired.plan)?);
- eprintln!(
+ // stdout is the RESULT, stderr is PROGRESS — so this explanation
+ // goes through the progress surface, never a bare `eprintln!`
+ // that `--progress none` could not silence and `--progress json`
+ // could not parse (#241).
+ pr.note(&format!(
"(dry run — nothing indexed; corpus generation {} is already sealed and pending \
replay from its own durable snapshot. Re-run without --dry-run to finish it.)",
pending.desired.generation
- );
+ ));
+ pr.finish(true, 0, "dry-run", &[]);
return Ok((0, None));
}
}
@@ -1921,7 +2259,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// `--progress json` stays one parseable stream (#241).
pr.note("autoindex: resumed and committed pending corpus generation from durable source");
let summary = finish_generated_run(&es, &mut journal, &cfg)?;
- return Ok((generated_exit_code(&summary), Some(summary)));
+ let code = generated_exit_code(&summary);
+ finish_generated_progress(&pr, code, &summary);
+ return Ok((code, Some(summary)));
}
// Totals are unknown until the walk returns, so this phase honestly
// reports `pct=unknown` and proves liveness with the clock alone.
@@ -2036,7 +2376,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
}
}
let summary = finish_generated_run(&es, &mut journal, &cfg)?;
- return Ok((generated_exit_code(&summary), Some(summary)));
+ let code = generated_exit_code(&summary);
+ finish_generated_progress(&pr, code, &summary);
+ return Ok((code, Some(summary)));
}
begin_non_graph_generation(
&es,
@@ -2050,7 +2392,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);
sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;
let summary = finish_generated_run(&es, &mut journal, &cfg)?;
- return Ok((generated_exit_code(&summary), Some(summary)));
+ let code = generated_exit_code(&summary);
+ finish_generated_progress(&pr, code, &summary);
+ return Ok((code, Some(summary)));
}
if cfg.no_graph && preflight.plan.is_some() && !genesis_recovery {
let replacement_state = state_dir.with_extension("generation-v1");
@@ -2102,6 +2446,32 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
rebuild_argv
);
}
+ if let Some(prior_plan) = preflight.plan.as_ref() {
+ let comparison_keys = if cfg.fresh {
+ inventory.keys.clone()
+ } else {
+ // Ordinary resume preserves planned-key identity for supported
+ // same-path replacement and legacy plans.
+ select_resume_plan_keys(
+ &inventory.files,
+ &inventory.keys,
+ prior_plan,
+ &state_dir.join("journal.ndjson"),
+ )?
+ .into_iter()
+ .zip(inventory.keys.iter())
+ .map(|(planned, current)| planned.unwrap_or_else(|| current.clone()))
+ .collect()
+ };
+ // `--fresh` is checked here too: discarding the plan does not delete
+ // the documents already published for a file that is now gone, so a
+ // removal is unsafe in place whether or not the plan is kept.
+ let delta =
+ UnsupportedInventoryDelta::between(&inventory.files, &comparison_keys, prior_plan);
+ if delta.refuses() {
+ return Err(delta.into_error(RefusalTargets::describe(&cfg, &state_dir, prior_plan)));
+ }
+ }
let mut journal = state::Journal::open_after_preflight(
preflight,
&root_str,
@@ -2340,6 +2710,22 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
plan
};
+ // Second gate, after key selection has settled: fail before index
+ // creation/mapping, replacement intent, graph invalidation,
+ // delete-by-query, bulk publication, refresh, or catalog writes when a
+ // canonical content group the plan published is gone from the folder.
+ // Continuing would leave those documents searchable with no source file.
+ // Additions, same-path replacement and crash repair remain supported.
+ //
+ // It runs before the #238 junk sweep below on purpose: a refused rerun
+ // must compute nothing and mutate nothing.
+ if resumed_with_plan {
+ let delta = UnsupportedInventoryDelta::between(&files, &keys, &plan);
+ if delta.refuses() {
+ return Err(delta.into_error(RefusalTargets::describe(&cfg, &state_dir, &plan)));
+ }
+ }
+
// ── stale junk-catalog sweep (#238) ──────────────────────────────────
//
// A junk/skipped file is never indexed and never enters the graph corpus
@@ -2391,7 +2777,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
let mut backend = sync_executor::EsSyncBackend::new(&es, &state_dir, cfg.bulk_mb << 20);
sync_executor::replay_pending_operations(&state_dir, &mut journal, &mut backend)?;
let summary = finish_generated_run(&es, &mut journal, &cfg)?;
- return Ok((generated_exit_code(&summary), Some(summary)));
+ let code = generated_exit_code(&summary);
+ finish_generated_progress(&pr, code, &summary);
+ return Ok((code, Some(summary)));
}
if plan
@@ -2528,9 +2916,9 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
rel: files[i].rel.clone(),
format: "unknown".into(),
status: "skipped".into(),
- reason: "not in the frozen resume plan; no destination change was made. For an \
- exact rebuild use a new --state-dir, a new --prefix, and, when graph \
- detection is enabled, a new --brain (or use --no-graph)"
+ reason: "not in the frozen resume plan, so nothing was published for it. Re-run \
+ with --fresh to rebuild the plan in place and include it (ids stay \
+ idempotent), or index it under a new --state-dir and --prefix"
.into(),
bytes: files[i].size,
});
@@ -4289,6 +4677,215 @@ mod unsafe_fresh_generation_tests {
}
}
+#[cfg(test)]
+mod inventory_delta_tests {
+ use super::*;
+ use std::path::PathBuf;
+
+ fn file(path: &str) -> walk::FileEntry {
+ walk::FileEntry {
+ path: PathBuf::from(path),
+ rel: path.to_owned(),
+ rel_id: format!("id:{path}"),
+ is_symlink: false,
+ size: 1,
+ }
+ }
+
+ fn assignment(path: &str) -> FileAssignment {
+ FileAssignment {
+ rel: path.to_owned(),
+ path_id: format!("id:{path}"),
+ family: "csv".into(),
+ gzip: false,
+ content_digest: Some(format!("digest:{path}")),
+ assignments: vec![(None, "rows".into())],
+ as_document: false,
+ is_symlink: Some(false),
+ }
+ }
+
+ fn targets() -> RefusalTargets {
+ RefusalTargets {
+ state_dir: "/state".into(),
+ data_indices: vec!["ax-rows".into()],
+ edges_index: Some(".xerj-memory-corpus-edges".into()),
+ }
+ }
+
+ #[test]
+ fn only_a_removed_content_group_refuses_a_rerun() {
+ let mut plan = Plan::default();
+ plan.files.insert("keep".into(), assignment("keep.csv"));
+ assert!(
+ !UnsupportedInventoryDelta::between(&[file("keep.csv")], &["keep".into()], &plan)
+ .refuses()
+ );
+ // An added file is skipped by the frozen plan, not a refusal: the
+ // documented rerun-then---fresh workflow has to keep working.
+ let added = UnsupportedInventoryDelta::between(
+ &[file("keep.csv"), file("new.csv")],
+ &["keep".into(), "new".into()],
+ &plan,
+ );
+ assert_eq!(added.added_content_groups.len(), 1);
+ assert!(!added.refuses(), "an addition alone must not fail the run");
+ // A removal leaves live documents with no source file behind them.
+ assert!(UnsupportedInventoryDelta::between(&[], &[], &plan).refuses());
+ }
+
+ #[test]
+ fn classifier_sorts_added_and_vanished_groups() {
+ let mut plan = Plan::default();
+ plan.files.insert("keep".into(), assignment("keep.csv"));
+ plan.junk_files.push(JunkFile {
+ file_key: "junk".into(),
+ rel: "broken.pdf".into(),
+ format: "pdf".into(),
+ status: "junk".into(),
+ reason: "fixture".into(),
+ bytes: 1,
+ });
+ assert!(
+ !UnsupportedInventoryDelta::between(
+ &[file("keep.csv"), file("broken.pdf")],
+ &["keep".into(), "junk".into()],
+ &plan,
+ )
+ .refuses(),
+ "unchanged durable junk is neither added nor vanished"
+ );
+
+ plan.files.insert("old-z".into(), assignment("z-old.csv"));
+ plan.files.insert("old-a".into(), assignment("a-old.csv"));
+ let delta = UnsupportedInventoryDelta::between(
+ &[
+ file("keep.csv"),
+ file("broken.pdf"),
+ file("z-new.csv"),
+ file("m-new.csv"),
+ ],
+ &["keep".into(), "junk".into(), "new-z".into(), "new-m".into()],
+ &plan,
+ );
+ assert_eq!(
+ delta
+ .added_content_groups
+ .iter()
+ .map(|entry| entry.path.as_str())
+ .collect::<Vec<_>>(),
+ ["m-new.csv", "z-new.csv"]
+ );
+ assert_eq!(
+ delta
+ .vanished_content_groups
+ .iter()
+ .map(|entry| entry.path.as_str())
+ .collect::<Vec<_>>(),
+ ["a-old.csv", "z-old.csv"]
+ );
+ }
+
+ #[test]
+ fn refusal_names_the_removed_files_and_every_recovery_route() {
+ let mut plan = Plan::default();
+ plan.files.insert("gone".into(), assignment("gone.csv"));
+ let error = UnsupportedInventoryDelta::between(&[], &[], &plan).into_error(targets());
+ let message = format!("{error:#}");
+ assert!(message.contains("gone.csv"), "{message}");
+ assert!(
+ message.contains("no longer exist in the folder"),
+ "{message}"
+ );
+ assert!(message.contains("made no remote mutations"), "{message}");
+ assert!(message.contains("restore the removed file(s)"), "{message}");
+ assert!(message.contains("ax-rows"), "{message}");
+ assert!(message.contains("/state"), "{message}");
+ assert!(message.contains(".xerj-memory-corpus-edges"), "{message}");
+ assert!(message.contains("new --state-dir"), "{message}");
+ assert!(message.contains("`--fresh`"), "{message}");
+ }
+
+ /// The prose refusal is bounded by REFUSAL_LIST_CAP, not by the corpus: an
+ /// unmounted bind mount under an indexed root vanishes every group at once,
+ /// and rendering 82k entries into one String is megabytes of stderr in the
+ /// one path whose job is to be read. `--json` still carries all of them.
+ #[test]
+ fn refusal_prose_caps_its_listings_while_json_keeps_every_entry() {
+ let mut plan = Plan::default();
+ let total = REFUSAL_LIST_CAP * 3;
+ for i in 0..total {
+ plan.files.insert(
+ format!("gone{i:03}"),
+ assignment(&format!("gone{i:03}.csv")),
+ );
+ }
+ let error = UnsupportedInventoryDelta::between(&[], &[], &plan).into_error(targets());
+ let message = format!("{error:#}");
+ assert!(message.contains("gone000.csv"), "{message}");
+ assert!(
+ message.contains(&format!(", … and {} more", total - REFUSAL_LIST_CAP)),
+ "{message}"
+ );
+ // The tail past the cap must not be rendered at all, not merely elided.
+ let last = format!("gone{:03}.csv", total - 1);
+ assert!(!message.contains(&last), "{message}");
+ assert_eq!(
+ message.matches(".csv (").count(),
+ REFUSAL_LIST_CAP,
+ "{message}"
+ );
+
+ let stdout = route_cli_error(&error, true)
+ .stdout
+ .expect("the typed refusal renders as JSON under --json");
+ let value: Value = serde_json::from_str(&stdout).unwrap();
+ assert_eq!(
+ value["vanished_content_groups"].as_array().unwrap().len(),
+ total
+ );
+ }
+
+ #[test]
+ fn refusal_prose_below_the_cap_has_no_more_tail() {
+ let mut plan = Plan::default();
+ plan.files.insert("gone".into(), assignment("gone.csv"));
+ let error = UnsupportedInventoryDelta::between(&[], &[], &plan).into_error(targets());
+ let message = format!("{error:#}");
+ assert!(!message.contains("… and "), "{message}");
+ }
+
+ #[test]
+ fn cli_error_routing_separates_typed_json_from_unrelated_human_errors() {
+ let typed = UnsupportedInventoryDelta {
+ added_content_groups: Vec::new(),
+ vanished_content_groups: vec![InventoryDeltaEntry {
+ file_key: "key".into(),
+ path: "gone.csv".into(),
+ }],
+ }
+ .into_error(targets());
+ let route = route_cli_error(&typed, true);
+ assert_eq!(route.exit_code, 1);
+ assert!(route.stderr.is_none());
+ let stdout = route.stdout.unwrap();
+ let value: Value = serde_json::from_str(&stdout).unwrap();
+ assert_eq!(value["schema"], "xerj.autoindex.unsupported_sync_delta.v1");
+ assert_eq!(value["error"], "unsupported_content_group_removal");
+ assert_eq!(value["vanished_content_groups"][0]["path"], "gone.csv");
+ assert!(value["recovery"]["rebuild_in_place"]
+ .as_str()
+ .unwrap()
+ .contains("ax-rows"));
+
+ let unrelated = anyhow::anyhow!("endpoint unavailable");
+ let route = route_cli_error(&unrelated, true);
+ assert_eq!(route.exit_code, 1);
+ assert!(route.stdout.is_none());
+ assert_eq!(route.stderr.as_deref(), Some("error: endpoint unavailable"));
+ }
+}
+
#[cfg(test)]
mod duplicate_integration_tests {
use super::*;
diff --git a/engine/crates/xerj-autoindex/src/reconcile_plan.rs b/engine/crates/xerj-autoindex/src/reconcile_plan.rs
index eee92c704..8cfd9150f 100644
--- a/engine/crates/xerj-autoindex/src/reconcile_plan.rs
+++ b/engine/crates/xerj-autoindex/src/reconcile_plan.rs
@@ -468,6 +468,11 @@ mod tests {
records: 1,
}],
junk: None,
+ // Reconciliation re-samples from the source tree and publishes
+ // through the sealed snapshot, so it never carries a run-local
+ // PDF artifact (#248 is a phase A→B accelerator).
+ pdf_spool: None,
+ pdf_spool_fallbacks: Vec::new(),
}
}
diff --git a/engine/crates/xerj-autoindex/src/state.rs b/engine/crates/xerj-autoindex/src/state.rs
index a089c101c..5bd21e261 100644
--- a/engine/crates/xerj-autoindex/src/state.rs
+++ b/engine/crates/xerj-autoindex/src/state.rs
@@ -202,21 +202,6 @@ mod tests {
}
}
- #[test]
- fn preflight_holds_the_same_exclusive_lock_through_authoritative_open() {
- let dir = tempfile::tempdir().unwrap();
- let preflight = Journal::preflight(dir.path(), "root", "url", "prefix").unwrap();
- let error = Journal::open(dir.path(), "root", "url", "prefix", 300, false)
- .err()
- .expect("a second opener must not pass the held preflight lock");
- assert!(format!("{error:#}").contains("already in use"));
-
- let journal =
- Journal::open_after_preflight(preflight, "root", "url", "prefix", 300, false).unwrap();
- drop(journal);
- assert!(Journal::open(dir.path(), "root", "url", "prefix", 300, false).is_ok());
- }
-
/// The preflight must never be more fatal than the open it precedes.
/// `open_after_preflight` deletes the journal unread when `--fresh` is set,
/// so a preflight that hard-errors while parsing that journal would make
@@ -534,10 +519,14 @@ fn read_plan_for_preflight(
"journal corruption at byte {record_start} in {}: malformed \
newline-terminated record. Refusing to discard later records. Restore \
the journal from a backup, or truncate it to exactly {record_start} \
- bytes to keep every completion recorded before the corruption. For an \
- isolated rebuild use a new --state-dir, new --prefix, and new --brain \
- when graph detection is enabled (or --no-graph); explicitly validate \
- and clean the shared catalog and old target",
+ bytes to keep every completion recorded before the corruption. Deleting \
+ the whole journal (or rerunning with --fresh) also recovers, but \
+ re-extracts and re-embeds the entire corpus — and it discards any \
+ durable generation records in this journal along with it, which is \
+ why --fresh is reachable here at all. For an isolated rebuild use a \
+ new --state-dir, new --prefix, and new --brain when graph detection \
+ is enabled (or --no-graph); explicitly validate and clean the shared \
+ catalog and old target",
path.display()
)
})?;
@@ -608,14 +597,25 @@ pub struct JournalPreflight {
pub pending_sync: Option<crate::sync::PendingSync>,
pub legacy_migration_reasons: Vec<String>,
pub journal_exists: bool,
+ /// Set only under `--fresh`, when the durable plan could not be read and
+ /// the run is about to discard the journal anyway. The caller owns the
+ /// decision to print it (autoindex honours `--quiet`).
+ pub unreadable_plan: Option<String>,
}
impl Journal {
+ /// `fresh` is not a formality here. `open_after_preflight` removes the
+ /// journal before replaying it when `--fresh` is set, so nothing inside a
+ /// journal can be fatal to that path — and a preflight that hard-errors on
+ /// a corrupt record or a root/url/prefix mismatch would make `--fresh`
+ /// unreachable in exactly the cases whose error text recommends it. The
+ /// preflight must therefore never be more fatal than the open it precedes.
pub fn preflight(
state_dir: &Path,
root: &str,
url: &str,
prefix: &str,
+ fresh: bool,
) -> Result<JournalPreflight> {
std::fs::create_dir_all(state_dir)
.with_context(|| format!("create state dir {}", state_dir.display()))?;
@@ -635,8 +635,26 @@ impl Journal {
})?;
let journal_path = state_dir.join("journal.ndjson");
let journal_exists = journal_path.exists();
- let (plan, committed_manifest, pending_sync, legacy_migration_reasons) =
- read_plan_for_preflight(&journal_path, root, url, prefix)?;
+ let (plan, committed_manifest, pending_sync, legacy_migration_reasons, unreadable_plan) =
+ match read_plan_for_preflight(&journal_path, root, url, prefix) {
+ Ok((plan, committed, pending, reasons)) => {
+ (plan, committed, pending, reasons, None)
+ }
+ // The journal is about to be deleted unread. Losing the plan here
+ // costs the removed-file gate its comparison basis, which is
+ // exactly what a full in-place rebuild accepts; the alternative is
+ // a state directory that no supported flag can recover.
+ //
+ // Every generated record is discarded with it, so the durable
+ // generation this run might otherwise have refused to overwrite is
+ // reported as absent rather than half-parsed. That is what keeps
+ // `--fresh` genuinely reachable here: the caller's generation
+ // refusal reads `committed_manifest`/`pending_sync`, and firing it
+ // off a partially-replayed journal would contradict the recovery
+ // this very error recommends.
+ Err(error) if fresh => (None, None, None, Vec::new(), Some(format!("{error:#}"))),
+ Err(error) => return Err(error),
+ };
Ok(JournalPreflight {
state_dir: state_dir.to_owned(),
state_lock,
@@ -645,6 +663,7 @@ impl Journal {
pending_sync,
legacy_migration_reasons,
journal_exists,
+ unreadable_plan,
})
}
@@ -656,7 +675,7 @@ impl Journal {
bulk_timeout_secs: u64,
fresh: bool,
) -> Result<Journal> {
- let preflight = Self::preflight(state_dir, root, url, prefix)?;
+ let preflight = Self::preflight(state_dir, root, url, prefix, fresh)?;
Self::open_after_preflight(preflight, root, url, prefix, bulk_timeout_secs, fresh)
}
@@ -732,10 +751,9 @@ impl Journal {
anyhow::bail!(
"journal at {} was created for root={jr} url={ju} prefix={jp}; \
current run has root={root} url={url} prefix={prefix}. \
- Refusing to discard history under the same destination. For an \
- isolated rebuild use a new --state-dir, new --prefix, and new \
- --brain when graph detection is enabled (or --no-graph); \
- explicitly validate and clean the shared catalog and old target.",
+ Use --state-dir for separate state, or --fresh to rebuild the \
+ plan in place — note that --fresh never deletes documents \
+ already published under the other root, url or prefix.",
jpath.display()
);
}
@@ -1452,7 +1470,7 @@ mod sync_journal_tests {
.err()
.unwrap();
assert!(format!("{error:#}").contains("while a sync is pending"));
- let preflight = Journal::preflight(dir.path(), "root", "url", "prefix")
+ let preflight = Journal::preflight(dir.path(), "root", "url", "prefix", false)
.err()
.unwrap();
assert!(format!("{preflight:#}").contains("while a sync is pending"));
@@ -1530,7 +1548,7 @@ mod sync_journal_tests {
let mut journal = journal_with_plan(dir.path());
journal.sync_begin(&pending(&journal)).unwrap();
drop(journal);
- let preflight = Journal::preflight(dir.path(), "root", "url", "prefix").unwrap();
+ let preflight = Journal::preflight(dir.path(), "root", "url", "prefix", false).unwrap();
assert_eq!(preflight.committed_manifest.as_ref().unwrap().generation, 0);
assert_eq!(preflight.pending_sync.as_ref().unwrap().tx_id, "sync-1");
drop(preflight);
@@ -1623,6 +1641,7 @@ mod sync_journal_tests {
records: 1,
junk: 0,
bytes: 10,
+ dropped_by_dataset: std::collections::HashMap::new(),
generation: None,
})
.unwrap();
diff --git a/engine/crates/xerj-autoindex/src/sync_executor.rs b/engine/crates/xerj-autoindex/src/sync_executor.rs
index 2ee8f6543..1a621a83c 100644
--- a/engine/crates/xerj-autoindex/src/sync_executor.rs
+++ b/engine/crates/xerj-autoindex/src/sync_executor.rs
@@ -1721,7 +1721,7 @@ mod tests {
use std::collections::BTreeMap;
fn inventory(root: &Path) -> Inventory {
- crate::content::resolve(walk::walk(root, false).unwrap()).unwrap()
+ crate::content::resolve_reporting(walk::walk(root, false).unwrap(), &|_| {}).unwrap()
}
fn plan_for(inventory: &Inventory) -> Plan {
@@ -2045,8 +2045,11 @@ mod tests {
fn prepared_snapshot_budget_aborts_and_removes_staging() {
let corpus = tempfile::tempdir().unwrap();
std::fs::write(corpus.path().join("a.txt"), "budgeted source bytes").unwrap();
- let inventory =
- crate::content::resolve(crate::walk::walk(corpus.path(), false).unwrap()).unwrap();
+ let inventory = crate::content::resolve_reporting(
+ crate::walk::walk(corpus.path(), false).unwrap(),
+ &|_| {},
+ )
+ .unwrap();
let plan = plan_for(&inventory);
let state = tempfile::tempdir().unwrap();
let error = create_prepared_snapshot(
@@ -2116,8 +2119,11 @@ mod tests {
.unwrap_or_else(|poisoned| poisoned.into_inner());
let corpus = tempfile::tempdir().unwrap();
std::fs::write(corpus.path().join("a.txt"), "protected bytes").unwrap();
- let inventory =
- crate::content::resolve(crate::walk::walk(corpus.path(), false).unwrap()).unwrap();
+ let inventory = crate::content::resolve_reporting(
+ crate::walk::walk(corpus.path(), false).unwrap(),
+ &|_| {},
+ )
+ .unwrap();
let plan = plan_for(&inventory);
let state = tempfile::tempdir().unwrap();
let snapshot = create_snapshot(state.path(), "tx-protected", &inventory).unwrap();
diff --git a/landing/docs/recipes/zero-config-autoindex.html b/landing/docs/recipes/zero-config-autoindex.html
index 625485b98..9ccca2fe9 100644
--- a/landing/docs/recipes/zero-config-autoindex.html
+++ b/landing/docs/recipes/zero-config-autoindex.html
@@ -239,7 +239,8 @@ live indices at http://localhost:9280:
ax-docs 1 docs
ax-exports 800 docs
autoindex-catalog 9 docs</pre>
- <p><code class="inline">--fresh</code> ignores the resume journal and restarts (ids stay idempotent). It never removes stale records from the destination, and it is refused once the state directory holds a durable corpus generation — re-run without it and the generated <code class="inline">--no-graph</code> path reconciles additions, changes, deletions, renames and no-op reruns incrementally. For an independent rebuild, use a new <code class="inline">--state-dir</code> and a new <code class="inline">--prefix</code>, plus a new <code class="inline">--brain</code> when graph detection is enabled (or add <code class="inline">--no-graph</code>). Validate before switching readers; the shared <code class="inline">autoindex-catalog</code> and old target require explicit, validated cleanup. A <code class="inline">--no-graph</code> state directory written before the generation format cannot be adopted in place and must be rebuilt the same way.</p>
+ <p><code class="inline">--fresh</code> ignores the resume journal and restarts (ids stay idempotent), which on a graph-enabled or pre-generation state directory is how a rerun picks up files added since the last one. It never removes stale records from the destination, and it is refused once the state directory holds a durable corpus generation — re-run without it and the generated <code class="inline">--no-graph</code> path reconciles additions, changes, deletions, renames and no-op reruns incrementally.</p>
+ <p>On a graph-enabled or pre-generation journal, a rerun that resumes an existing plan indexes changed files and reports added ones as skipped. Deleting a file is refused there: nothing on that path removes the documents it already published, so the rerun stops before touching the destination and names what is gone — the first ten, then an <code class="inline">… and N more</code> tail, with <code class="inline">--json</code> carrying every entry. Restore the file and rerun, or rebuild — in place by deleting the named indices and the state directory, or isolated under a new <code class="inline">--state-dir</code>, <code class="inline">--prefix</code> and <code class="inline">--brain</code> (or <code class="inline">--no-graph</code>), validated before you switch readers. The shared <code class="inline">autoindex-catalog</code> and the old target require explicit cleanup. A <code class="inline">--no-graph</code> state directory written before the generation format cannot be adopted in place and must be rebuilt the same way.</p>
<hr>
diff --git a/landing/llms-full.txt b/landing/llms-full.txt
index 6145ed2eb..282110ffb 100644
--- a/landing/llms-full.txt
+++ b/landing/llms-full.txt
@@ -140,19 +140,28 @@ destabilize the server.
xerj autoindex map [--url U] [--json] [--dataset SLUG]
xerj autoindex status [--url U] [--state-dir P]
---fresh ignores the resume journal and restarts (ids stay idempotent). It never
-removes stale destination records, and it is refused once the state directory
-holds a durable corpus generation — re-run without it and the generated
---no-graph path reconciles additions, changes, deletions, renames and no-op
-reruns incrementally. An independent rebuild requires a new --state-dir and a
-new --prefix, plus a new --brain when graph detection is enabled (or
---no-graph), followed by validation before switching readers. The shared
-autoindex-catalog and old target require explicit, validated cleanup. A
+--fresh ignores the resume journal and restarts (ids stay idempotent); on a
+graph-enabled or pre-generation state directory that is how you pick up files
+added since the last run. It never removes stale destination records, and it is
+refused once the state directory holds a durable corpus generation — re-run
+without it and the generated --no-graph path reconciles additions, changes,
+deletions, renames and no-op reruns incrementally.
+
+On a graph-enabled or pre-generation journal, a rerun that resumes an existing
+plan indexes changed files and reports added ones as skipped. Deleting an
+indexed file is refused there before any destination change, because nothing on
+that path removes the documents it already published: restore the file and
+rerun, or rebuild — in place by deleting the named indices and the state
+directory, or isolated under a new --state-dir, --prefix and --brain (or
+--no-graph), validated before switching readers. The refusal names the first ten
+removed files with an "… and N more" tail; --json carries every entry. The
+shared autoindex-catalog and the old target require explicit cleanup. A
--no-graph state directory written before the generation format cannot be
adopted in place and must be rebuilt the same way.
Exit codes: 0 complete · 3 completed-with-junk (junk recorded, never fatal) ·
-2 usage · 1 endpoint/journal error or refused unsafe corpus-state transition.
+2 usage · 1 endpoint/journal error, a refused corpus removal, or a refused
+unsafe corpus-state transition.
Pipeline: walk → sniff → sample → infer → map → extract → bulk → correlate →
catalog → verify.