AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): refuse only the reruns that strand live documents
xerj-org3049723e06e4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 191e86fa5..26338783f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -529,22 +529,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`/v1/embedding/identity` reports `lexical` — the backend the server willreally use — rather than `proxy`.-### Changed — fail-closed autoindex restart semantics--- **`autoindex --fresh` no longer discards a durable resume plan.** A durable- plan contains the alias, path, graph, and stale-record knowledge needed for- safe reconciliation. `--fresh` is accepted only when the selected state- directory has no durable plan and never cleans the destination. An- independent rebuild must use a new state directory, new prefix, and new brain- namespace when graph detection is enabled (or disable graph writes), be- validated, and only then replace the old reader target. The shared catalog- and old target require explicit, validated cleanup. Membership additions and- removals are refused; same-path replacement remains supported.+### Changed — autoindex refuses reruns that would strand documents++- **Deleting an indexed file and rerunning `xerj autoindex` now fails instead+ of exiting 0.** Nothing in the pipeline removes the documents, aliases,+ graph edges or catalog entries that the deleted file published, so a rerun+ that ignored the deletion left them live and searchable with no source file+ behind them. The rerun is now refused before any remote call other than the+ endpoint-readiness ping: no mapping, delete-by-query, bulk, refresh, graph+ or catalog write is attempted, and the journal is not appended to. The error+ names every removed file and its content key and gives three recovery+ routes — restore the files and rerun, rebuild in place by deleting the named+ indices and the state directory, or rebuild under a new state directory,+ prefix and brain. `--fresh` is refused for the same case, because it does+ not delete those documents either. `--json` emits the same facts as+ `xerj.autoindex.unsupported_sync_delta.v1` on stdout with exit 1.+- **Files added after a plan was frozen are now called out on stderr.** They+ are still not indexed by a rerun that resumes an existing plan — the plan is+ a crash-resume boundary, not a folder-sync generation — but the run now+ lists them, records them as skipped (exit 3, completed-with-junk) and points+ at `--fresh`, which rebuilds the plan in place and picks them up. Adding or+ changing files and rerunning keeps working; only removals are refused.- **`xerj brain` no longer turns an absent or zero node-count probe into anautomatic reset.** Journal/server disagreement now fails with the journal,- URL, prefix, and brain identity plus recovery instructions. Probe transport- and malformed-response failures remain errors instead of being reported as- an empty destination.+ URL, prefix, and brain identity plus recovery instructions, including the+ explicit `--fresh` rerun for a genuinely wiped data directory. Probe+ transport and malformed-response failures remain errors instead of being+ reported as an empty destination.## [1.0.0-rc.10] - 2026-08-03diff --git a/demo/usecases/autoindex/RECIPE_DRAFT.md b/demo/usecases/autoindex/RECIPE_DRAFT.mdindex faa1b6ec8..023080bac 100644--- a/demo/usecases/autoindex/RECIPE_DRAFT.md+++ b/demo/usecases/autoindex/RECIPE_DRAFT.md@@ -203,13 +203,11 @@ done in 0.1s — 3 datasets, 5801 records live, 0 junk records, 1 junk/skipped fjust polite re-runs: in the robustness evaluation, a `kill -9` at 18 s into a200 MB file followed by a re-run converged to **byte-identical counts acrossall 9 datasets** — no duplicates. `xerj autoindex status` shows the journal-and live index counts. `--fresh` starts without resume state only when the-selected state directory has no durable plan; it never removes stale-destination records. For an independent rebuild, use a new `--state-dir`, new-`--prefix`, and new `--brain` when graph detection is enabled (or add-`--no-graph`). Validate before switching readers. The shared-`autoindex-catalog` and old target require explicit, validated cleanup. Added-or removed content groups are refused; same-path replacement remains supported.+and live index counts; `--fresh` ignores the journal and rebuilds the plan in+place (ids stay idempotent), which is how a rerun picks up files added since+the last one. Deleting an indexed file is refused before any destination+change — nothing here removes the documents it already published — and the+error names what is gone plus the recovery routes.## Reproduce it yourselfdiff --git a/demo/usecases/autoindex/run-eval.sh b/demo/usecases/autoindex/run-eval.shindex dc36277ec..d7dff87ef 100755--- a/demo/usecases/autoindex/run-eval.sh+++ b/demo/usecases/autoindex/run-eval.sh@@ -2,9 +2,11 @@# End-to-end eval of `xerj autoindex` against a corpus folder.## Boots a dedicated server on es_compat $PORT (rest PORT+100, grpc PORT+101),-# runs autoindex over $CORPUS, proves same-state resume, then performs an-# isolated data/graph rebuild under a new state directory, prefix, and brain.-# The global autoindex-catalog remains shared and is excluded from comparison.+# runs autoindex over $CORPUS, then re-runs twice to prove idempotency — the+# resume path (all files done) and a full --fresh re-extract — and requires the+# per-index doc counts to be byte-identical to run 1. Ends with the data map.+# The shared autoindex-catalog is excluded from the comparison: every run+# appends its own run record, so a growing catalog is the design, not drift.## Nothing here is bound to one machine: every path is an env override with a# default under the repo or $TMPDIR, so the same script gates CI and drives a@@ -33,20 +35,11 @@ CFG="$WORK/server-$PORT.toml"LOG="$WORK/server-$PORT.log"URL="http://localhost:$PORT"STATE="$WORK/state"-REBUILD_STATE="$WORK/rebuild-state"-PREFIX="ax-eval-a"-REBUILD_PREFIX="ax-eval-b"-BRAIN="eval-a"-REBUILD_BRAIN="eval-b"+PREFIX="ax-eval"+BRAIN="eval"[ -x "$BIN" ] || { echo "xerj binary not found at $BIN"; exit 1; }[ -d "$CORPUS" ] || { echo "corpus folder not found at $CORPUS"; exit 1; }-[ ! -e "$DATA" ] && [ ! -e "$STATE" ] && [ ! -e "$REBUILD_STATE" ] || {- echo "work directory already contains eval state: $WORK"- echo "choose a new XERJ_AUTOINDEX_WORK or explicitly remove the old isolated eval directory"- exit 1-}-mkdir -p "$WORK"cat >"$CFG" <<EOF[server]@@ -85,10 +78,10 @@ if [ "$RC" != "0" ] && [ "$RC" != "3" ]; thenFAIL=1fi-# Name + doc count only. The _cat row also carries on-disk size, which an-# isolated data/graph rebuild legitimately changes (segments are rewritten).-# The shared global catalog is excluded because every run appends its own run-# record — a growing catalog is the design, not drift. Columns of the _cat row:+# Name + doc count only. The _cat row also carries on-disk size, which a+# --fresh re-extract legitimately changes (segments are rewritten), and the+# catalog is excluded outright because every run appends its own run record —+# a growing catalog is the design, not drift. Columns of the _cat row:# green(1) open(2) name(3) uuid(4) pri(5) rep(6) docs(7).counts() { curl -s "$URL/_cat/indices" | awk -v prefix="$1-" '$3 ~ ("^" prefix) {sub("^" prefix, "", $3); print $3, $7}' | sort; }@@ -104,16 +97,16 @@ if [ "$RC" != "0" ] && [ "$RC" != "3" ]; thenFAIL=1fi-echo "=== isolated data/graph rebuild reproducibility (new state, prefix, and brain; shared catalog excluded) ==="-time "$BIN" autoindex "$CORPUS" --url "$URL" --state-dir "$REBUILD_STATE" --prefix "$REBUILD_PREFIX" --brain "$REBUILD_BRAIN"+echo "=== idempotency: re-run with --fresh (full re-extract, idempotent ids) ==="+time "$BIN" autoindex "$CORPUS" --url "$URL" --state-dir "$STATE" --prefix "$PREFIX" --brain "$BRAIN" --freshRC=$?if [ "$RC" != "0" ] && [ "$RC" != "3" ]; then- echo "isolated data/graph rebuild exited $RC (expected 0 or 3)"+ echo "--fresh re-extract exited $RC (expected 0 or 3)"FAIL=1fi-counts "$REBUILD_PREFIX" > "$WORK/counts-run3.txt"-echo "=== normalized data-index count diff original vs isolated data/graph rebuild (shared catalog excluded; must be empty) ==="+counts "$PREFIX" > "$WORK/counts-run3.txt"+echo "=== doc-count diff run1 vs run3 (must be empty) ==="cat "$WORK/counts-run3.txt"if diff "$WORK/counts-run1.txt" "$WORK/counts-run3.txt"; thenecho "IDENTICAL COUNTS ✓"diff --git a/docs/design/second_brain_graph_wf.js b/docs/design/second_brain_graph_wf.jsindex 404b5d591..3225781a3 100644--- a/docs/design/second_brain_graph_wf.js+++ b/docs/design/second_brain_graph_wf.js@@ -99,9 +99,8 @@ heterogeneous content (code + PDFs + docs), and captured as a screencast.automaticPresenceSimulation:true})\`, then claim via the setup magic-link the server printsat first boot. A working reference script is ${REPO}/sb_capture.mjs.- \`xerj brain\` gotcha: a stale resume journal in \`~/.xerj/autoindex\` makes re-runs skip- detection. For an independent clean run, choose a new \`--state-dir\`, new- \`--prefix\`, and new \`--brain\` (or \`--no-graph\`); \`--fresh\` cannot- discard a durable plan or clean the shared catalog or old target.+ detection. Use \`--fresh\` (or clear it) for clean runs; note that \`--fresh\` rebuilds+ the plan in place and never removes documents for notes you deleted.- Commits authored \`xerj-org <[email protected]>\`, NO Claude co-author trailer, never mentionctrl-frk. Do NOT push. Commit on \`feat/second-brain\`.diff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.mdindex d63a24e1b..b75f062df 100644--- a/docs/recipes/zero-config-indexing.md+++ b/docs/recipes/zero-config-indexing.md@@ -228,13 +228,17 @@ live indices at http://localhost:9280:autoindex-catalog 9 docs```-`--fresh` starts without resume state only when the selected state directory-has no durable plan; it never removes stale destination records. For an-independent rebuild, use a new `--state-dir`, new `--prefix`, and new-`--brain` when graph detection is enabled (or add `--no-graph`). Validate-before switching readers. The shared `autoindex-catalog` and old target require-explicit, validated cleanup. Added or removed content groups are refused;-same-path replacement remains supported.+`--fresh` ignores the journal and rebuilds the plan in place (ids stay+idempotent), which is how you pick up files added since the last run.++A rerun that resumes an existing plan indexes changed files and reports added+ones as skipped. Deleting a file is refused: nothing here removes the+documents it already published, so the rerun stops before touching the+destination and names what is gone. Restore the file and rerun, or rebuild —+in place by deleting the named indices and the state directory, or isolated+under a new `--state-dir`, `--prefix` and `--brain` (or `--no-graph`),+validated before you switch readers. The shared `autoindex-catalog` and the+old target require explicit cleanup.## Reproduce it yourselfdiff --git a/docs/usecases/second-brain/README.md b/docs/usecases/second-brain/README.mdindex 41c74feec..0e181b66f 100644--- a/docs/usecases/second-brain/README.md+++ b/docs/usecases/second-brain/README.md@@ -205,12 +205,13 @@ other note tools. No such numbers are claimed.root-caused.- **YAML-frontmatter markdown can sniff as YAML:** one `SKILL.md` withfrontmatter landed in junk/skipped instead of being indexed.-- **Stale resume journals can pollute links** after a wiped data dir;- `--fresh` cannot discard a durable plan and is not stale-journal recovery.- Restore the server data directory used by the journal. For an explicitly- isolated rebuild, use a new state directory, prefix, and brain namespace,- validate it, then switch readers. The shared `autoindex-catalog` and old- target require explicit, validated cleanup.+- **Stale resume journals can pollute links** after a wiped data dir. `xerj+ brain` no longer guesses: it reports the disagreement between the journal+ and the server instead of resetting on its own. If the data directory really+ was wiped, rerun with `--fresh` to rebuild the plan and republish every note+ in place; if it was not, point at the data directory the journal was written+ against, or rebuild under a new state directory, prefix and brain and+ validate before switching readers.- **Wikilink detection was not exercised on this corpus** (real docs usemarkdown links); it is covered by the demo-corpus run (30 wikilinks) andthe autoindex test suite.diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex e225b4732..4136e817a 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -72,9 +72,8 @@ pub enum Cmd {Help,}-const FRESH_HELP: &str =- "start without resume state only when the selected state directory has no \-durable plan; an existing plan is refused and destination records are never reset";+const FRESH_HELP: &str = "ignore the existing journal and rebuild the plan in place\n\+ (ids stay idempotent) — see RESUME POLICY";pub fn print_help() {println!(@@ -171,14 +170,16 @@ pub fn print_help() {\n\RESUME POLICY:\n\A durable plan supports no-op resume and same-path content replacement.\n\- Added or removed content groups are refused before remote mutation.\n\- An independent rebuild needs a new --state-dir, new --prefix, and, when\n\- graph detection is enabled, new --brain (or --no-graph). Validate before\n\- switching readers; the shared autoindex-catalog and old target require\n\- explicit, validated cleanup.\n\+ Files added after the plan was frozen are reported as skipped and are not\n\+ indexed; --fresh rebuilds the plan in place and picks them up.\n\+ Removing an indexed file is refused before any remote mutation: its\n\+ documents are already live and nothing here deletes them. Restore the file\n\+ and rerun, or rebuild — in place by deleting the published indices and the\n\+ state directory, or isolated under a new --state-dir, --prefix and --brain\n\+ (or --no-graph), validated before you switch readers.\n\\n\EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\- 2 usage; 1 endpoint/journal failure or unsupported corpus delta\n",+ 2 usage; 1 endpoint/journal failure or a refused corpus removal\n",fresh_help = FRESH_HELP);}@@ -476,10 +477,10 @@ mod tests {}#[test]- fn fresh_help_warns_that_it_is_not_destination_reconciliation() {- assert!(super::FRESH_HELP.contains("no durable plan"));- assert!(super::FRESH_HELP.contains("existing plan is refused"));- assert!(super::FRESH_HELP.contains("destination records are never reset"));+ fn fresh_help_points_at_the_resume_policy_that_bounds_it() {+ assert!(super::FRESH_HELP.contains("rebuild the plan in place"));+ assert!(super::FRESH_HELP.contains("ids stay idempotent"));+ assert!(super::FRESH_HELP.contains("RESUME POLICY"));}#[test]diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rsindex 74188e21c..bf700acce 100644--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs@@ -532,11 +532,15 @@ fn assert_unsupported_delta_without_remote_mutation(let message = format!("{error:#}");assert!(message.contains("made no remote mutations"), "{message}");assert!(- message.contains("existing destination may already be partial or stale"),+ message.contains("no longer exist in the folder"),"{message}");assert!(- message.contains("`--fresh` is not recovery or destination reconciliation"),+ message.contains("restore the removed file(s) and rerun"),+ "{message}"+ );+ assert!(+ message.contains("rebuild in place by deleting the indices"),"{message}");assert!(@@ -558,19 +562,52 @@ fn assert_unsupported_delta_without_remote_mutation(}}+/// The documented headline workflow: point autoindex at a folder, add a file,+/// rerun. The rerun must not fail, must say plainly that the added file was+/// not indexed, and `--fresh` must then absorb it in place.#[test]-fn completed_plan_rejects_added_content_group_before_remote_mutation() {+fn a_rerun_after_an_added_file_succeeds_and_fresh_absorbs_it() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();let state_dir = tempfile::tempdir().unwrap();fs::write(corpus.path().join("first.csv"), "id,value\n1,first\n").unwrap();let endpoint = MockEndpoint::start(usize::MAX);- let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);assert_eq!(run_index(config.clone()).unwrap(), 0);fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();- assert_unsupported_delta_without_remote_mutation(&endpoint, config, &["second.csv"], &[]);+ // 3 = completed-with-junk: the added file is reported as skipped, not+ // indexed, because the frozen plan cannot absorb it.+ let (code, report) = run_index_report(config.clone()).unwrap();+ assert_eq!(code, 3);+ let report = report.unwrap();+ assert_eq!(report["files_junk"], 1, "{report}");+ assert_eq!(report["records_total"], 0, "{report}");+ {+ let locked = endpoint.state.lock().unwrap();+ let live: Vec<&str> = locked+ .docs+ .values()+ .filter_map(|doc| doc["ax_path"].as_str())+ .collect();+ assert!(+ live.iter().all(|path| *path == "first.csv"),+ "the added file must not be published by a plan that predates it: {live:?}"+ );+ }++ // --fresh rebuilds the plan in place and picks the new file up.+ config.fresh = true;+ assert_eq!(run_index(config).unwrap(), 0);+ let locked = endpoint.state.lock().unwrap();+ let indexed: std::collections::HashSet<&str> = locked+ .docs+ .values()+ .filter_map(|doc| doc["ax_path"].as_str())+ .collect();+ assert!(indexed.contains("first.csv"), "{indexed:?}");+ assert!(indexed.contains("second.csv"), "{indexed:?}");}#[test]@@ -636,7 +673,7 @@ fn completed_plan_rejects_mixed_membership_delta_in_stable_order() {}#[test]-fn nonempty_fresh_cannot_erase_plan_and_bypass_membership_gate() {+fn fresh_cannot_erase_the_plan_and_bypass_the_removal_gate() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();@@ -653,7 +690,7 @@ fn nonempty_fresh_cannot_erase_plan_and_bypass_membership_gate() {}#[test]-fn fresh_rejects_same_content_canonical_promotion_even_when_group_key_matches() {+fn deleting_one_path_of_a_duplicate_pair_is_not_a_removed_content_group() {let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();let corpus = tempfile::tempdir().unwrap();@@ -662,14 +699,24 @@ fn fresh_rejects_same_content_canonical_promotion_even_when_group_key_matches()fs::write(corpus.path().join("a.csv"), bytes).unwrap();fs::write(corpus.path().join("b.csv"), bytes).unwrap();let endpoint = MockEndpoint::start(usize::MAX);- let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);assert_eq!(run_index(config.clone()).unwrap(), 0);- // b.csv becomes canonical with the same content key. A key-only delta is- // empty, but fresh would discard the alias/path cleanup knowledge.+ // b.csv becomes canonical under the same content key: the group survives+ // the deletion, so no document is stranded and the rerun is allowed.fs::remove_file(corpus.path().join("a.csv")).unwrap();- config.fresh = true;- assert_unsupported_delta_without_remote_mutation(&endpoint, config, &[], &[]);+ let result = run_index(config);+ assert!(+ result.is_ok(),+ "a surviving content group must not trip the removal gate: {result:?}"+ );+ let locked = endpoint.state.lock().unwrap();+ let live: Vec<&str> = locked+ .docs+ .values()+ .filter_map(|doc| doc["ax_path"].as_str())+ .collect();+ assert_eq!(live, ["b.csv"], "canonical path follows the surviving file");}#[test]@@ -726,6 +773,7 @@ fn completed_plan_rejects_empty_current_folder_before_remote_mutation() {assert_eq!(run_index(config.clone()).unwrap(), 0);fs::remove_file(corpus.path().join("only.csv")).unwrap();+ assert_unsupported_delta_without_remote_mutation(&endpoint, config.clone(), &[], &["only.csv"]);// Even `--fresh` must not erase the only durable inventory evidence and// then report success while the destination still contains the document.config.fresh = true;@@ -1518,12 +1566,12 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {).unwrap();fs::write(corpus.path().join("b.csv"), original).unwrap();- assert_unsupported_delta_without_remote_mutation(&endpoint, config.clone(), &["b.csv"], &[]);+ assert_eq!(run_index(config.clone()).unwrap(), 3);{let locked = endpoint.state.lock().unwrap();assert_eq!(locked.docs.len(), 2);assert!(locked.docs.values().all(|doc| {- doc["ax_path"].as_str() == Some("a.csv") && doc["value"].as_str() == Some("original")+ doc["ax_path"].as_str() == Some("a.csv") && doc["value"].as_str() == Some("rewritten")}));}let replay = state::Journal::open(@@ -1538,20 +1586,24 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {assert!(replay.pending_replacements.is_empty());assert_eq!(replay.done.len(), 1);let plan = replay.plan.clone().unwrap();- assert!(plan.junk_files.is_empty());+ assert_eq!(plan.junk_files.len(), 1);+ assert_eq!(plan.junk_files[0].rel, "b.csv");+ assert!(plan.junk_files[0]+ .reason+ .contains("key ownership exclusive"));drop(replay);- // The refusal is deterministic: an identical rerun keeps the old owner,- // appends no new plan, and changes no documents.+ // The divergence is durable and deterministic: an identical rerun keeps+ // the same owner, appends no new plan, and changes no documents.let plans_before = event_count(state_dir.path(), "plan");- assert_unsupported_delta_without_remote_mutation(&endpoint, config, &["b.csv"], &[]);+ assert_eq!(run_index(config).unwrap(), 3);assert_eq!(event_count(state_dir.path(), "plan"), plans_before);let locked = endpoint.state.lock().unwrap();assert_eq!(locked.docs.len(), 2);assert!(locked.docs.values()- .all(|doc| doc["value"].as_str() == Some("original")));+ .all(|doc| doc["value"].as_str() == Some("rewritten")));}/// #238: a file added after the plan was frozen is skipped and reported in the@@ -1664,9 +1716,11 @@ fn deleting_an_entire_duplicate_group_strands_no_pending_replacement() {assert_eq!(run_index(config.clone()).unwrap(), 0);let starts = event_count(state_dir.path(), "file_replace_start");- // The whole duplicate group disappears. There is no current file left to- // republish its key, so no replacement intent may be journaled — a- // stranded intent would be re-appended forever without ever committing.+ // The whole duplicate group disappears: its documents stay live with no+ // source file, so the rerun is refused. The original invariant still+ // holds and is still checked — a key with no current file must never get+ // a replacement intent journaled, which would be re-appended forever+ // without ever committing.fs::remove_file(corpus.path().join("dup-a.csv")).unwrap();fs::remove_file(corpus.path().join("dup-b.csv")).unwrap();for _ in 0..2 {diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 618c2d6c0..688d3de31 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -1034,10 +1034,8 @@ fn select_resume_plan_keys(for (key, assignment) in &plan.files {if let Some(previous) = planned_by_rel.insert(&assignment.rel, key) {anyhow::bail!(- "resume plan assigns path {} to both {} and {}; refusing to discard history under \- the same destination. For an isolated rebuild use a new --state-dir, new \- --prefix, and new --brain when graph detection is enabled (or --no-graph); \- explicitly validate and clean the shared catalog and old target",+ "resume plan assigns path {} to both {} and {}; use --fresh after verifying the \+ existing index",assignment.rel,previous,key@@ -1046,11 +1044,8 @@ fn select_resume_plan_keys(if !assignment.path_id.is_empty() {if let Some(previous) = planned_by_path_id.insert(&assignment.path_id, key) {anyhow::bail!(- "resume plan assigns one native path identity to both {} and {}; refusing to \- discard history under the same destination. For an isolated rebuild use a \- new --state-dir, new --prefix, and new --brain when graph detection is \- enabled (or --no-graph); explicitly validate and clean the shared catalog \- and old target",+ "resume plan assigns one native path identity to both {} and {}; use --fresh \+ after verifying the existing index",previous,key);@@ -1088,11 +1083,9 @@ fn select_resume_plan_keys(anyhow::bail!("{} collides with legacy resume key {} already owned by {}. No documents \were changed; remove or move one of these two files out of the corpus \- and rerun — every other file keeps its resume state. Refusing to discard \- history under the same destination. For an isolated rebuild use a new \- --state-dir, new --prefix, and new --brain when graph detection is \- enabled (or --no-graph); explicitly validate and clean the shared \- catalog and old target. Journal: {}",+ and rerun — every other file keeps its resume state. Deleting the \+ journal at {} (or rerunning with --fresh) also clears the collision, \+ but re-extracts and re-embeds the entire corpus",file.rel,legacy_key,assignment.rel,@@ -1136,10 +1129,58 @@ struct UnsupportedInventoryDelta {vanished_content_groups: Vec<InventoryDeltaEntry>,}+/// Everything the refusal needs to name the destination it is protecting.+/// Collected at the gate so the message can list the exact indices and state+/// directory an operator has to act on, instead of describing them abstractly.+#[derive(Debug, Clone, PartialEq, Eq)]+struct RefusalTargets {+ state_dir: String,+ data_indices: Vec<String>,+ edges_index: Option<String>,+}++impl RefusalTargets {+ fn describe(cfg: &IndexCfg, state_dir: &Path, plan: &Plan) -> Self {+ let mut data_indices: Vec<String> = plan.datasets.iter().map(|d| d.index.clone()).collect();+ data_indices.sort();+ data_indices.dedup();+ let edges_index = (!cfg.no_graph).then(|| {+ let brain = cfg+ .brain+ .clone()+ .unwrap_or_else(|| derive_brain_name(&cfg.root));+ detect::edges_index_name(&brain)+ });+ Self {+ state_dir: state_dir.display().to_string(),+ data_indices,+ edges_index,+ }+ }++ fn indices_phrase(&self) -> String {+ if self.data_indices.is_empty() {+ "the indices this plan publishes".to_string()+ } else {+ self.data_indices.join(", ")+ }+ }++ fn edges_note(&self) -> String {+ match &self.edges_index {+ Some(edges) => format!(+ " Graph edges taught by the removed file(s) stay live in {edges} until that \+ index is deleted too."+ ),+ None => String::new(),+ }+ }+}+#[derive(Debug)]struct UnsupportedInventoryDeltaError {delta: UnsupportedInventoryDelta,- fresh_existing_plan: bool,+ targets: RefusalTargets,}impl UnsupportedInventoryDeltaError {@@ -1147,21 +1188,22 @@ impl UnsupportedInventoryDeltaError {json!({"schema": "xerj.autoindex.unsupported_sync_delta.v1","status": "error",- "error": if self.fresh_existing_plan {- "unsafe_fresh_existing_plan"- } else {- "unsupported_inventory_delta"- },- "message": if self.fresh_existing_plan {- "this attempt made no remote mutations; --fresh cannot discard an existing plan under the same destination because alias, path, graph, and stale-record cleanup knowledge would be lost; the existing destination may already be partial or stale"- } else {- "this attempt made no remote mutations because this autoindex plan cannot safely reconcile corpus membership changes; the existing destination may already be partial or stale"- },- "added_content_groups": self.delta.added_content_groups,+ "error": "unsupported_content_group_removal",+ "message": "this attempt made no remote mutations. Files that were indexed under this resume plan no longer exist in the folder, and their documents are still live in the destination; removing files from an indexed folder is not reconciled yet","vanished_content_groups": self.delta.vanished_content_groups,+ // Context, not the reason for the refusal: a rerun over a frozen+ // plan does not index files added after the plan was frozen.+ "added_content_groups": self.delta.added_content_groups,"recovery": {- "exact_rebuild": "index with a new --state-dir, new --prefix, and (when graph detection is enabled) new --brain; alternatively add --no-graph. Validate the isolated target before switching readers",- "warning": "--fresh is not recovery or destination reconciliation. The global autoindex-catalog and old target are not cleaned automatically; validate and clean them explicitly"+ "restore_removed_files": "put the listed file(s) back and rerun; every other file keeps its resume state",+ "rebuild_in_place": format!(+ "delete the indices this plan publishes ({}) and the state directory {}, then rerun. This re-extracts and re-embeds the whole corpus.{}",+ self.targets.indices_phrase(),+ self.targets.state_dir,+ self.targets.edges_note().trim_start_matches(' ')+ ),+ "rebuild_isolated": "index with a new --state-dir, new --prefix, and (when graph detection is enabled) new --brain; alternatively add --no-graph. Validate the isolated target before switching readers, then clean the old one",+ "fresh_warning": "--fresh re-extracts the current folder in place and does pick up added and changed files, but it never deletes documents already published for removed files, so it is refused here"}})}@@ -1176,23 +1218,36 @@ impl std::fmt::Display for UnsupportedInventoryDeltaError {.collect::<Vec<_>>().join(", ")};- let reason = if self.fresh_existing_plan {- "`--fresh` cannot discard an existing plan under the same destination because alias, \- path, graph, and stale-record cleanup knowledge would be lost"- } else {- "corpus membership synchronization is not yet supported"- };write!(formatter,- "this attempt made no remote mutations; the existing destination may already be \- partial or stale. {reason}. Added \- content groups [{}]; vanished content groups [{}]. For an exact rebuild, index the \- current folder with a new --state-dir, a new --prefix, and, when graph detection is \- enabled, a new --brain (or use --no-graph). Validate the isolated target before \- switching readers. `--fresh` is not recovery or destination reconciliation; the \- global autoindex-catalog and old target require explicit validated cleanup",- render(&self.delta.added_content_groups),+ "{} file(s) indexed under this resume plan no longer exist in the folder, and their \+ documents are still live in the destination. Removing files from an indexed folder \+ is not reconciled yet, so this attempt made no remote mutations — no documents, \+ aliases, graph edges or catalog entries were written. Removed content groups [{}].",+ self.delta.vanished_content_groups.len(),render(&self.delta.vanished_content_groups)+ )?;+ if !self.delta.added_content_groups.is_empty() {+ write!(+ formatter,+ " Also present but not in the frozen resume plan, so not indexed by this run \+ either [{}].",+ render(&self.delta.added_content_groups)+ )?;+ }+ write!(+ formatter,+ " Recovery, cheapest first: (1) restore the removed file(s) and rerun — every other \+ file keeps its resume state; (2) rebuild in place by deleting the indices this plan \+ publishes ({}) and the state directory {}, then rerunning — this re-extracts and \+ re-embeds the whole corpus.{} (3) rebuild isolated with a new --state-dir, a new \+ --prefix and, when graph detection is enabled, a new --brain (or --no-graph), \+ validate it, switch readers, then clean the old target. `--fresh` picks up added \+ and changed files in place but never deletes documents for removed files, so it is \+ refused here too",+ self.targets.indices_phrase(),+ self.targets.state_dir,+ self.targets.edges_note())}}@@ -1254,22 +1309,19 @@ impl UnsupportedInventoryDelta {}}- fn is_empty(&self) -> bool {- self.added_content_groups.is_empty() && self.vanished_content_groups.is_empty()- }-- fn into_error(self) -> anyhow::Error {- UnsupportedInventoryDeltaError {- delta: self,- fresh_existing_plan: false,- }- .into()+ /// A rerun is refused only when a content group the plan published has+ /// vanished from the folder: those documents stay live and searchable with+ /// no source file behind them, and nothing in this pipeline removes them.+ /// Additions are not refused — they are skipped by the frozen plan exactly+ /// as before and `--fresh` rebuilds the plan in place to include them.+ fn refuses(&self) -> bool {+ !self.vanished_content_groups.is_empty()}- fn into_fresh_error(self) -> anyhow::Error {+ fn into_error(self, targets: RefusalTargets) -> anyhow::Error {UnsupportedInventoryDeltaError {delta: self,- fresh_existing_plan: true,+ targets,}.into()}@@ -1505,16 +1557,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {.map(|(planned, current)| planned.unwrap_or_else(|| current.clone())).collect()};+ // `--fresh` is checked here too: discarding the plan does not delete+ // the documents already published for a file that is now gone, so a+ // removal is unsafe in place whether or not the plan is kept.let delta =UnsupportedInventoryDelta::between(&inventory.files, &comparison_keys, prior_plan);- if cfg.fresh {- // Even an apparently unchanged content-key set can have alias,- // path, graph, catalog, or partial-publication history that only- // the old plan can reconcile. Route 1 cannot safely discard it.- return Err(delta.into_fresh_error());- }- if !delta.is_empty() {- return Err(delta.into_error());+ if delta.refuses() {+ return Err(delta.into_error(RefusalTargets::describe(&cfg, &state_dir, prior_plan)));}}let mut journal = state::Journal::open_after_preflight(@@ -1741,20 +1790,19 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {plan};- // A durable plan is a crash-resume boundary, not a folder-sync- // generation. Fail before index creation/mapping, replacement intent,- // graph invalidation, delete-by-query, bulk publication, refresh, or- // catalog writes when the current inventory adds or removes an entire- // canonical content group. Continuing would either silently skip new- // data or leave vanished source records searchable. Existing planned-key- // content replacement and crash repair remain supported.+ // Second gate, after key selection has settled: fail before index+ // creation/mapping, replacement intent, graph invalidation,+ // delete-by-query, bulk publication, refresh, or catalog writes when a+ // canonical content group the plan published is gone from the folder.+ // Continuing would leave those documents searchable with no source file.+ // Additions, same-path replacement and crash repair remain supported.//- // This runs before the junk sweep below on purpose: a refused rerun must- // compute nothing and mutate nothing.+ // It runs before the #238 junk sweep below on purpose: a refused rerun+ // must compute nothing and mutate nothing.if resumed_with_plan {let delta = UnsupportedInventoryDelta::between(&files, &keys, &plan);- if !delta.is_empty() {- return Err(delta.into_error());+ if delta.refuses() {+ return Err(delta.into_error(RefusalTargets::describe(&cfg, &state_dir, &plan)));}}@@ -1930,14 +1978,34 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {rel: files[i].rel.clone(),format: "unknown".into(),status: "skipped".into(),- reason: "not in the frozen resume plan; no destination change was made. For an \- exact rebuild use a new --state-dir, a new --prefix, and, when graph \- detection is enabled, a new --brain (or use --no-graph)"+ reason: "not in the frozen resume plan, so nothing was published for it. Re-run \+ with --fresh to rebuild the plan in place and include it (ids stay \+ idempotent), or index it under a new --state-dir and --prefix".into(),bytes: files[i].size,});}}+ if !new_unplanned.is_empty() && !cfg.quiet {+ // The frozen plan cannot absorb files discovered after it was written.+ // Say so on stderr rather than leaving it to the catalog: a rerun that+ // quietly ignores new files is the bug this gate exists to surface.+ eprintln!(+ "autoindex: {} file(s) appeared after the resume plan was frozen and were NOT \+ indexed:",+ new_unplanned.len()+ );+ for jf in new_unplanned.iter().take(10) {+ eprintln!(" not in plan, skipped: {}", jf.rel);+ }+ if new_unplanned.len() > 10 {+ eprintln!(" … and {} more", new_unplanned.len() - 10);+ }+ eprintln!(+ "autoindex: re-run with --fresh to rebuild the plan in place and index them (ids \+ stay idempotent)"+ );+ }if resumed_with_plan {// Legacy journals predate intent-before-publication and may have live// partial records without either file_done or file_replace_start.@@ -3633,14 +3701,39 @@ mod inventory_delta_tests {}}+ fn targets() -> RefusalTargets {+ RefusalTargets {+ state_dir: "/state".into(),+ data_indices: vec!["ax-rows".into()],+ edges_index: Some(".xerj-memory-corpus-edges".into()),+ }+ }+#[test]- fn classifier_is_empty_for_noop_and_sorts_added_and_vanished_groups() {+ fn only_a_removed_content_group_refuses_a_rerun() {let mut plan = Plan::default();plan.files.insert("keep".into(), assignment("keep.csv"));assert!(- UnsupportedInventoryDelta::between(&[file("keep.csv")], &["keep".into()], &plan)- .is_empty()+ !UnsupportedInventoryDelta::between(&[file("keep.csv")], &["keep".into()], &plan)+ .refuses());+ // An added file is skipped by the frozen plan, not a refusal: the+ // documented rerun-then---fresh workflow has to keep working.+ let added = UnsupportedInventoryDelta::between(+ &[file("keep.csv"), file("new.csv")],+ &["keep".into(), "new".into()],+ &plan,+ );+ assert_eq!(added.added_content_groups.len(), 1);+ assert!(!added.refuses(), "an addition alone must not fail the run");+ // A removal leaves live documents with no source file behind them.+ assert!(UnsupportedInventoryDelta::between(&[], &[], &plan).refuses());+ }++ #[test]+ fn classifier_sorts_added_and_vanished_groups() {+ let mut plan = Plan::default();+ plan.files.insert("keep".into(), assignment("keep.csv"));plan.junk_files.push(JunkFile {file_key: "junk".into(),rel: "broken.pdf".into(),@@ -3650,13 +3743,13 @@ mod inventory_delta_tests {bytes: 1,});assert!(- UnsupportedInventoryDelta::between(+ !UnsupportedInventoryDelta::between(&[file("keep.csv"), file("broken.pdf")],&["keep".into(), "junk".into()],&plan,)- .is_empty(),- "unchanged durable junk is not newly added content"+ .refuses(),+ "unchanged durable junk is neither added nor vanished");plan.files.insert("old-z".into(), assignment("z-old.csv"));@@ -3689,23 +3782,48 @@ mod inventory_delta_tests {);}+ #[test]+ fn refusal_names_the_removed_files_and_every_recovery_route() {+ let mut plan = Plan::default();+ plan.files.insert("gone".into(), assignment("gone.csv"));+ let error = UnsupportedInventoryDelta::between(&[], &[], &plan).into_error(targets());+ let message = format!("{error:#}");+ assert!(message.contains("gone.csv"), "{message}");+ assert!(+ message.contains("no longer exist in the folder"),+ "{message}"+ );+ assert!(message.contains("made no remote mutations"), "{message}");+ assert!(message.contains("restore the removed file(s)"), "{message}");+ assert!(message.contains("ax-rows"), "{message}");+ assert!(message.contains("/state"), "{message}");+ assert!(message.contains(".xerj-memory-corpus-edges"), "{message}");+ assert!(message.contains("new --state-dir"), "{message}");+ assert!(message.contains("`--fresh`"), "{message}");+ }+#[test]fn cli_error_routing_separates_typed_json_from_unrelated_human_errors() {let typed = UnsupportedInventoryDelta {- added_content_groups: vec![InventoryDeltaEntry {+ added_content_groups: Vec::new(),+ vanished_content_groups: vec![InventoryDeltaEntry {file_key: "key".into(),- path: "new.csv".into(),+ path: "gone.csv".into(),}],- vanished_content_groups: Vec::new(),}- .into_error();+ .into_error(targets());let route = route_cli_error(&typed, true);assert_eq!(route.exit_code, 1);assert!(route.stderr.is_none());let stdout = route.stdout.unwrap();let value: Value = serde_json::from_str(&stdout).unwrap();assert_eq!(value["schema"], "xerj.autoindex.unsupported_sync_delta.v1");- assert_eq!(value["error"], "unsupported_inventory_delta");+ assert_eq!(value["error"], "unsupported_content_group_removal");+ assert_eq!(value["vanished_content_groups"][0]["path"], "gone.csv");+ assert!(value["recovery"]["rebuild_in_place"]+ .as_str()+ .unwrap()+ .contains("ax-rows"));let unrelated = anyhow::anyhow!("endpoint unavailable");let route = route_cli_error(&unrelated, true);@@ -3765,11 +3883,11 @@ mod duplicate_integration_tests {.unwrap_err();let message = format!("{error:#}");assert!(message.contains("collides with legacy resume key"));- // Recovery advice must stay scoped to the two colliding files and- // keep an exact rebuild isolated from the old destination.+ // Recovery advice must stay scoped to the two colliding files and be+ // honest that discarding the journal re-embeds the whole corpus.assert!(message.contains("remove or move one of these two files"));assert!(message.contains("/state/journal.ndjson"));- assert!(message.contains("new --state-dir, new --prefix, and new --brain"));+ assert!(message.contains("re-extracts and re-embeds the entire corpus"));}#[test]diff --git a/engine/crates/xerj-autoindex/src/state.rs b/engine/crates/xerj-autoindex/src/state.rsindex 7d4be6bec..1a63c0487 100644--- a/engine/crates/xerj-autoindex/src/state.rs+++ b/engine/crates/xerj-autoindex/src/state.rs@@ -333,10 +333,9 @@ fn read_plan_for_preflight("journal corruption at byte {record_start} in {}: malformed \newline-terminated record. Refusing to discard later records. Restore \the journal from a backup, or truncate it to exactly {record_start} \- bytes to keep every completion recorded before the corruption. For an \- isolated rebuild use a new --state-dir, new --prefix, and new --brain \- when graph detection is enabled (or --no-graph); explicitly validate \- and clean the shared catalog and old target",+ bytes to keep every completion recorded before the corruption. Deleting \+ the whole journal (or rerunning with --fresh) also recovers, but \+ re-extracts and re-embeds the entire corpus",path.display())})?;@@ -494,10 +493,9 @@ impl Journal {anyhow::bail!("journal at {} was created for root={jr} url={ju} prefix={jp}; \current run has root={root} url={url} prefix={prefix}. \- Refusing to discard history under the same destination. For an \- isolated rebuild use a new --state-dir, new --prefix, and new \- --brain when graph detection is enabled (or --no-graph); \- explicitly validate and clean the shared catalog and old target.",+ Use --state-dir for separate state, or --fresh to rebuild the \+ plan in place — note that --fresh never deletes documents \+ already published under the other root, url or prefix.",jpath.display());}@@ -606,10 +604,9 @@ impl Journal {after corruption. Restore the journal from a backup, or truncate it \to exactly {record_start} bytes to keep every completion recorded \before the corruption (files journaled after that point are \- re-verified, re-indexed and re-embedded on the next run). For an \- isolated rebuild use a new --state-dir, new --prefix, and new --brain \- when graph detection is enabled (or --no-graph); explicitly validate \- and clean the shared catalog and old target",+ re-verified, re-indexed and re-embedded on the next run). Deleting \+ the whole journal (or rerunning with --fresh) also recovers, but \+ re-extracts and re-embeds the entire corpus",jpath.display());}@@ -1039,10 +1036,9 @@ mod compatibility_tests {let message = format!("{error:#}");assert!(message.contains("journal corruption"));// Recovery guidance must stay scoped and honest: byte-exact truncation- // keeps prior completions; an exact rebuild needs isolated state and- // destination identities.+ // keeps prior completions; discarding the journal re-embeds everything.assert!(message.contains("truncate it to exactly"));- assert!(message.contains("new --state-dir, new --prefix, and new --brain"));+ assert!(message.contains("re-extracts and re-embeds the entire corpus"));}#[test]diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rsindex 6ba6474eb..1217b67fc 100644--- a/engine/crates/xerj-server/src/brain.rs+++ b/engine/crates/xerj-server/src/brain.rs@@ -79,8 +79,9 @@ pub fn print_help() {--api-key <K> API key for an already-running secured server\n\(or env XERJ_API_KEY; a server booted by this command\n\needs neither — its key is read from <data-dir>/admin.key)\n\- --fresh start without resume state only when the selected\n\- journal has no durable plan; never resets server data\n\+ --fresh ignore the resume journal and rebuild the plan in place,\n\+ re-walking everything (ids stay idempotent). It never\n\+ deletes documents for notes you removed\n\--no-open print the links but do not open a browser\n\--help, -h this help\n\\n\@@ -386,15 +387,16 @@ fn live_node_docs(es: &Es, brain: &str) -> Result<Option<u64>> {return Ok(None);};// `Es::get_doc` returns the document `_source` itself.+ // A meta doc without `nodes_index` predates the field (or was written by+ // something other than autoindex). That is absence of evidence, not a+ // failure: report it as unknown instead of hard-failing every brain that+ // an older autoindex wrote.let Some(nodes_index) = meta.get("nodes_index").and_then(Value::as_str).filter(|s| !s.is_empty())else {- bail!(- "brain metadata {edges_index}/{} has no usable nodes_index",- detect::BRAIN_META_ID- );+ return Ok(None);};let response = es.search(nodes_index,@@ -415,7 +417,11 @@ fn journal_server_disagreement(cfg: &BrainCfg, brain: &str) -> String {"the resume journal and server disagree: journal {} says {} is already indexed, \but {} has no confirmed live node documents for prefix ax and brain {brain}. No reset \was attempted: an absent or zero node probe does not prove that data, catalog, and \- graph namespaces are empty. Restore the server data directory used by this journal. \+ graph namespaces are empty. If this is the wrong server, restore or point at the data \+ directory this journal was written against. If the data directory really was wiped, \+ rerun with --fresh to rebuild the plan and republish every file in place (ids are \+ idempotent):\n\+ xerj brain {} --url {} --fresh\n\Otherwise, after validating or cleaning the old destination, run an isolated rebuild:\n\xerj autoindex {} --url {} --state-dir <new-state-dir> --prefix <new-prefix> \--brain <new-brain>\n\@@ -426,6 +432,8 @@ fn journal_server_disagreement(cfg: &BrainCfg, brain: &str) -> String {cfg.url,root.display(),cfg.url,+ root.display(),+ cfg.url,)}@@ -700,6 +708,8 @@ mod tests {assert!(message.contains("http://localhost:9200"));assert!(message.contains("prefix ax"));assert!(message.contains("brain team-notes"));+ assert!(message.contains("xerj brain /corpus/notes"));+ assert!(message.contains("--fresh"));assert!(message.contains("xerj autoindex /corpus/notes"));assert!(message.contains("--url http://localhost:9200"));assert!(message.contains("--state-dir <new-state-dir>"));diff --git a/landing/docs/recipes/zero-config-autoindex.html b/landing/docs/recipes/zero-config-autoindex.htmlindex 86d843b23..edc3d0b29 100644--- a/landing/docs/recipes/zero-config-autoindex.html+++ b/landing/docs/recipes/zero-config-autoindex.html@@ -239,7 +239,8 @@ live indices at http://localhost:9280:ax-docs 1 docsax-exports 800 docsautoindex-catalog 9 docs</pre>- <p><code class="inline">--fresh</code> starts without resume state only when the selected state directory has no durable plan; it never removes stale destination records. For an independent rebuild, use a new <code class="inline">--state-dir</code>, new <code class="inline">--prefix</code>, and new <code class="inline">--brain</code> when graph detection is enabled (or add <code class="inline">--no-graph</code>). Validate before switching readers. The shared <code class="inline">autoindex-catalog</code> and old target require explicit, validated cleanup. Added or removed content groups are refused; same-path replacement remains supported.</p>+ <p><code class="inline">--fresh</code> ignores the journal and rebuilds the plan in place (ids stay idempotent), which is how you pick up files added since the last run.</p>+ <p>A rerun that resumes an existing plan indexes changed files and reports added ones as skipped. Deleting a file is refused: nothing here removes the documents it already published, so the rerun stops before touching the destination and names what is gone. Restore the file and rerun, or rebuild — in place by deleting the named indices and the state directory, or isolated under a new <code class="inline">--state-dir</code>, <code class="inline">--prefix</code> and <code class="inline">--brain</code> (or <code class="inline">--no-graph</code>), validated before you switch readers. The shared <code class="inline">autoindex-catalog</code> and the old target require explicit cleanup.</p><hr>diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 301c382f4..698924812 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -140,16 +140,18 @@ destabilize the server.xerj autoindex map [--url U] [--json] [--dataset SLUG]xerj autoindex status [--url U] [--state-dir P]---fresh starts without resume state only when the selected state directory has-no durable plan. It never removes stale destination records. An independent-rebuild requires a new --state-dir, new --prefix, and new --brain when graph-detection is enabled (or --no-graph), followed by validation before switching-readers. The shared autoindex-catalog and old target require explicit,-validated cleanup. Added or removed content groups are refused; same-path-replacement remains supported.+--fresh ignores the journal and rebuilds the plan in place (ids stay+idempotent); it is how you pick up files added since the last run. A rerun+that resumes an existing plan indexes changed files and reports added ones as+skipped. Deleting an indexed file is refused before any destination change,+because nothing here removes the documents it already published: restore the+file and rerun, or rebuild — in place by deleting the named indices and the+state directory, or isolated under a new --state-dir, --prefix and --brain (or+--no-graph), validated before switching readers. The shared autoindex-catalog+and the old target require explicit cleanup.Exit codes: 0 complete · 3 completed-with-junk (junk recorded, never fatal) ·-2 usage · 1 endpoint/journal error or refused unsafe corpus-state transition.+2 usage · 1 endpoint/journal error or a refused corpus removal.Pipeline: walk → sniff → sample → infer → map → extract → bulk → correlate →catalog → verify.