Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(autoindex): refuse only the reruns that strand live documents

xerj-org3049723e06e4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 191e86fa5..26338783f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -529,22 +529,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`/v1/embedding/identity` reports `lexical` — the backend the server will
really use — rather than `proxy`.
-### Changed — fail-closed autoindex restart semantics
-
-- **`autoindex --fresh` no longer discards a durable resume plan.** A durable
- plan contains the alias, path, graph, and stale-record knowledge needed for
- safe reconciliation. `--fresh` is accepted only when the selected state
- directory has no durable plan and never cleans the destination. An
- independent rebuild must use a new state directory, new prefix, and new brain
- namespace when graph detection is enabled (or disable graph writes), be
- validated, and only then replace the old reader target. The shared catalog
- and old target require explicit, validated cleanup. Membership additions and
- removals are refused; same-path replacement remains supported.
+### Changed — autoindex refuses reruns that would strand documents
+
+- **Deleting an indexed file and rerunning `xerj autoindex` now fails instead
+ of exiting 0.** Nothing in the pipeline removes the documents, aliases,
+ graph edges or catalog entries that the deleted file published, so a rerun
+ that ignored the deletion left them live and searchable with no source file
+ behind them. The rerun is now refused before any remote call other than the
+ endpoint-readiness ping: no mapping, delete-by-query, bulk, refresh, graph
+ or catalog write is attempted, and the journal is not appended to. The error
+ names every removed file and its content key and gives three recovery
+ routes — restore the files and rerun, rebuild in place by deleting the named
+ indices and the state directory, or rebuild under a new state directory,
+ prefix and brain. `--fresh` is refused for the same case, because it does
+ not delete those documents either. `--json` emits the same facts as
+ `xerj.autoindex.unsupported_sync_delta.v1` on stdout with exit 1.
+- **Files added after a plan was frozen are now called out on stderr.** They
+ are still not indexed by a rerun that resumes an existing plan — the plan is
+ a crash-resume boundary, not a folder-sync generation — but the run now
+ lists them, records them as skipped (exit 3, completed-with-junk) and points
+ at `--fresh`, which rebuilds the plan in place and picks them up. Adding or
+ changing files and rerunning keeps working; only removals are refused.
- **`xerj brain` no longer turns an absent or zero node-count probe into an
automatic reset.** Journal/server disagreement now fails with the journal,
- URL, prefix, and brain identity plus recovery instructions. Probe transport
- and malformed-response failures remain errors instead of being reported as
- an empty destination.
+ URL, prefix, and brain identity plus recovery instructions, including the
+ explicit `--fresh` rerun for a genuinely wiped data directory. Probe
+ transport and malformed-response failures remain errors instead of being
+ reported as an empty destination.
## [1.0.0-rc.10] - 2026-08-03
diff --git a/demo/usecases/autoindex/RECIPE_DRAFT.md b/demo/usecases/autoindex/RECIPE_DRAFT.md
index faa1b6ec8..023080bac 100644
--- a/demo/usecases/autoindex/RECIPE_DRAFT.md
+++ b/demo/usecases/autoindex/RECIPE_DRAFT.md
@@ -203,13 +203,11 @@ done in 0.1s — 3 datasets, 5801 records live, 0 junk records, 1 junk/skipped f
just polite re-runs: in the robustness evaluation, a `kill -9` at 18 s into a
200 MB file followed by a re-run converged to **byte-identical counts across
all 9 datasets** — no duplicates. `xerj autoindex status` shows the journal
-and live index counts. `--fresh` starts without resume state only when the
-selected state directory has no durable plan; it never removes stale
-destination records. For an independent rebuild, use a new `--state-dir`, new
-`--prefix`, and new `--brain` when graph detection is enabled (or add
-`--no-graph`). Validate before switching readers. The shared
-`autoindex-catalog` and old target require explicit, validated cleanup. Added
-or removed content groups are refused; same-path replacement remains supported.
+and live index counts; `--fresh` ignores the journal and rebuilds the plan in
+place (ids stay idempotent), which is how a rerun picks up files added since
+the last one. Deleting an indexed file is refused before any destination
+change — nothing here removes the documents it already published — and the
+error names what is gone plus the recovery routes.
## Reproduce it yourself
diff --git a/demo/usecases/autoindex/run-eval.sh b/demo/usecases/autoindex/run-eval.sh
index dc36277ec..d7dff87ef 100755
--- a/demo/usecases/autoindex/run-eval.sh
+++ b/demo/usecases/autoindex/run-eval.sh
@@ -2,9 +2,11 @@
# End-to-end eval of `xerj autoindex` against a corpus folder.
#
# Boots a dedicated server on es_compat $PORT (rest PORT+100, grpc PORT+101),
-# runs autoindex over $CORPUS, proves same-state resume, then performs an
-# isolated data/graph rebuild under a new state directory, prefix, and brain.
-# The global autoindex-catalog remains shared and is excluded from comparison.
+# runs autoindex over $CORPUS, then re-runs twice to prove idempotency — the
+# resume path (all files done) and a full --fresh re-extract — and requires the
+# per-index doc counts to be byte-identical to run 1. Ends with the data map.
+# The shared autoindex-catalog is excluded from the comparison: every run
+# appends its own run record, so a growing catalog is the design, not drift.
#
# Nothing here is bound to one machine: every path is an env override with a
# default under the repo or $TMPDIR, so the same script gates CI and drives a
@@ -33,20 +35,11 @@ CFG="$WORK/server-$PORT.toml"
LOG="$WORK/server-$PORT.log"
URL="http://localhost:$PORT"
STATE="$WORK/state"
-REBUILD_STATE="$WORK/rebuild-state"
-PREFIX="ax-eval-a"
-REBUILD_PREFIX="ax-eval-b"
-BRAIN="eval-a"
-REBUILD_BRAIN="eval-b"
+PREFIX="ax-eval"
+BRAIN="eval"
[ -x "$BIN" ] || { echo "xerj binary not found at $BIN"; exit 1; }
[ -d "$CORPUS" ] || { echo "corpus folder not found at $CORPUS"; exit 1; }
-[ ! -e "$DATA" ] && [ ! -e "$STATE" ] && [ ! -e "$REBUILD_STATE" ] || {
- echo "work directory already contains eval state: $WORK"
- echo "choose a new XERJ_AUTOINDEX_WORK or explicitly remove the old isolated eval directory"
- exit 1
-}
-
mkdir -p "$WORK"
cat >"$CFG" <<EOF
[server]
@@ -85,10 +78,10 @@ if [ "$RC" != "0" ] && [ "$RC" != "3" ]; then
FAIL=1
fi
-# Name + doc count only. The _cat row also carries on-disk size, which an
-# isolated data/graph rebuild legitimately changes (segments are rewritten).
-# The shared global catalog is excluded because every run appends its own run
-# record — a growing catalog is the design, not drift. Columns of the _cat row:
+# Name + doc count only. The _cat row also carries on-disk size, which a
+# --fresh re-extract legitimately changes (segments are rewritten), and the
+# catalog is excluded outright because every run appends its own run record —
+# a growing catalog is the design, not drift. Columns of the _cat row:
# green(1) open(2) name(3) uuid(4) pri(5) rep(6) docs(7).
counts() { curl -s "$URL/_cat/indices" | awk -v prefix="$1-" '$3 ~ ("^" prefix) {sub("^" prefix, "", $3); print $3, $7}' | sort; }
@@ -104,16 +97,16 @@ if [ "$RC" != "0" ] && [ "$RC" != "3" ]; then
FAIL=1
fi
-echo "=== isolated data/graph rebuild reproducibility (new state, prefix, and brain; shared catalog excluded) ==="
-time "$BIN" autoindex "$CORPUS" --url "$URL" --state-dir "$REBUILD_STATE" --prefix "$REBUILD_PREFIX" --brain "$REBUILD_BRAIN"
+echo "=== idempotency: re-run with --fresh (full re-extract, idempotent ids) ==="
+time "$BIN" autoindex "$CORPUS" --url "$URL" --state-dir "$STATE" --prefix "$PREFIX" --brain "$BRAIN" --fresh
RC=$?
if [ "$RC" != "0" ] && [ "$RC" != "3" ]; then
- echo "isolated data/graph rebuild exited $RC (expected 0 or 3)"
+ echo "--fresh re-extract exited $RC (expected 0 or 3)"
FAIL=1
fi
-counts "$REBUILD_PREFIX" > "$WORK/counts-run3.txt"
-echo "=== normalized data-index count diff original vs isolated data/graph rebuild (shared catalog excluded; must be empty) ==="
+counts "$PREFIX" > "$WORK/counts-run3.txt"
+echo "=== doc-count diff run1 vs run3 (must be empty) ==="
cat "$WORK/counts-run3.txt"
if diff "$WORK/counts-run1.txt" "$WORK/counts-run3.txt"; then
echo "IDENTICAL COUNTS ✓"
diff --git a/docs/design/second_brain_graph_wf.js b/docs/design/second_brain_graph_wf.js
index 404b5d591..3225781a3 100644
--- a/docs/design/second_brain_graph_wf.js
+++ b/docs/design/second_brain_graph_wf.js
@@ -99,9 +99,8 @@ heterogeneous content (code + PDFs + docs), and captured as a screencast.
automaticPresenceSimulation:true})\`, then claim via the setup magic-link the server prints
at first boot. A working reference script is ${REPO}/sb_capture.mjs.
- \`xerj brain\` gotcha: a stale resume journal in \`~/.xerj/autoindex\` makes re-runs skip
- detection. For an independent clean run, choose a new \`--state-dir\`, new
- \`--prefix\`, and new \`--brain\` (or \`--no-graph\`); \`--fresh\` cannot
- discard a durable plan or clean the shared catalog or old target.
+ detection. Use \`--fresh\` (or clear it) for clean runs; note that \`--fresh\` rebuilds
+ the plan in place and never removes documents for notes you deleted.
- Commits authored \`xerj-org <[email protected]>\`, NO Claude co-author trailer, never mention
ctrl-frk. Do NOT push. Commit on \`feat/second-brain\`.
diff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.md
index d63a24e1b..b75f062df 100644
--- a/docs/recipes/zero-config-indexing.md
+++ b/docs/recipes/zero-config-indexing.md
@@ -228,13 +228,17 @@ live indices at http://localhost:9280:
autoindex-catalog 9 docs
```
-`--fresh` starts without resume state only when the selected state directory
-has no durable plan; it never removes stale destination records. For an
-independent rebuild, use a new `--state-dir`, new `--prefix`, and new
-`--brain` when graph detection is enabled (or add `--no-graph`). Validate
-before switching readers. The shared `autoindex-catalog` and old target require
-explicit, validated cleanup. Added or removed content groups are refused;
-same-path replacement remains supported.
+`--fresh` ignores the journal and rebuilds the plan in place (ids stay
+idempotent), which is how you pick up files added since the last run.
+
+A rerun that resumes an existing plan indexes changed files and reports added
+ones as skipped. Deleting a file is refused: nothing here removes the
+documents it already published, so the rerun stops before touching the
+destination and names what is gone. Restore the file and rerun, or rebuild —
+in place by deleting the named indices and the state directory, or isolated
+under a new `--state-dir`, `--prefix` and `--brain` (or `--no-graph`),
+validated before you switch readers. The shared `autoindex-catalog` and the
+old target require explicit cleanup.
## Reproduce it yourself
diff --git a/docs/usecases/second-brain/README.md b/docs/usecases/second-brain/README.md
index 41c74feec..0e181b66f 100644
--- a/docs/usecases/second-brain/README.md
+++ b/docs/usecases/second-brain/README.md
@@ -205,12 +205,13 @@ other note tools. No such numbers are claimed.
root-caused.
- **YAML-frontmatter markdown can sniff as YAML:** one `SKILL.md` with
frontmatter landed in junk/skipped instead of being indexed.
-- **Stale resume journals can pollute links** after a wiped data dir;
- `--fresh` cannot discard a durable plan and is not stale-journal recovery.
- Restore the server data directory used by the journal. For an explicitly
- isolated rebuild, use a new state directory, prefix, and brain namespace,
- validate it, then switch readers. The shared `autoindex-catalog` and old
- target require explicit, validated cleanup.
+- **Stale resume journals can pollute links** after a wiped data dir. `xerj
+ brain` no longer guesses: it reports the disagreement between the journal
+ and the server instead of resetting on its own. If the data directory really
+ was wiped, rerun with `--fresh` to rebuild the plan and republish every note
+ in place; if it was not, point at the data directory the journal was written
+ against, or rebuild under a new state directory, prefix and brain and
+ validate before switching readers.
- **Wikilink detection was not exercised on this corpus** (real docs use
markdown links); it is covered by the demo-corpus run (30 wikilinks) and
the autoindex test suite.
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index e225b4732..4136e817a 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -72,9 +72,8 @@ pub enum Cmd {
Help,
}
-const FRESH_HELP: &str =
- "start without resume state only when the selected state directory has no \
-durable plan; an existing plan is refused and destination records are never reset";
+const FRESH_HELP: &str = "ignore the existing journal and rebuild the plan in place\n\
+ (ids stay idempotent) — see RESUME POLICY";
pub fn print_help() {
println!(
@@ -171,14 +170,16 @@ pub fn print_help() {
\n\
RESUME POLICY:\n\
A durable plan supports no-op resume and same-path content replacement.\n\
- Added or removed content groups are refused before remote mutation.\n\
- An independent rebuild needs a new --state-dir, new --prefix, and, when\n\
- graph detection is enabled, new --brain (or --no-graph). Validate before\n\
- switching readers; the shared autoindex-catalog and old target require\n\
- explicit, validated cleanup.\n\
+ Files added after the plan was frozen are reported as skipped and are not\n\
+ indexed; --fresh rebuilds the plan in place and picks them up.\n\
+ Removing an indexed file is refused before any remote mutation: its\n\
+ documents are already live and nothing here deletes them. Restore the file\n\
+ and rerun, or rebuild — in place by deleting the published indices and the\n\
+ state directory, or isolated under a new --state-dir, --prefix and --brain\n\
+ (or --no-graph), validated before you switch readers.\n\
\n\
EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\
- 2 usage; 1 endpoint/journal failure or unsupported corpus delta\n",
+ 2 usage; 1 endpoint/journal failure or a refused corpus removal\n",
fresh_help = FRESH_HELP
);
}
@@ -476,10 +477,10 @@ mod tests {
}
#[test]
- fn fresh_help_warns_that_it_is_not_destination_reconciliation() {
- assert!(super::FRESH_HELP.contains("no durable plan"));
- assert!(super::FRESH_HELP.contains("existing plan is refused"));
- assert!(super::FRESH_HELP.contains("destination records are never reset"));
+ fn fresh_help_points_at_the_resume_policy_that_bounds_it() {
+ assert!(super::FRESH_HELP.contains("rebuild the plan in place"));
+ assert!(super::FRESH_HELP.contains("ids stay idempotent"));
+ assert!(super::FRESH_HELP.contains("RESUME POLICY"));
}
#[test]
diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
index 74188e21c..bf700acce 100644
--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
@@ -532,11 +532,15 @@ fn assert_unsupported_delta_without_remote_mutation(
let message = format!("{error:#}");
assert!(message.contains("made no remote mutations"), "{message}");
assert!(
- message.contains("existing destination may already be partial or stale"),
+ message.contains("no longer exist in the folder"),
"{message}"
);
assert!(
- message.contains("`--fresh` is not recovery or destination reconciliation"),
+ message.contains("restore the removed file(s) and rerun"),
+ "{message}"
+ );
+ assert!(
+ message.contains("rebuild in place by deleting the indices"),
"{message}"
);
assert!(
@@ -558,19 +562,52 @@ fn assert_unsupported_delta_without_remote_mutation(
}
}
+/// The documented headline workflow: point autoindex at a folder, add a file,
+/// rerun. The rerun must not fail, must say plainly that the added file was
+/// not indexed, and `--fresh` must then absorb it in place.
#[test]
-fn completed_plan_rejects_added_content_group_before_remote_mutation() {
+fn a_rerun_after_an_added_file_succeeds_and_fresh_absorbs_it() {
let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
let corpus = tempfile::tempdir().unwrap();
let state_dir = tempfile::tempdir().unwrap();
fs::write(corpus.path().join("first.csv"), "id,value\n1,first\n").unwrap();
let endpoint = MockEndpoint::start(usize::MAX);
- let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
assert_eq!(run_index(config.clone()).unwrap(), 0);
fs::write(corpus.path().join("second.csv"), "id,value\n2,second\n").unwrap();
- assert_unsupported_delta_without_remote_mutation(&endpoint, config, &["second.csv"], &[]);
+ // 3 = completed-with-junk: the added file is reported as skipped, not
+ // indexed, because the frozen plan cannot absorb it.
+ let (code, report) = run_index_report(config.clone()).unwrap();
+ assert_eq!(code, 3);
+ let report = report.unwrap();
+ assert_eq!(report["files_junk"], 1, "{report}");
+ assert_eq!(report["records_total"], 0, "{report}");
+ {
+ let locked = endpoint.state.lock().unwrap();
+ let live: Vec<&str> = locked
+ .docs
+ .values()
+ .filter_map(|doc| doc["ax_path"].as_str())
+ .collect();
+ assert!(
+ live.iter().all(|path| *path == "first.csv"),
+ "the added file must not be published by a plan that predates it: {live:?}"
+ );
+ }
+
+ // --fresh rebuilds the plan in place and picks the new file up.
+ config.fresh = true;
+ assert_eq!(run_index(config).unwrap(), 0);
+ let locked = endpoint.state.lock().unwrap();
+ let indexed: std::collections::HashSet<&str> = locked
+ .docs
+ .values()
+ .filter_map(|doc| doc["ax_path"].as_str())
+ .collect();
+ assert!(indexed.contains("first.csv"), "{indexed:?}");
+ assert!(indexed.contains("second.csv"), "{indexed:?}");
}
#[test]
@@ -636,7 +673,7 @@ fn completed_plan_rejects_mixed_membership_delta_in_stable_order() {
}
#[test]
-fn nonempty_fresh_cannot_erase_plan_and_bypass_membership_gate() {
+fn fresh_cannot_erase_the_plan_and_bypass_the_removal_gate() {
let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
let corpus = tempfile::tempdir().unwrap();
@@ -653,7 +690,7 @@ fn nonempty_fresh_cannot_erase_plan_and_bypass_membership_gate() {
}
#[test]
-fn fresh_rejects_same_content_canonical_promotion_even_when_group_key_matches() {
+fn deleting_one_path_of_a_duplicate_pair_is_not_a_removed_content_group() {
let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
let _io_guard = state::FILE_DONE_IO_FAILPOINT_TEST_LOCK.lock().unwrap();
let corpus = tempfile::tempdir().unwrap();
@@ -662,14 +699,24 @@ fn fresh_rejects_same_content_canonical_promotion_even_when_group_key_matches()
fs::write(corpus.path().join("a.csv"), bytes).unwrap();
fs::write(corpus.path().join("b.csv"), bytes).unwrap();
let endpoint = MockEndpoint::start(usize::MAX);
- let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
assert_eq!(run_index(config.clone()).unwrap(), 0);
- // b.csv becomes canonical with the same content key. A key-only delta is
- // empty, but fresh would discard the alias/path cleanup knowledge.
+ // b.csv becomes canonical under the same content key: the group survives
+ // the deletion, so no document is stranded and the rerun is allowed.
fs::remove_file(corpus.path().join("a.csv")).unwrap();
- config.fresh = true;
- assert_unsupported_delta_without_remote_mutation(&endpoint, config, &[], &[]);
+ let result = run_index(config);
+ assert!(
+ result.is_ok(),
+ "a surviving content group must not trip the removal gate: {result:?}"
+ );
+ let locked = endpoint.state.lock().unwrap();
+ let live: Vec<&str> = locked
+ .docs
+ .values()
+ .filter_map(|doc| doc["ax_path"].as_str())
+ .collect();
+ assert_eq!(live, ["b.csv"], "canonical path follows the surviving file");
}
#[test]
@@ -726,6 +773,7 @@ fn completed_plan_rejects_empty_current_folder_before_remote_mutation() {
assert_eq!(run_index(config.clone()).unwrap(), 0);
fs::remove_file(corpus.path().join("only.csv")).unwrap();
+ assert_unsupported_delta_without_remote_mutation(&endpoint, config.clone(), &[], &["only.csv"]);
// Even `--fresh` must not erase the only durable inventory evidence and
// then report success while the destination still contains the document.
config.fresh = true;
@@ -1518,12 +1566,12 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {
)
.unwrap();
fs::write(corpus.path().join("b.csv"), original).unwrap();
- assert_unsupported_delta_without_remote_mutation(&endpoint, config.clone(), &["b.csv"], &[]);
+ assert_eq!(run_index(config.clone()).unwrap(), 3);
{
let locked = endpoint.state.lock().unwrap();
assert_eq!(locked.docs.len(), 2);
assert!(locked.docs.values().all(|doc| {
- doc["ax_path"].as_str() == Some("a.csv") && doc["value"].as_str() == Some("original")
+ doc["ax_path"].as_str() == Some("a.csv") && doc["value"].as_str() == Some("rewritten")
}));
}
let replay = state::Journal::open(
@@ -1538,20 +1586,24 @@ fn a_planned_key_never_gains_two_owners_when_old_content_moves_paths() {
assert!(replay.pending_replacements.is_empty());
assert_eq!(replay.done.len(), 1);
let plan = replay.plan.clone().unwrap();
- assert!(plan.junk_files.is_empty());
+ assert_eq!(plan.junk_files.len(), 1);
+ assert_eq!(plan.junk_files[0].rel, "b.csv");
+ assert!(plan.junk_files[0]
+ .reason
+ .contains("key ownership exclusive"));
drop(replay);
- // The refusal is deterministic: an identical rerun keeps the old owner,
- // appends no new plan, and changes no documents.
+ // The divergence is durable and deterministic: an identical rerun keeps
+ // the same owner, appends no new plan, and changes no documents.
let plans_before = event_count(state_dir.path(), "plan");
- assert_unsupported_delta_without_remote_mutation(&endpoint, config, &["b.csv"], &[]);
+ assert_eq!(run_index(config).unwrap(), 3);
assert_eq!(event_count(state_dir.path(), "plan"), plans_before);
let locked = endpoint.state.lock().unwrap();
assert_eq!(locked.docs.len(), 2);
assert!(locked
.docs
.values()
- .all(|doc| doc["value"].as_str() == Some("original")));
+ .all(|doc| doc["value"].as_str() == Some("rewritten")));
}
/// #238: a file added after the plan was frozen is skipped and reported in the
@@ -1664,9 +1716,11 @@ fn deleting_an_entire_duplicate_group_strands_no_pending_replacement() {
assert_eq!(run_index(config.clone()).unwrap(), 0);
let starts = event_count(state_dir.path(), "file_replace_start");
- // The whole duplicate group disappears. There is no current file left to
- // republish its key, so no replacement intent may be journaled — a
- // stranded intent would be re-appended forever without ever committing.
+ // The whole duplicate group disappears: its documents stay live with no
+ // source file, so the rerun is refused. The original invariant still
+ // holds and is still checked — a key with no current file must never get
+ // a replacement intent journaled, which would be re-appended forever
+ // without ever committing.
fs::remove_file(corpus.path().join("dup-a.csv")).unwrap();
fs::remove_file(corpus.path().join("dup-b.csv")).unwrap();
for _ in 0..2 {
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index 618c2d6c0..688d3de31 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -1034,10 +1034,8 @@ fn select_resume_plan_keys(
for (key, assignment) in &plan.files {
if let Some(previous) = planned_by_rel.insert(&assignment.rel, key) {
anyhow::bail!(
- "resume plan assigns path {} to both {} and {}; refusing to discard history under \
- the same destination. For an isolated rebuild use a new --state-dir, new \
- --prefix, and new --brain when graph detection is enabled (or --no-graph); \
- explicitly validate and clean the shared catalog and old target",
+ "resume plan assigns path {} to both {} and {}; use --fresh after verifying the \
+ existing index",
assignment.rel,
previous,
key
@@ -1046,11 +1044,8 @@ fn select_resume_plan_keys(
if !assignment.path_id.is_empty() {
if let Some(previous) = planned_by_path_id.insert(&assignment.path_id, key) {
anyhow::bail!(
- "resume plan assigns one native path identity to both {} and {}; refusing to \
- discard history under the same destination. For an isolated rebuild use a \
- new --state-dir, new --prefix, and new --brain when graph detection is \
- enabled (or --no-graph); explicitly validate and clean the shared catalog \
- and old target",
+ "resume plan assigns one native path identity to both {} and {}; use --fresh \
+ after verifying the existing index",
previous,
key
);
@@ -1088,11 +1083,9 @@ fn select_resume_plan_keys(
anyhow::bail!(
"{} collides with legacy resume key {} already owned by {}. No documents \
were changed; remove or move one of these two files out of the corpus \
- and rerun — every other file keeps its resume state. Refusing to discard \
- history under the same destination. For an isolated rebuild use a new \
- --state-dir, new --prefix, and new --brain when graph detection is \
- enabled (or --no-graph); explicitly validate and clean the shared \
- catalog and old target. Journal: {}",
+ and rerun — every other file keeps its resume state. Deleting the \
+ journal at {} (or rerunning with --fresh) also clears the collision, \
+ but re-extracts and re-embeds the entire corpus",
file.rel,
legacy_key,
assignment.rel,
@@ -1136,10 +1129,58 @@ struct UnsupportedInventoryDelta {
vanished_content_groups: Vec<InventoryDeltaEntry>,
}
+/// Everything the refusal needs to name the destination it is protecting.
+/// Collected at the gate so the message can list the exact indices and state
+/// directory an operator has to act on, instead of describing them abstractly.
+#[derive(Debug, Clone, PartialEq, Eq)]
+struct RefusalTargets {
+ state_dir: String,
+ data_indices: Vec<String>,
+ edges_index: Option<String>,
+}
+
+impl RefusalTargets {
+ fn describe(cfg: &IndexCfg, state_dir: &Path, plan: &Plan) -> Self {
+ let mut data_indices: Vec<String> = plan.datasets.iter().map(|d| d.index.clone()).collect();
+ data_indices.sort();
+ data_indices.dedup();
+ let edges_index = (!cfg.no_graph).then(|| {
+ let brain = cfg
+ .brain
+ .clone()
+ .unwrap_or_else(|| derive_brain_name(&cfg.root));
+ detect::edges_index_name(&brain)
+ });
+ Self {
+ state_dir: state_dir.display().to_string(),
+ data_indices,
+ edges_index,
+ }
+ }
+
+ fn indices_phrase(&self) -> String {
+ if self.data_indices.is_empty() {
+ "the indices this plan publishes".to_string()
+ } else {
+ self.data_indices.join(", ")
+ }
+ }
+
+ fn edges_note(&self) -> String {
+ match &self.edges_index {
+ Some(edges) => format!(
+ " Graph edges taught by the removed file(s) stay live in {edges} until that \
+ index is deleted too."
+ ),
+ None => String::new(),
+ }
+ }
+}
+
#[derive(Debug)]
struct UnsupportedInventoryDeltaError {
delta: UnsupportedInventoryDelta,
- fresh_existing_plan: bool,
+ targets: RefusalTargets,
}
impl UnsupportedInventoryDeltaError {
@@ -1147,21 +1188,22 @@ impl UnsupportedInventoryDeltaError {
json!({
"schema": "xerj.autoindex.unsupported_sync_delta.v1",
"status": "error",
- "error": if self.fresh_existing_plan {
- "unsafe_fresh_existing_plan"
- } else {
- "unsupported_inventory_delta"
- },
- "message": if self.fresh_existing_plan {
- "this attempt made no remote mutations; --fresh cannot discard an existing plan under the same destination because alias, path, graph, and stale-record cleanup knowledge would be lost; the existing destination may already be partial or stale"
- } else {
- "this attempt made no remote mutations because this autoindex plan cannot safely reconcile corpus membership changes; the existing destination may already be partial or stale"
- },
- "added_content_groups": self.delta.added_content_groups,
+ "error": "unsupported_content_group_removal",
+ "message": "this attempt made no remote mutations. Files that were indexed under this resume plan no longer exist in the folder, and their documents are still live in the destination; removing files from an indexed folder is not reconciled yet",
"vanished_content_groups": self.delta.vanished_content_groups,
+ // Context, not the reason for the refusal: a rerun over a frozen
+ // plan does not index files added after the plan was frozen.
+ "added_content_groups": self.delta.added_content_groups,
"recovery": {
- "exact_rebuild": "index with a new --state-dir, new --prefix, and (when graph detection is enabled) new --brain; alternatively add --no-graph. Validate the isolated target before switching readers",
- "warning": "--fresh is not recovery or destination reconciliation. The global autoindex-catalog and old target are not cleaned automatically; validate and clean them explicitly"
+ "restore_removed_files": "put the listed file(s) back and rerun; every other file keeps its resume state",
+ "rebuild_in_place": format!(
+ "delete the indices this plan publishes ({}) and the state directory {}, then rerun. This re-extracts and re-embeds the whole corpus.{}",
+ self.targets.indices_phrase(),
+ self.targets.state_dir,
+ self.targets.edges_note().trim_start_matches(' ')
+ ),
+ "rebuild_isolated": "index with a new --state-dir, new --prefix, and (when graph detection is enabled) new --brain; alternatively add --no-graph. Validate the isolated target before switching readers, then clean the old one",
+ "fresh_warning": "--fresh re-extracts the current folder in place and does pick up added and changed files, but it never deletes documents already published for removed files, so it is refused here"
}
})
}
@@ -1176,23 +1218,36 @@ impl std::fmt::Display for UnsupportedInventoryDeltaError {
.collect::<Vec<_>>()
.join(", ")
};
- let reason = if self.fresh_existing_plan {
- "`--fresh` cannot discard an existing plan under the same destination because alias, \
- path, graph, and stale-record cleanup knowledge would be lost"
- } else {
- "corpus membership synchronization is not yet supported"
- };
write!(
formatter,
- "this attempt made no remote mutations; the existing destination may already be \
- partial or stale. {reason}. Added \
- content groups [{}]; vanished content groups [{}]. For an exact rebuild, index the \
- current folder with a new --state-dir, a new --prefix, and, when graph detection is \
- enabled, a new --brain (or use --no-graph). Validate the isolated target before \
- switching readers. `--fresh` is not recovery or destination reconciliation; the \
- global autoindex-catalog and old target require explicit validated cleanup",
- render(&self.delta.added_content_groups),
+ "{} file(s) indexed under this resume plan no longer exist in the folder, and their \
+ documents are still live in the destination. Removing files from an indexed folder \
+ is not reconciled yet, so this attempt made no remote mutations — no documents, \
+ aliases, graph edges or catalog entries were written. Removed content groups [{}].",
+ self.delta.vanished_content_groups.len(),
render(&self.delta.vanished_content_groups)
+ )?;
+ if !self.delta.added_content_groups.is_empty() {
+ write!(
+ formatter,
+ " Also present but not in the frozen resume plan, so not indexed by this run \
+ either [{}].",
+ render(&self.delta.added_content_groups)
+ )?;
+ }
+ write!(
+ formatter,
+ " Recovery, cheapest first: (1) restore the removed file(s) and rerun — every other \
+ file keeps its resume state; (2) rebuild in place by deleting the indices this plan \
+ publishes ({}) and the state directory {}, then rerunning — this re-extracts and \
+ re-embeds the whole corpus.{} (3) rebuild isolated with a new --state-dir, a new \
+ --prefix and, when graph detection is enabled, a new --brain (or --no-graph), \
+ validate it, switch readers, then clean the old target. `--fresh` picks up added \
+ and changed files in place but never deletes documents for removed files, so it is \
+ refused here too",
+ self.targets.indices_phrase(),
+ self.targets.state_dir,
+ self.targets.edges_note()
)
}
}
@@ -1254,22 +1309,19 @@ impl UnsupportedInventoryDelta {
}
}
- fn is_empty(&self) -> bool {
- self.added_content_groups.is_empty() && self.vanished_content_groups.is_empty()
- }
-
- fn into_error(self) -> anyhow::Error {
- UnsupportedInventoryDeltaError {
- delta: self,
- fresh_existing_plan: false,
- }
- .into()
+ /// A rerun is refused only when a content group the plan published has
+ /// vanished from the folder: those documents stay live and searchable with
+ /// no source file behind them, and nothing in this pipeline removes them.
+ /// Additions are not refused — they are skipped by the frozen plan exactly
+ /// as before and `--fresh` rebuilds the plan in place to include them.
+ fn refuses(&self) -> bool {
+ !self.vanished_content_groups.is_empty()
}
- fn into_fresh_error(self) -> anyhow::Error {
+ fn into_error(self, targets: RefusalTargets) -> anyhow::Error {
UnsupportedInventoryDeltaError {
delta: self,
- fresh_existing_plan: true,
+ targets,
}
.into()
}
@@ -1505,16 +1557,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
.map(|(planned, current)| planned.unwrap_or_else(|| current.clone()))
.collect()
};
+ // `--fresh` is checked here too: discarding the plan does not delete
+ // the documents already published for a file that is now gone, so a
+ // removal is unsafe in place whether or not the plan is kept.
let delta =
UnsupportedInventoryDelta::between(&inventory.files, &comparison_keys, prior_plan);
- if cfg.fresh {
- // Even an apparently unchanged content-key set can have alias,
- // path, graph, catalog, or partial-publication history that only
- // the old plan can reconcile. Route 1 cannot safely discard it.
- return Err(delta.into_fresh_error());
- }
- if !delta.is_empty() {
- return Err(delta.into_error());
+ if delta.refuses() {
+ return Err(delta.into_error(RefusalTargets::describe(&cfg, &state_dir, prior_plan)));
}
}
let mut journal = state::Journal::open_after_preflight(
@@ -1741,20 +1790,19 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
plan
};
- // A durable plan is a crash-resume boundary, not a folder-sync
- // generation. Fail before index creation/mapping, replacement intent,
- // graph invalidation, delete-by-query, bulk publication, refresh, or
- // catalog writes when the current inventory adds or removes an entire
- // canonical content group. Continuing would either silently skip new
- // data or leave vanished source records searchable. Existing planned-key
- // content replacement and crash repair remain supported.
+ // Second gate, after key selection has settled: fail before index
+ // creation/mapping, replacement intent, graph invalidation,
+ // delete-by-query, bulk publication, refresh, or catalog writes when a
+ // canonical content group the plan published is gone from the folder.
+ // Continuing would leave those documents searchable with no source file.
+ // Additions, same-path replacement and crash repair remain supported.
//
- // This runs before the junk sweep below on purpose: a refused rerun must
- // compute nothing and mutate nothing.
+ // It runs before the #238 junk sweep below on purpose: a refused rerun
+ // must compute nothing and mutate nothing.
if resumed_with_plan {
let delta = UnsupportedInventoryDelta::between(&files, &keys, &plan);
- if !delta.is_empty() {
- return Err(delta.into_error());
+ if delta.refuses() {
+ return Err(delta.into_error(RefusalTargets::describe(&cfg, &state_dir, &plan)));
}
}
@@ -1930,14 +1978,34 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
rel: files[i].rel.clone(),
format: "unknown".into(),
status: "skipped".into(),
- reason: "not in the frozen resume plan; no destination change was made. For an \
- exact rebuild use a new --state-dir, a new --prefix, and, when graph \
- detection is enabled, a new --brain (or use --no-graph)"
+ reason: "not in the frozen resume plan, so nothing was published for it. Re-run \
+ with --fresh to rebuild the plan in place and include it (ids stay \
+ idempotent), or index it under a new --state-dir and --prefix"
.into(),
bytes: files[i].size,
});
}
}
+ if !new_unplanned.is_empty() && !cfg.quiet {
+ // The frozen plan cannot absorb files discovered after it was written.
+ // Say so on stderr rather than leaving it to the catalog: a rerun that
+ // quietly ignores new files is the bug this gate exists to surface.
+ eprintln!(
+ "autoindex: {} file(s) appeared after the resume plan was frozen and were NOT \
+ indexed:",
+ new_unplanned.len()
+ );
+ for jf in new_unplanned.iter().take(10) {
+ eprintln!(" not in plan, skipped: {}", jf.rel);
+ }
+ if new_unplanned.len() > 10 {
+ eprintln!(" … and {} more", new_unplanned.len() - 10);
+ }
+ eprintln!(
+ "autoindex: re-run with --fresh to rebuild the plan in place and index them (ids \
+ stay idempotent)"
+ );
+ }
if resumed_with_plan {
// Legacy journals predate intent-before-publication and may have live
// partial records without either file_done or file_replace_start.
@@ -3633,14 +3701,39 @@ mod inventory_delta_tests {
}
}
+ fn targets() -> RefusalTargets {
+ RefusalTargets {
+ state_dir: "/state".into(),
+ data_indices: vec!["ax-rows".into()],
+ edges_index: Some(".xerj-memory-corpus-edges".into()),
+ }
+ }
+
#[test]
- fn classifier_is_empty_for_noop_and_sorts_added_and_vanished_groups() {
+ fn only_a_removed_content_group_refuses_a_rerun() {
let mut plan = Plan::default();
plan.files.insert("keep".into(), assignment("keep.csv"));
assert!(
- UnsupportedInventoryDelta::between(&[file("keep.csv")], &["keep".into()], &plan)
- .is_empty()
+ !UnsupportedInventoryDelta::between(&[file("keep.csv")], &["keep".into()], &plan)
+ .refuses()
);
+ // An added file is skipped by the frozen plan, not a refusal: the
+ // documented rerun-then---fresh workflow has to keep working.
+ let added = UnsupportedInventoryDelta::between(
+ &[file("keep.csv"), file("new.csv")],
+ &["keep".into(), "new".into()],
+ &plan,
+ );
+ assert_eq!(added.added_content_groups.len(), 1);
+ assert!(!added.refuses(), "an addition alone must not fail the run");
+ // A removal leaves live documents with no source file behind them.
+ assert!(UnsupportedInventoryDelta::between(&[], &[], &plan).refuses());
+ }
+
+ #[test]
+ fn classifier_sorts_added_and_vanished_groups() {
+ let mut plan = Plan::default();
+ plan.files.insert("keep".into(), assignment("keep.csv"));
plan.junk_files.push(JunkFile {
file_key: "junk".into(),
rel: "broken.pdf".into(),
@@ -3650,13 +3743,13 @@ mod inventory_delta_tests {
bytes: 1,
});
assert!(
- UnsupportedInventoryDelta::between(
+ !UnsupportedInventoryDelta::between(
&[file("keep.csv"), file("broken.pdf")],
&["keep".into(), "junk".into()],
&plan,
)
- .is_empty(),
- "unchanged durable junk is not newly added content"
+ .refuses(),
+ "unchanged durable junk is neither added nor vanished"
);
plan.files.insert("old-z".into(), assignment("z-old.csv"));
@@ -3689,23 +3782,48 @@ mod inventory_delta_tests {
);
}
+ #[test]
+ fn refusal_names_the_removed_files_and_every_recovery_route() {
+ let mut plan = Plan::default();
+ plan.files.insert("gone".into(), assignment("gone.csv"));
+ let error = UnsupportedInventoryDelta::between(&[], &[], &plan).into_error(targets());
+ let message = format!("{error:#}");
+ assert!(message.contains("gone.csv"), "{message}");
+ assert!(
+ message.contains("no longer exist in the folder"),
+ "{message}"
+ );
+ assert!(message.contains("made no remote mutations"), "{message}");
+ assert!(message.contains("restore the removed file(s)"), "{message}");
+ assert!(message.contains("ax-rows"), "{message}");
+ assert!(message.contains("/state"), "{message}");
+ assert!(message.contains(".xerj-memory-corpus-edges"), "{message}");
+ assert!(message.contains("new --state-dir"), "{message}");
+ assert!(message.contains("`--fresh`"), "{message}");
+ }
+
#[test]
fn cli_error_routing_separates_typed_json_from_unrelated_human_errors() {
let typed = UnsupportedInventoryDelta {
- added_content_groups: vec![InventoryDeltaEntry {
+ added_content_groups: Vec::new(),
+ vanished_content_groups: vec![InventoryDeltaEntry {
file_key: "key".into(),
- path: "new.csv".into(),
+ path: "gone.csv".into(),
}],
- vanished_content_groups: Vec::new(),
}
- .into_error();
+ .into_error(targets());
let route = route_cli_error(&typed, true);
assert_eq!(route.exit_code, 1);
assert!(route.stderr.is_none());
let stdout = route.stdout.unwrap();
let value: Value = serde_json::from_str(&stdout).unwrap();
assert_eq!(value["schema"], "xerj.autoindex.unsupported_sync_delta.v1");
- assert_eq!(value["error"], "unsupported_inventory_delta");
+ assert_eq!(value["error"], "unsupported_content_group_removal");
+ assert_eq!(value["vanished_content_groups"][0]["path"], "gone.csv");
+ assert!(value["recovery"]["rebuild_in_place"]
+ .as_str()
+ .unwrap()
+ .contains("ax-rows"));
let unrelated = anyhow::anyhow!("endpoint unavailable");
let route = route_cli_error(&unrelated, true);
@@ -3765,11 +3883,11 @@ mod duplicate_integration_tests {
.unwrap_err();
let message = format!("{error:#}");
assert!(message.contains("collides with legacy resume key"));
- // Recovery advice must stay scoped to the two colliding files and
- // keep an exact rebuild isolated from the old destination.
+ // Recovery advice must stay scoped to the two colliding files and be
+ // honest that discarding the journal re-embeds the whole corpus.
assert!(message.contains("remove or move one of these two files"));
assert!(message.contains("/state/journal.ndjson"));
- assert!(message.contains("new --state-dir, new --prefix, and new --brain"));
+ assert!(message.contains("re-extracts and re-embeds the entire corpus"));
}
#[test]
diff --git a/engine/crates/xerj-autoindex/src/state.rs b/engine/crates/xerj-autoindex/src/state.rs
index 7d4be6bec..1a63c0487 100644
--- a/engine/crates/xerj-autoindex/src/state.rs
+++ b/engine/crates/xerj-autoindex/src/state.rs
@@ -333,10 +333,9 @@ fn read_plan_for_preflight(
"journal corruption at byte {record_start} in {}: malformed \
newline-terminated record. Refusing to discard later records. Restore \
the journal from a backup, or truncate it to exactly {record_start} \
- bytes to keep every completion recorded before the corruption. For an \
- isolated rebuild use a new --state-dir, new --prefix, and new --brain \
- when graph detection is enabled (or --no-graph); explicitly validate \
- and clean the shared catalog and old target",
+ bytes to keep every completion recorded before the corruption. Deleting \
+ the whole journal (or rerunning with --fresh) also recovers, but \
+ re-extracts and re-embeds the entire corpus",
path.display()
)
})?;
@@ -494,10 +493,9 @@ impl Journal {
anyhow::bail!(
"journal at {} was created for root={jr} url={ju} prefix={jp}; \
current run has root={root} url={url} prefix={prefix}. \
- Refusing to discard history under the same destination. For an \
- isolated rebuild use a new --state-dir, new --prefix, and new \
- --brain when graph detection is enabled (or --no-graph); \
- explicitly validate and clean the shared catalog and old target.",
+ Use --state-dir for separate state, or --fresh to rebuild the \
+ plan in place — note that --fresh never deletes documents \
+ already published under the other root, url or prefix.",
jpath.display()
);
}
@@ -606,10 +604,9 @@ impl Journal {
after corruption. Restore the journal from a backup, or truncate it \
to exactly {record_start} bytes to keep every completion recorded \
before the corruption (files journaled after that point are \
- re-verified, re-indexed and re-embedded on the next run). For an \
- isolated rebuild use a new --state-dir, new --prefix, and new --brain \
- when graph detection is enabled (or --no-graph); explicitly validate \
- and clean the shared catalog and old target",
+ re-verified, re-indexed and re-embedded on the next run). Deleting \
+ the whole journal (or rerunning with --fresh) also recovers, but \
+ re-extracts and re-embeds the entire corpus",
jpath.display()
);
}
@@ -1039,10 +1036,9 @@ mod compatibility_tests {
let message = format!("{error:#}");
assert!(message.contains("journal corruption"));
// Recovery guidance must stay scoped and honest: byte-exact truncation
- // keeps prior completions; an exact rebuild needs isolated state and
- // destination identities.
+ // keeps prior completions; discarding the journal re-embeds everything.
assert!(message.contains("truncate it to exactly"));
- assert!(message.contains("new --state-dir, new --prefix, and new --brain"));
+ assert!(message.contains("re-extracts and re-embeds the entire corpus"));
}
#[test]
diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rs
index 6ba6474eb..1217b67fc 100644
--- a/engine/crates/xerj-server/src/brain.rs
+++ b/engine/crates/xerj-server/src/brain.rs
@@ -79,8 +79,9 @@ pub fn print_help() {
--api-key <K> API key for an already-running secured server\n\
(or env XERJ_API_KEY; a server booted by this command\n\
needs neither — its key is read from <data-dir>/admin.key)\n\
- --fresh start without resume state only when the selected\n\
- journal has no durable plan; never resets server data\n\
+ --fresh ignore the resume journal and rebuild the plan in place,\n\
+ re-walking everything (ids stay idempotent). It never\n\
+ deletes documents for notes you removed\n\
--no-open print the links but do not open a browser\n\
--help, -h this help\n\
\n\
@@ -386,15 +387,16 @@ fn live_node_docs(es: &Es, brain: &str) -> Result<Option<u64>> {
return Ok(None);
};
// `Es::get_doc` returns the document `_source` itself.
+ // A meta doc without `nodes_index` predates the field (or was written by
+ // something other than autoindex). That is absence of evidence, not a
+ // failure: report it as unknown instead of hard-failing every brain that
+ // an older autoindex wrote.
let Some(nodes_index) = meta
.get("nodes_index")
.and_then(Value::as_str)
.filter(|s| !s.is_empty())
else {
- bail!(
- "brain metadata {edges_index}/{} has no usable nodes_index",
- detect::BRAIN_META_ID
- );
+ return Ok(None);
};
let response = es.search(
nodes_index,
@@ -415,7 +417,11 @@ fn journal_server_disagreement(cfg: &BrainCfg, brain: &str) -> String {
"the resume journal and server disagree: journal {} says {} is already indexed, \
but {} has no confirmed live node documents for prefix ax and brain {brain}. No reset \
was attempted: an absent or zero node probe does not prove that data, catalog, and \
- graph namespaces are empty. Restore the server data directory used by this journal. \
+ graph namespaces are empty. If this is the wrong server, restore or point at the data \
+ directory this journal was written against. If the data directory really was wiped, \
+ rerun with --fresh to rebuild the plan and republish every file in place (ids are \
+ idempotent):\n\
+ xerj brain {} --url {} --fresh\n\
Otherwise, after validating or cleaning the old destination, run an isolated rebuild:\n\
xerj autoindex {} --url {} --state-dir <new-state-dir> --prefix <new-prefix> \
--brain <new-brain>\n\
@@ -426,6 +432,8 @@ fn journal_server_disagreement(cfg: &BrainCfg, brain: &str) -> String {
cfg.url,
root.display(),
cfg.url,
+ root.display(),
+ cfg.url,
)
}
@@ -700,6 +708,8 @@ mod tests {
assert!(message.contains("http://localhost:9200"));
assert!(message.contains("prefix ax"));
assert!(message.contains("brain team-notes"));
+ assert!(message.contains("xerj brain /corpus/notes"));
+ assert!(message.contains("--fresh"));
assert!(message.contains("xerj autoindex /corpus/notes"));
assert!(message.contains("--url http://localhost:9200"));
assert!(message.contains("--state-dir <new-state-dir>"));
diff --git a/landing/docs/recipes/zero-config-autoindex.html b/landing/docs/recipes/zero-config-autoindex.html
index 86d843b23..edc3d0b29 100644
--- a/landing/docs/recipes/zero-config-autoindex.html
+++ b/landing/docs/recipes/zero-config-autoindex.html
@@ -239,7 +239,8 @@ live indices at http://localhost:9280:
ax-docs 1 docs
ax-exports 800 docs
autoindex-catalog 9 docs</pre>
- <p><code class="inline">--fresh</code> starts without resume state only when the selected state directory has no durable plan; it never removes stale destination records. For an independent rebuild, use a new <code class="inline">--state-dir</code>, new <code class="inline">--prefix</code>, and new <code class="inline">--brain</code> when graph detection is enabled (or add <code class="inline">--no-graph</code>). Validate before switching readers. The shared <code class="inline">autoindex-catalog</code> and old target require explicit, validated cleanup. Added or removed content groups are refused; same-path replacement remains supported.</p>
+ <p><code class="inline">--fresh</code> ignores the journal and rebuilds the plan in place (ids stay idempotent), which is how you pick up files added since the last run.</p>
+ <p>A rerun that resumes an existing plan indexes changed files and reports added ones as skipped. Deleting a file is refused: nothing here removes the documents it already published, so the rerun stops before touching the destination and names what is gone. Restore the file and rerun, or rebuild — in place by deleting the named indices and the state directory, or isolated under a new <code class="inline">--state-dir</code>, <code class="inline">--prefix</code> and <code class="inline">--brain</code> (or <code class="inline">--no-graph</code>), validated before you switch readers. The shared <code class="inline">autoindex-catalog</code> and the old target require explicit cleanup.</p>
<hr>
diff --git a/landing/llms-full.txt b/landing/llms-full.txt
index 301c382f4..698924812 100644
--- a/landing/llms-full.txt
+++ b/landing/llms-full.txt
@@ -140,16 +140,18 @@ destabilize the server.
xerj autoindex map [--url U] [--json] [--dataset SLUG]
xerj autoindex status [--url U] [--state-dir P]
---fresh starts without resume state only when the selected state directory has
-no durable plan. It never removes stale destination records. An independent
-rebuild requires a new --state-dir, new --prefix, and new --brain when graph
-detection is enabled (or --no-graph), followed by validation before switching
-readers. The shared autoindex-catalog and old target require explicit,
-validated cleanup. Added or removed content groups are refused; same-path
-replacement remains supported.
+--fresh ignores the journal and rebuilds the plan in place (ids stay
+idempotent); it is how you pick up files added since the last run. A rerun
+that resumes an existing plan indexes changed files and reports added ones as
+skipped. Deleting an indexed file is refused before any destination change,
+because nothing here removes the documents it already published: restore the
+file and rerun, or rebuild — in place by deleting the named indices and the
+state directory, or isolated under a new --state-dir, --prefix and --brain (or
+--no-graph), validated before switching readers. The shared autoindex-catalog
+and the old target require explicit cleanup.
Exit codes: 0 complete · 3 completed-with-junk (junk recorded, never fatal) ·
-2 usage · 1 endpoint/journal error or refused unsafe corpus-state transition.
+2 usage · 1 endpoint/journal error or a refused corpus removal.
Pipeline: walk → sniff → sample → infer → map → extract → bulk → correlate →
catalog → verify.