AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): refuse an unreadable corpus source instead of pruning it (#2258)
xerj-orgb0b63423d42c
diff --git a/docs/CORPUS_PACKS.md b/docs/CORPUS_PACKS.mdindex b451e523a..084fb5b77 100644--- a/docs/CORPUS_PACKS.md+++ b/docs/CORPUS_PACKS.md@@ -86,7 +86,14 @@ There is no `--sync` mode to get wrong. Every normalised record is stored atsees the file, and skips. Only genuinely new or changed records are written,and the pack is re-emitted deterministically (fixed shard count, recordsordered by id, a uniform key set on every record) from whatever the store-holds. `--fresh` discards the build state for a full re-harvest.+holds. `--fresh` discards the build state for a full re-harvest. A `dir`+source whose path is missing, is not a directory, or cannot be listed fails+the build and does not prune envelopes already stored for that slug; an+existing empty directory is still a real empty upstream and does prune.+A previous `manifest.json` that is not valid JSON, or that has no integer+`generation`, fails the same way — `--fresh` is the recovery. Records whose+`id` is empty or not a string are refused rather than shipped under one+collapsed id.Build state lives under `~/.xerj-code/builds/<name>/` (override the root with`XERJ_CODE_HOME`) — persistent across reboots, never under `/tmp`.@@ -169,6 +176,45 @@ Publishing your own pack needs none of this — a pack directory isself-contained, and `corpus add --from <dir|zip>` is the whole install. Therelease machinery exists because *our* pack promises freshness.+## Fresh-machine portability (2026-10-23)++Checked on one Linux x86_64 host with the published `xerj v1.0.0-rc.98`+tarball (`xerj-1.0.0-rc.98-x86_64-unknown-linux-gnu.tar.gz`, `sha256sum -c`+against the release `.sha256` exited 0) and the public pack release+`pack-rust-vulns-2026-10-22`. This was not a build of the tree that produced+the pack.++- Loose `rust-vulns-SHA256SUMS` was byte-identical to `SHA256SUMS` inside+ `rust-vulns-pack.zip`. `sha256sum -c` on that file exited 0 (every named+ file printed `OK`).+- The public key fetched from+ `https://raw.githubusercontent.com/xerj-org/xerj/main/tools/packs/keys/rust-vulns.pub`+ matched `tools/packs/keys/rust-vulns.pub` in this checkout.+- `xerj corpus add rust-vulns --from rust-vulns-pack.zip --verify-sig rust-vulns.pub`+ printed `signature verified` and exited 0. It reported 1297 `rustsec`+ records and 748 `osv` records, 2045 in total, pack generation 1. The same+ 2045 is the count in that release's `rust-vulns-freshness.json` (`2045+ advisories, built 2026-10-22, gen 1`).+- A public key with one flipped nibble: `corpus add` exited 2 with+ `signature does not verify`, and the corpus directory stayed empty.+ Running `corpus index` after that refusal (exit status discarded) exited 2+ with `no corpus` and still wrote nothing.+- A zip whose `records-00000.jsonl` had one byte flipped, checksums not+ rewritten: `corpus add --verify-sig` exited 2 with `checksum mismatch` and+ wrote nothing.+- Against `xerj --insecure --port 19200`, `xerj corpus index rust-vulns`+ exited 0. The `xerj-done` line was `ok=true exit=0 reason=completed+ wall=6.9s files=2 records=2045 generation=1`.+- `xerj code rust-vulns "smallvec insert_many buffer overflow" --url+ http://127.0.0.1:19200` exited 0. The top passage was the RustSec record+ titled "Buffer overflow in SmallVec::insert_many" (score 27.50). The hit+ was labelled `line unknown`.++`cisa-kev` was not part of this run. `gh release list -R xerj-org/xerj -L 200`+on 2026-10-23 returned no tag containing `cisa` or `kev`, and+`tools/packs/cisa-kev/README.md` says that pack is not published or signed.+There is no public URL to download, checksum, or query.+## Honest limits- **Indexing still costs each consumer the same CPU-hours.** The packdiff --git a/engine/crates/xerj-autoindex/src/harvest/mod.rs b/engine/crates/xerj-autoindex/src/harvest/mod.rsindex 567fbccde..58dd1b60c 100644--- a/engine/crates/xerj-autoindex/src/harvest/mod.rs+++ b/engine/crates/xerj-autoindex/src/harvest/mod.rs@@ -309,14 +309,35 @@ fn walk(root: &Path, glob: &str) -> Result<Vec<(String, PathBuf)>> {}let re = glob_regex(glob);let mut out = Vec::new();- fn recurse(dir: &Path, root: &Path, re: ®ex::Regex, out: &mut Vec<(String, PathBuf)>) {- let Ok(entries) = std::fs::read_dir(dir) else {- return;- };- for e in entries.flatten() {+ // A listing error must not become "no files". The autoindex walker+ // skips an unreadable entry (`walk.rs:327`, the `Err` arm of `it.next()`)+ // because a skipped file is only absent from that pass. Here `prune`+ // treats "not seen" as "deleted upstream", so swallowing `read_dir` —+ // on the root or on any nested directory — deletes every envelope under+ // it and the build still exits 0 (#1110 gap 1). `metadata` above already+ // refuses a missing root; this is the root that exists but cannot be+ // listed, and the same failure one directory down.+ fn recurse(+ dir: &Path,+ root: &Path,+ re: ®ex::Regex,+ out: &mut Vec<(String, PathBuf)>,+ ) -> Result<()> {+ let entries = std::fs::read_dir(dir).with_context(|| {+ format!(+ "source directory {} cannot be listed — an unreadable tree must \+ fail the build, not silently empty the pack",+ dir.display()+ )+ })?;+ for e in entries {+ let e = e.with_context(|| format!("cannot read entry in {}", dir.display()))?;let p = e.path();+ // `Path::is_dir` follows symlinks, which is the walk this+ // replaced. A stat failure reports "not a directory" (same as+ // before); the failure this function refuses is `read_dir`.if p.is_dir() {- recurse(&p, root, re, out);+ recurse(&p, root, re, out)?;} else {let rel = p.strip_prefix(root).unwrap_or(&p);let rel = rel.to_string_lossy().replace('\\', "/");@@ -325,8 +346,9 @@ fn walk(root: &Path, glob: &str) -> Result<Vec<(String, PathBuf)>> {}}}+ Ok(())}- recurse(root, root, &re, &mut out);+ recurse(root, root, &re, &mut out)?;out.sort();Ok(out)}@@ -485,6 +507,84 @@ mod tests {assert!(err.contains("is not a directory"), "{err}");}+ /// `chmod` a path for the duration of the test, then put the previous+ /// mode back — including on panic — so the tempdir can be removed.+ #[cfg(unix)]+ struct RestoreMode {+ path: PathBuf,+ mode: u32,+ }++ #[cfg(unix)]+ impl RestoreMode {+ fn lock(path: &Path, mode: u32) -> Self {+ use std::os::unix::fs::PermissionsExt;+ let prev = std::fs::metadata(path).unwrap().permissions().mode();+ let mut perms = std::fs::metadata(path).unwrap().permissions();+ perms.set_mode(mode);+ std::fs::set_permissions(path, perms).unwrap();+ RestoreMode {+ path: path.to_path_buf(),+ mode: prev,+ }+ }+ }++ #[cfg(unix)]+ impl Drop for RestoreMode {+ fn drop(&mut self) {+ use std::os::unix::fs::PermissionsExt;+ if let Ok(meta) = std::fs::metadata(&self.path) {+ let mut perms = meta.permissions();+ perms.set_mode(self.mode);+ std::fs::set_permissions(&self.path, perms).ok();+ }+ }+ }++ #[cfg(unix)]+ #[test]+ fn walk_refuses_an_unreadable_root() {+ // #1110 gap 1, the read_dir half: metadata succeeding is not enough.+ // A directory that exists but cannot be listed (mode 000) used to+ // fall through `recurse`'s swallowed `read_dir` and come back as an+ // empty file list — the same silent prune as a typo'd path.+ let tmp = tempfile::tempdir().unwrap();+ let root = tmp.path().join("src");+ std::fs::create_dir_all(root.join("keep")).unwrap();+ std::fs::write(root.join("keep/a.json"), "{}").unwrap();+ let _restore = RestoreMode::lock(&root, 0o000);+ let err = walk(&root, "**/*.json").unwrap_err().to_string();+ assert!(+ err.contains("cannot be listed"),+ "unreadable root must fail the walk, not return no files: {err}"+ );+ }++ #[cfg(unix)]+ #[test]+ fn walk_refuses_an_unreadable_subdirectory() {+ // A nested directory that cannot be listed must fail the walk too.+ // Skipping it would omit its files from the live key set and `prune`+ // would delete their envelopes as if upstream had removed them.+ let tmp = tempfile::tempdir().unwrap();+ let root = tmp.path().join("src");+ std::fs::create_dir_all(root.join("keep")).unwrap();+ std::fs::create_dir_all(root.join("secret")).unwrap();+ std::fs::write(root.join("keep/a.json"), "{}").unwrap();+ std::fs::write(root.join("secret/b.json"), "{}").unwrap();+ let _restore = RestoreMode::lock(&root.join("secret"), 0o000);+ let err = walk(&root, "**/*.json").unwrap_err().to_string();+ assert!(+ err.contains("cannot be listed"),+ "unreadable subdirectory must fail the walk: {err}"+ );+ assert!(+ err.contains("secret"),+ "the error must name the directory that could not be listed: {err}"+ );+ }+#[test]fn parse_file_understands_json_and_jsonl() {let tmp = tempfile::tempdir().unwrap();@@ -834,6 +934,254 @@ shards = 4}}+ fn store_files(home: &Path, build: &str, prefix: &str) -> Vec<String> {+ let store = home.join("builds").join(build).join("store");+ let mut names: Vec<String> = std::fs::read_dir(&store)+ .unwrap()+ .flatten()+ .map(|e| e.file_name().to_string_lossy().to_string())+ .filter(|n| n.starts_with(prefix))+ .collect();+ names.sort();+ names+ }++ fn pack_generation(home: &Path, build: &str) -> u64 {+ let manifest: Value = serde_json::from_str(+ &std::fs::read_to_string(+ home.join("builds")+ .join(build)+ .join("pack")+ .join(build)+ .join("manifest.json"),+ )+ .unwrap(),+ )+ .unwrap();+ manifest["generation"].as_u64().unwrap()+ }++ #[test]+ fn missing_source_root_fails_the_build_and_does_not_prune() {+ // #1110 gap 1 at the pipeline: a typo'd `dir` path must exit as an+ // error AND leave the previously stored envelopes on disk. `walk`+ // failing is not enough if `prune` already ran.+ let home = tempfile::tempdir().unwrap();+ e2e_env(home.path());+ let args: Vec<String> = vec!["e2e".to_string()];+ assert_eq!(run_build_inner(&args, home.path()).unwrap(), 0);+ let before = store_files(home.path(), "e2e", "rustsec.");+ assert_eq!(before.len(), 2, "two rustsec envelopes stored");+ let gen = pack_generation(home.path(), "e2e");++ std::fs::rename(+ home.path().join("recipes/data/rustsec"),+ home.path().join("recipes/data/rustsec-moved"),+ )+ .unwrap();+ let err = run_build_inner(&args, home.path()).unwrap_err().to_string();+ assert!(+ err.contains("cannot be read"),+ "missing source root must fail the build: {err}"+ );+ assert_eq!(+ store_files(home.path(), "e2e", "rustsec."),+ before,+ "a failed walk must not prune the slug's envelopes"+ );+ assert_eq!(+ pack_generation(home.path(), "e2e"),+ gen,+ "a failed walk must not publish a new generation"+ );+ }++ #[cfg(unix)]+ #[test]+ fn unreadable_source_root_fails_the_build_and_does_not_prune() {+ // Same contract as the missing-root test, for a directory that+ // exists but whose `read_dir` fails. That is the path `recurse`+ // used to swallow.+ let home = tempfile::tempdir().unwrap();+ e2e_env(home.path());+ let args: Vec<String> = vec!["e2e".to_string()];+ assert_eq!(run_build_inner(&args, home.path()).unwrap(), 0);+ let before = store_files(home.path(), "e2e", "rustsec.");+ assert_eq!(before.len(), 2);+ let gen = pack_generation(home.path(), "e2e");++ let root = home.path().join("recipes/data/rustsec");+ let _restore = RestoreMode::lock(&root, 0o000);+ let err = run_build_inner(&args, home.path()).unwrap_err().to_string();+ assert!(+ err.contains("cannot be listed"),+ "unreadable source root must fail the build: {err}"+ );+ assert_eq!(+ store_files(home.path(), "e2e", "rustsec."),+ before,+ "an unreadable root must not prune the slug's envelopes"+ );+ assert_eq!(pack_generation(home.path(), "e2e"), gen);+ }++ fn idless_recipe(name: &str, glob: &str) -> String {+ format!(+ r#"[recipe]+format = 1+name = "{name}"++[[sources]]+slug = "notes"+kind = "dir"+path = "notes"+glob = "{glob}"+format = "flat"+licence = "CC0-1.0"++[identity]+edges = [{{ field = "id" }}]+"#+ )+ }++ #[test]+ fn markdown_blobs_without_ids_are_refused() {+ // #1110 gap 2, the normalize path: `.md` files flattened by `flat`+ // become `{"value": …}` with no id. N of them must not ship in one+ // shard under id "".+ let home = tempfile::tempdir().unwrap();+ let notes = home.path().join("recipes/notes");+ std::fs::create_dir_all(¬es).unwrap();+ std::fs::write(+ home.path().join("recipes/idless.toml"),+ idless_recipe("idless", "**/*.md"),+ )+ .unwrap();+ for (i, body) in ["alpha", "bravo", "charlie", "delta"].iter().enumerate() {+ std::fs::write(notes.join(format!("n{i}.md")), format!("# {body}\n")).unwrap();+ }+ let err = run_build_inner(&["idless".into()], home.path())+ .unwrap_err()+ .to_string();+ assert!(err.contains("empty id"), "{err}");+ assert_eq!(+ store_files(home.path(), "idless", "notes.").len(),+ 4,+ "all four blobs were harvested; the refusal is at emit, not a silent drop"+ );+ let pack = home.path().join("builds/idless/pack/idless");+ assert!(pack.is_dir(), "emit creates the pack dir before refusing");+ let shards: Vec<_> = std::fs::read_dir(&pack)+ .unwrap()+ .flatten()+ .map(|e| e.file_name().to_string_lossy().to_string())+ .filter(|n| n.starts_with("records-"))+ .collect();+ assert!(+ shards.is_empty(),+ "refused records must not be written into the pack: {shards:?}"+ );+ }++ #[test]+ fn non_string_and_empty_ids_are_refused() {+ // #1110 gap 2: a non-string `id` (number, null, missing) is not a+ // stable identity. N such records must be refused, not collapsed.+ let home = tempfile::tempdir().unwrap();+ let notes = home.path().join("recipes/notes");+ std::fs::create_dir_all(¬es).unwrap();+ std::fs::write(+ home.path().join("recipes/idless.toml"),+ idless_recipe("idless", "**/*.json"),+ )+ .unwrap();+ let bodies = [+ r#"{"id":""}"#,+ r#"{"id":7}"#,+ r#"{"id":null,"name":"n"}"#,+ r#"{"note":"no id field"}"#,+ ];+ for (i, body) in bodies.iter().enumerate() {+ std::fs::write(notes.join(format!("n{i}.json")), body).unwrap();+ }+ let err = run_build_inner(&["idless".into()], home.path())+ .unwrap_err()+ .to_string();+ assert!(err.contains("empty id"), "{err}");+ assert_eq!(store_files(home.path(), "idless", "notes.").len(), 4);+ let pack = home.path().join("builds/idless/pack/idless");+ if pack.is_dir() {+ for entry in std::fs::read_dir(&pack).unwrap().flatten() {+ let name = entry.file_name().to_string_lossy().to_string();+ assert!(+ !name.starts_with("records-"),+ "refused records must not be written: {name}"+ );+ }+ }+ }++ #[test]+ fn corrupt_store_envelope_fails_the_rebuild_without_shrinking_the_pack() {+ // #1110 gap 4 at the pipeline: a truncated `<slug>.<key>.json` must+ // fail the rebuild. The previous pack stays, and the corrupt file is+ // not pruned (its key is still live) and then quietly omitted.+ let home = tempfile::tempdir().unwrap();+ e2e_env(home.path());+ let args: Vec<String> = vec!["e2e".to_string()];+ assert_eq!(run_build_inner(&args, home.path()).unwrap(), 0);+ let before = store_files(home.path(), "e2e", "");+ let gen = pack_generation(home.path(), "e2e");+ let records_before = read_records(&home.path().join("builds/e2e/pack/e2e")).len();++ let victim = home+ .path()+ .join("builds/e2e/store")+ .join(before.iter().find(|n| n.starts_with("rustsec.")).unwrap());+ std::fs::write(&victim, "{truncated").unwrap();++ let err = run_build_inner(&args, home.path()).unwrap_err().to_string();+ assert!(err.contains("corrupt store envelope"), "{err}");+ assert!(+ victim.is_file(),+ "the corrupt envelope must still be on disk, not pruned"+ );+ assert_eq!(pack_generation(home.path(), "e2e"), gen);+ assert_eq!(+ read_records(&home.path().join("builds/e2e/pack/e2e")).len(),+ records_before,+ "the published pack must not shrink"+ );+ }++ #[test]+ fn fresh_rebuilds_after_a_corrupt_manifest() {+ // #1110 gap 5: a corrupt manifest.json fails a normal rebuild (it+ // must not reset generation to 0), and `--fresh` is the recovery —+ // it discards the build dir and harvests again.+ let home = tempfile::tempdir().unwrap();+ e2e_env(home.path());+ let args: Vec<String> = vec!["e2e".to_string()];+ assert_eq!(run_build_inner(&args, home.path()).unwrap(), 0);+ let manifest_path = home.path().join("builds/e2e/pack/e2e/manifest.json");+ std::fs::write(&manifest_path, "{truncated").unwrap();++ let err = run_build_inner(&args, home.path()).unwrap_err().to_string();+ assert!(err.contains("corrupt previous manifest"), "{err}");+ // the failed rebuild must not have replaced the torn manifest with+ // a generation-1 success+ assert_eq!(std::fs::read(&manifest_path).unwrap(), b"{truncated");++ let fresh: Vec<String> = vec!["e2e".into(), "--fresh".into()];+ assert_eq!(run_build_inner(&fresh, home.path()).unwrap(), 0);+ let manifest: Value =+ serde_json::from_str(&std::fs::read_to_string(&manifest_path).unwrap()).unwrap();+ assert_eq!(manifest["generation"], json!(1));+ assert_eq!(manifest["counts"]["records"], json!(2));+ assert_eq!(read_records(manifest_path.parent().unwrap()).len(), 2);+ }+// ── end to end: a git source, watermark-skipped on the second run ─────#[test]diff --git a/engine/crates/xerj-autoindex/src/harvest/pack.rs b/engine/crates/xerj-autoindex/src/harvest/pack.rsindex 839035dc1..d78468589 100644--- a/engine/crates/xerj-autoindex/src/harvest/pack.rs+++ b/engine/crates/xerj-autoindex/src/harvest/pack.rs@@ -901,6 +901,99 @@ mod tests {assert!(err.contains(&fname), "{err}");}+ /// Point `manifest.files[fname].sha256` at the bytes now on disk and+ /// rewrite SHA256SUMS over every name it already lists. The per-file+ /// `records` integer is left untouched, so a following `read_manifest`+ /// can fail only on the line-count check.+ fn recompute_sums_leaving_record_counts(pack_dir: &Path, fname: &str) {+ let bytes = std::fs::read(pack_dir.join(fname)).unwrap();+ let sha = hex(&Sha256::digest(&bytes));+ let mut m: Value =+ serde_json::from_str(&std::fs::read_to_string(pack_dir.join("manifest.json")).unwrap())+ .unwrap();+ m["files"][fname]["sha256"] = json!(sha);+ std::fs::write(+ pack_dir.join("manifest.json"),+ serde_json::to_vec_pretty(&m).unwrap(),+ )+ .unwrap();+ let sums_raw = std::fs::read_to_string(pack_dir.join("SHA256SUMS")).unwrap();+ let mut fresh = String::new();+ for line in sums_raw.lines() {+ let (_, name) = line.split_once(" ").unwrap();+ let file_bytes = std::fs::read(pack_dir.join(name)).unwrap();+ fresh.push_str(&format!(+ "{} {}\n",+ hex(&Sha256::digest(&file_bytes)),+ name+ ));+ }+ std::fs::write(pack_dir.join("SHA256SUMS"), fresh).unwrap();+ }++ #[test]+ fn read_manifest_count_check_refuses_a_torn_line_when_sums_match() {+ // #1110 gap 3, the byte-tamper shape: SHA256SUMS is recomputed over+ // the tampered records (and the manifest sha is updated to match), so+ // the checksum branches all pass. A torn final line that changes the+ // line count is then refused by the record-count check alone.+ // `read_manifest_catches_a_record_count_lie_with_valid_checksums`+ // reaches the same branch by lying about the integer; this one+ // tampers the file.+ let tmp = tempfile::tempdir().unwrap();+ let records = vec![rec("a", &[]), rec("b", &[]), rec("c", &[])];+ let r = emit(+ tmp.path(),+ &recipe("t", 1),+ "# r\n",+ records.clone(),+ &stats_with_source(),+ &sugg(&records),+ )+ .unwrap();+ read_manifest(&r.dir).unwrap();+ let fname = meta_file_name(&r.dir);+ let path = r.dir.join(&fname);+ let original = std::fs::read(&path).unwrap();++ // appended torn final line: a partial JSON object with no closing+ // brace and no trailing newline. `.lines()` counts it, so the file+ // holds one more record than the manifest still claims.+ let mut torn = original.clone();+ torn.extend_from_slice(b"{\"id\":\"TORN\"");+ std::fs::write(&path, &torn).unwrap();+ recompute_sums_leaving_record_counts(&r.dir, &fname);+ let err = read_manifest(&r.dir).unwrap_err().to_string();+ assert!(+ err.contains("manifest says"),+ "count check must be the refusal, not a checksum: {err}"+ );+ assert!(+ !err.contains("checksum mismatch") && !err.contains("disagree"),+ "checksums were recomputed; they must not be why this fails: {err}"+ );+ assert!(err.contains("file holds"), "{err}");+ assert!(err.contains(&fname), "{err}");++ // truncated final record: the last complete line is cut away, as a+ // writer that died before flushing it. Count drops; checksums are+ // honest again; the count check still refuses.+ std::fs::write(&path, &original).unwrap();+ let text = String::from_utf8(original.clone()).unwrap();+ let trimmed = text.trim_end_matches('\n');+ let (head, last) = trimmed.rsplit_once('\n').unwrap();+ assert!(last.contains("\"id\""), "fixture has a final record line");+ std::fs::write(&path, format!("{head}\n")).unwrap();+ recompute_sums_leaving_record_counts(&r.dir, &fname);+ let err = read_manifest(&r.dir).unwrap_err().to_string();+ assert!(err.contains("manifest says"), "{err}");+ assert!(+ !err.contains("checksum mismatch") && !err.contains("disagree"),+ "{err}"+ );+ assert!(err.contains(&fname), "{err}");+ }+#[test]fn sums_cover_every_artifact() {let tmp = tempfile::tempdir().unwrap();