AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(hub): let `xerj hub pull` restore into a user-started node by default (#1734)
xerj-orga568f7cc5d35
diff --git a/docs/CORPUS_HUB_V2_IMPLEMENTATION.md b/docs/CORPUS_HUB_V2_IMPLEMENTATION.mdindex 1cd25498c..0c8d5eb8c 100644--- a/docs/CORPUS_HUB_V2_IMPLEMENTATION.md+++ b/docs/CORPUS_HUB_V2_IMPLEMENTATION.md@@ -425,6 +425,12 @@ health check fails, start `current_exe() --insecure --port <free port ≥ 9300>detached, PID to `~/.xerj/hub/node.pid`, wait for green (max 60 s), remember the portin `~/.xerj/hub/node.port`. Never start a node on a port that answers. `xerj hub status`shows it; `xerj hub stop` stops only that PID.+A node the user started themselves needs no config for `xerj hub pull` to restore+into it: by default a server allows snapshot **restores** (never creates) from the+pack cache of the user it runs as (`limits.hub_pack_restore`, `xerj_common::hub::hub_packs_dir`,+refused if that directory is a symlink or owned by another user). A server that does+not share the cache (other user, container, other machine) refuses, and the pull+error prints the exact `snapshot_repo_allowlist` line to add.**Tests** (`xerj-hub/tests/routing.rs`, `xerj-hub/tests/ask_flow.rs`):diff --git a/docs/SNAPSHOT_AND_RESTORE.md b/docs/SNAPSHOT_AND_RESTORE.mdindex 49c65f746..907a1b820 100644--- a/docs/SNAPSHOT_AND_RESTORE.md+++ b/docs/SNAPSHOT_AND_RESTORE.md@@ -53,6 +53,19 @@ to empty, which means only `data_dir` is permitted. This is checked when asnapshot is created or restored, not when the repository is registered, so abad location is accepted by `PUT /_snapshot/{repo}` and only fails later.+One more root is allowed for **restore only**: the Corpus Hub pack cache of the+user the server runs as (`$XERJ_HUB_HOME/packs`, else `~/.xerj/hub/packs`),+which is where `xerj hub pull` and `xerj ask` extract a verified, signed pack+before asking the server to restore it. It is controlled by+`limits.hub_pack_restore` (default `true`; `false` turns it off). The grant is+narrower than an allowlist entry: a snapshot can never be *created* there, the+directory must be a real directory (not a symlink) owned by the server's user,+and the repository location is still canonicalized, so a symlink inside the+cache that points elsewhere is refused. A server running as another user, in a+container or on another machine does not share the cache; there, add the+directory to `limits.snapshot_repo_allowlist` (the `xerj hub pull` error prints+the exact line).+If `settings.location` is absent the handler falls back to`/tmp/xerj-snapshots` (`es_compat.rs:23201`). That fallback is outside thedefault `data_dir` and will therefore be refused by the location check, sodiff --git a/engine/README.md b/engine/README.mdindex 3dade6255..36fe1da49 100644--- a/engine/README.md+++ b/engine/README.md@@ -53,7 +53,7 @@ HTTP wire payloads, single-node, tmpfs data dir, both freshly started| Per-batch (5K) bulk p99 | **34.8 ms** | 80.3 ms | ES 2.3× || Data loss on restart | 0% | 0% | tied || GC pauses | 20–40 ms (JVM) | **0** (no GC) | **xerj** |-| Config surface | 3,000+ knobs | **134**, all optional | **xerj** |+| Config surface | 3,000+ knobs | **135**, all optional | **xerj** |xerj wins on every operational and read-path metric. ES still winsbulk-ingest throughput on warm Lucene `IndexWriter` (perf-backlog item@@ -86,7 +86,7 @@ The resulting binary is at `target/release/xerj`.## ConfigurationSee [`xerj.default.toml`](xerj.default.toml) for the commonly-tuned settings, and-[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 134+[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 135— every one of them optional.The minimal config to get started:diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 6331904ef..30d36b6b6 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1,6 +1,6 @@//! xerj configuration system.//!-//! Configuration is intentionally minimal: **134 settings** versus+//! Configuration is intentionally minimal: **135 settings** versus//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible//! production-ready default. The format is TOML, loaded from a single file.//!@@ -108,7 +108,7 @@ pub struct Config {pub hub: HubConfig,}-// 24 sub-configs, 134 leaf settings in total. Do not maintain that sum by hand+// 24 sub-configs, 135 leaf settings in total. Do not maintain that sum by hand// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a// serialised `Config::default()` and fails if this comment and the module// header stop matching. `Default` is derived: every field is a sub-config that@@ -563,7 +563,7 @@ impl Config {}// ═════════════════════════════════════════════════════════════════════════════-// Sub-configs (134 user-facing settings total; counted by+// Sub-configs (135 user-facing settings total; counted by// `journey_zero_config`, not by hand)// ═════════════════════════════════════════════════════════════════════════════@@ -1703,7 +1703,7 @@ pub const AUTO_PROCESS_MEMORY_MB: u64 = u64::MAX;/// Resource limits.///-/// **14 settings.**+/// **15 settings.**#[derive(Debug, Clone, Serialize, Deserialize)]#[serde(default, deny_unknown_fields)]pub struct LimitsConfig {@@ -1733,6 +1733,22 @@ pub struct LimitsConfig {/// path inside `data_dir` or inside one of these bases.#[serde(default)]pub snapshot_repo_allowlist: Vec<String>,+ /// Let a snapshot **restore** (never a create) read from the Corpus Hub pack+ /// cache of the user the server runs as — `$XERJ_HUB_HOME/packs`, else+ /// `~/.xerj/hub/packs` (`xerj_common::hub::hub_packs_dir`) — without listing+ /// it in `snapshot_repo_allowlist` (default: `true`).+ ///+ /// `xerj hub pull` / `xerj ask` download a signed pack, verify it, extract it+ /// under that directory and ask the server to restore it. Before this, only+ /// the node XERJ starts for itself allowlisted the directory, so a server the+ /// user started by hand refused every pull. The grant is deliberately+ /// narrower than an allowlist entry: restore only (a client cannot write a+ /// snapshot into the user's home), honoured only while the directory is a+ /// real directory — not a symlink — owned by the server's own user, and the+ /// repo location is still canonicalized, so a symlink inside the cache that+ /// points elsewhere is refused. Set `false` to allow `data_dir` and the+ /// allowlist only.+ pub hub_pack_restore: bool,/// Maximum HTTP request body size in bytes (default: `100 * 1024 * 1024`,/// i.e. 100 MiB). Caps NDJSON bulk payloads, large mget bodies, etc./// Raise this only if your client routinely sends bigger requests; the@@ -1838,6 +1854,7 @@ impl Default for LimitsConfig {max_fields_per_index: 500,max_actions_per_bulk: 50_000,snapshot_repo_allowlist: Vec::new(),+ hub_pack_restore: true,max_body_bytes: 100 * 1024 * 1024,max_result_window: 10_000,max_mget_docs: 10_000,@@ -3702,7 +3719,7 @@ mod tests {("vector", 6),("logs", 2),("embedding", 20),- ("limits", 14),+ ("limits", 15),("indexing", 3),("engine", 4),("cluster", 5),@@ -3757,7 +3774,7 @@ mod tests {"the section table must sum to the whole config");assert_eq!(- total, 134,+ total, 135,"the total settings count changed. It is quoted in this module's \header, in xerj-common/src/lib.rs, in engine/README.md, in \xerj.default.toml and in EXPECTED_SETTINGS in \diff --git a/engine/crates/xerj-common/src/hub.rs b/engine/crates/xerj-common/src/hub.rsindex f9b0959e3..89bd7342b 100644--- a/engine/crates/xerj-common/src/hub.rs+++ b/engine/crates/xerj-common/src/hub.rs@@ -29,6 +29,33 @@ pub const PACK_FORMAT_VERSION: u32 = 1;/// Where the catalogue and the hosted query API live.pub const DEFAULT_HUB_URL: &str = "https://hub.xerj.org";+/// The hub's local root: `XERJ_HUB_HOME` when set and non-empty, else+/// `<home>/.xerj/hub` ([`crate::home::home_dir_from`]: `HOME`, else+/// `USERPROFILE`). `None` when neither is set.+///+/// This is THE rule. The CLI (`xerj hub pull`, which extracts packs under+/// `<root>/packs`) and the server (which, by default, lets a snapshot restore+/// read from that same `<root>/packs` — `limits.hub_pack_restore`) both resolve+/// it here, so a server started by the same user agrees with the CLI on where a+/// pulled pack lives.+pub fn hub_root_from(+ env: &dyn Fn(&str) -> Option<std::ffi::OsString>,+) -> Option<std::path::PathBuf> {+ if let Some(h) = env("XERJ_HUB_HOME").filter(|h| !h.is_empty()) {+ return Some(std::path::PathBuf::from(h));+ }+ crate::home::home_dir_from(env).map(|home| home.join(".xerj").join("hub"))+}++/// The directory `xerj hub pull` extracts verified packs into:+/// [`hub_root_from`]`/packs`, over the process environment.+pub fn hub_packs_dir() -> Option<std::path::PathBuf> {+ hub_root_from(&|key| std::env::var_os(key)).map(|root| root.join(HUB_PACKS_DIR))+}++/// The pack-cache directory name under the hub root.+pub const HUB_PACKS_DIR: &str = "packs";+/// Packs at or below this size are pulled automatically; larger ones are/// answered through the hosted route until the user runs `xerj hub pull`.pub const AUTO_PULL_MAX_BYTES: u64 = 50 * 1024 * 1024;@@ -338,6 +365,38 @@ pub fn render(answer: &Answer) -> String {#[cfg(test)]mod tests {+ fn hub_env<'a>(+ vars: &'a [(&'a str, &'a str)],+ ) -> impl Fn(&str) -> Option<std::ffi::OsString> + 'a {+ move |key| {+ vars.iter()+ .find(|(k, _)| *k == key)+ .map(|(_, v)| std::ffi::OsString::from(v))+ }+ }++ #[test]+ fn the_hub_root_rule_is_xerj_hub_home_then_home_then_nothing() {+ use std::path::PathBuf;+ assert_eq!(+ hub_root_from(&hub_env(&[+ ("XERJ_HUB_HOME", "/srv/hub"),+ ("HOME", "/home/u")+ ])),+ Some(PathBuf::from("/srv/hub"))+ );+ assert_eq!(+ hub_root_from(&hub_env(&[("XERJ_HUB_HOME", ""), ("HOME", "/home/u")])),+ Some(PathBuf::from("/home/u/.xerj/hub"))+ );+ assert_eq!(+ hub_root_from(&hub_env(&[("USERPROFILE", "/Users/u")])),+ Some(PathBuf::from("/Users/u/.xerj/hub"))+ );+ // No home at all: the server grants nothing (never `./.xerj/hub`).+ assert_eq!(hub_root_from(&hub_env(&[])), None);+ }+use super::*;use serde_json::json;diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rsindex 7aa309d5a..da8b7d4ab 100644--- a/engine/crates/xerj-common/src/lib.rs+++ b/engine/crates/xerj-common/src/lib.rs@@ -7,13 +7,13 @@//! ## Design philosophy//!//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes-//! **134 settings**, each meaningful and production-tested. Every default+//! **135 settings**, each meaningful and production-tested. Every default//! is chosen so that a fresh deployment with zero configuration changes performs//! well for the majority of workloads.//!//! ## Modules//!-//! - [`config`] — TOML-based configuration (134 settings)+//! - [`config`] — TOML-based configuration (135 settings)//! - [`code_query`] — query shaping shared by `xerj search` and the//! `xerj_search` MCP tool (multi-word glue-token stripping, #1364)//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex 047b9f13e..af70af5d1 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -4418,12 +4418,15 @@ impl Engine {) -> Result<Value> {self.ensure_cluster_state_writable()?;let snap_dir = std::path::Path::new(repo_path).join(name);+ // No hub-pack root here: the pack cache is a restore-only source, so a+ // client can never write a snapshot into the user's home directory.let snap_dir = validate_snapshot_path(repo_path,name,&snap_dir,&self.data_dir,&self.config.limits.snapshot_repo_allowlist,+ None,)?;std::fs::create_dir_all(&snap_dir).map_err(EngineError::Io)?;@@ -4517,12 +4520,22 @@ impl Engine {) -> Result<Vec<String>> {self.ensure_cluster_state_writable()?;let snap_dir = std::path::Path::new(repo_path).join(name);+ // `limits.hub_pack_restore`: the Corpus Hub pack cache of the user this+ // server runs as is a restore source by default, so `xerj hub pull`+ // works against a server the user started by hand (not only against the+ // node `xerj ask` starts, which allowlists it in its own config).+ let hub_packs = if self.config.limits.hub_pack_restore {+ xerj_common::hub::hub_packs_dir()+ } else {+ None+ };let snap_dir = validate_snapshot_path(repo_path,name,&snap_dir,&self.data_dir,&self.config.limits.snapshot_repo_allowlist,+ hub_packs.as_deref(),)?;let manifest_path = snap_dir.join("manifest.json");@@ -4740,6 +4753,7 @@ fn validate_snapshot_path(snap_dir: &std::path::Path,data_dir: &std::path::Path,allowlist: &[String],+ hub_packs: Option<&std::path::Path>,) -> Result<std::path::PathBuf> {if repo_path.is_empty() {return Err(EngineError::Common(@@ -4780,6 +4794,10 @@ fn validate_snapshot_path(.iter().map(|b| base_canon(std::path::Path::new(b))),);+ // The hub pack cache (restore only — create passes `None`). Unlike an+ // operator's allowlist entry it is implicit, so it is trusted only in the+ // shape `xerj hub pull` makes it; see `trusted_hub_pack_root`.+ allowed_bases.extend(hub_packs.and_then(trusted_hub_pack_root));if !allowed_bases.iter().any(|base| repo_base.starts_with(base)) {return Err(EngineError::Common(xerj_common::XerjError::invalid_mapping(format!(@@ -4833,6 +4851,50 @@ fn validate_snapshot_path(}}+/// The canonical hub pack-cache root a restore may read from, or `None` when+/// `packs` is not safe to trust implicitly.+///+/// `limits.snapshot_repo_allowlist` entries are canonicalized as given: an+/// operator who allowlists a symlink means its target. The hub root is NOT an+/// operator's choice — it is derived from the environment — so it is held to+/// the shape `xerj hub pull` creates (`create_dir_all`, by the same user):+///+/// - it exists and is a real directory, not a symlink (`symlink_metadata`),+/// so `~/.xerj/hub/packs -> /` cannot widen the grant to the whole disk;+/// - canonicalizing it changes nothing but its parent's resolution (its own+/// last component is not a link);+/// - it is not a filesystem root;+/// - on Unix it is owned by the server's effective user, so a shared or+/// pre-created directory (a `HOME` under a world-writable dir) is not+/// honoured.+///+/// What is inside it is still checked per request: the caller canonicalizes the+/// repo location, so `packs/x -> /etc` resolves outside this root and is+/// refused, and `..` components are rejected before this is consulted.+fn trusted_hub_pack_root(packs: &std::path::Path) -> Option<std::path::PathBuf> {+ if !packs.is_absolute() {+ return None;+ }+ let meta = std::fs::symlink_metadata(packs).ok()?;+ if meta.file_type().is_symlink() || !meta.is_dir() {+ return None;+ }+ #[cfg(unix)]+ {+ use std::os::unix::fs::MetadataExt;+ // SAFETY: geteuid has no preconditions and cannot fail.+ if meta.uid() != unsafe { libc::geteuid() } {+ return None;+ }+ }+ let canon = packs.canonicalize().ok()?;+ let parent = packs.parent()?.canonicalize().ok()?;+ if canon != parent.join(packs.file_name()?) || canon.parent().is_none() {+ return None;+ }+ Some(canon)+}+/// Recursively copy all files from `src` to `dst`, recording relative paths in `files`.fn copy_dir_recursive(src: &std::path::Path,@@ -4911,7 +4973,7 @@ mod snapshot_path_security_tests {let outside = tempfile::TempDir::new().unwrap();let repo = outside.path().to_str().unwrap();let snap = std::path::Path::new(repo).join("s1");- let err = validate_snapshot_path(repo, "s1", &snap, data.path(), &[])+ let err = validate_snapshot_path(repo, "s1", &snap, data.path(), &[], None).expect_err("external repo location must be rejected");assert!(err.to_string().contains("outside data_dir"), "{err}");}@@ -4923,7 +4985,7 @@ mod snapshot_path_security_tests {std::fs::create_dir_all(&repo_dir).unwrap();let repo = repo_dir.to_str().unwrap();let snap = repo_dir.join("s1");- validate_snapshot_path(repo, "s1", &snap, data.path(), &[])+ validate_snapshot_path(repo, "s1", &snap, data.path(), &[], None).expect("a repo under data_dir must be allowed");}@@ -4933,8 +4995,15 @@ mod snapshot_path_security_tests {let ext = tempfile::TempDir::new().unwrap();let repo = ext.path().to_str().unwrap().to_string();let snap = std::path::Path::new(&repo).join("s1");- validate_snapshot_path(&repo, "s1", &snap, data.path(), std::slice::from_ref(&repo))- .expect("an allowlisted external root must be permitted");+ validate_snapshot_path(+ &repo,+ "s1",+ &snap,+ data.path(),+ std::slice::from_ref(&repo),+ None,+ )+ .expect("an allowlisted external root must be permitted");}#[test]@@ -4942,7 +5011,7 @@ mod snapshot_path_security_tests {let data = tempfile::TempDir::new().unwrap();let snap = data.path().join("..");let repo = data.path().to_str().unwrap();- assert!(validate_snapshot_path(repo, "..", &snap, data.path(), &[]).is_err());+ assert!(validate_snapshot_path(repo, "..", &snap, data.path(), &[], None).is_err());}#[test]@@ -4955,18 +5024,149 @@ mod snapshot_path_security_tests {let data = tempfile::TempDir::new().unwrap();let repo = format!("{}/../../xerj-escape", data.path().display());let snap = std::path::Path::new(&repo).join("s1");- let err = validate_snapshot_path(&repo, "s1", &snap, data.path(), &[])+ let err = validate_snapshot_path(&repo, "s1", &snap, data.path(), &[], None).expect_err("a location containing `..` must be rejected");assert!(err.to_string().contains(".."), "unexpected error: {err}");// And it stays rejected even if the operator allowlists a broad root:// the `..` guard runs before containment, so a parent-ref cannot be// laundered through the allowlist either.- let err2 = validate_snapshot_path(&repo, "s1", &snap, data.path(), &["/".to_string()])- .expect_err("`..` must be rejected even with a permissive allowlist");+ let err2 =+ validate_snapshot_path(&repo, "s1", &snap, data.path(), &["/".to_string()], None)+ .expect_err("`..` must be rejected even with a permissive allowlist");assert!(err2.to_string().contains(".."), "unexpected error: {err2}");}}+#[cfg(test)]+mod hub_pack_restore_root_tests {+ //! `limits.hub_pack_restore`: the Corpus Hub pack cache is a restore-only+ //! snapshot root, trusted only in the shape `xerj hub pull` creates it.+ use super::{trusted_hub_pack_root, validate_snapshot_path};+ use std::path::{Path, PathBuf};++ /// `<tmp>/hub/packs/lz4/0774d055/snapshot/<snap>` as `xerj hub pull`+ /// extracts it, plus a data_dir elsewhere. Returns (tmp, data, packs, repo).+ fn layout() -> (tempfile::TempDir, PathBuf, PathBuf, PathBuf) {+ let tmp = tempfile::TempDir::new().unwrap();+ let root = tmp.path().canonicalize().unwrap();+ let data = root.join("data");+ let packs = root.join("hub").join("packs");+ let repo = packs.join("lz4").join("0774d055").join("snapshot");+ std::fs::create_dir_all(&data).unwrap();+ std::fs::create_dir_all(repo.join("snap-1")).unwrap();+ (tmp, data, packs, repo)+ }++ fn check(repo: &Path, data: &Path, hub: Option<&Path>) -> Result<PathBuf, String> {+ let repo_s = repo.to_str().unwrap();+ validate_snapshot_path(repo_s, "snap-1", &repo.join("snap-1"), data, &[], hub)+ .map_err(|e| e.to_string())+ }++ #[test]+ fn a_pulled_pack_is_restorable_from_the_hub_cache_without_an_allowlist() {+ let (_t, data, packs, repo) = layout();+ let snap = check(&repo, &data, Some(&packs)).expect("restore from the pack cache");+ assert!(snap.starts_with(&packs), "{}", snap.display());+ }++ #[test]+ fn without_the_hub_root_the_same_location_is_refused_as_before() {+ // Create passes `None`, and so does `hub_pack_restore = false`: the+ // pack cache is then exactly as foreign as any other path.+ let (_t, data, _packs, repo) = layout();+ let err = check(&repo, &data, None).expect_err("no hub root, no grant");+ assert!(err.contains("snapshot_repo_allowlist"), "{err}");+ }++ #[test]+ fn the_grant_covers_the_pack_cache_only_not_its_parent_or_a_sibling() {+ let (_t, data, packs, _repo) = layout();+ let hub = packs.parent().unwrap();+ // The hub root itself (catalogue, node data, config) is not a restore source.+ std::fs::create_dir_all(hub.join("node").join("snap-1")).unwrap();+ assert!(check(&hub.join("node"), &data, Some(&packs)).is_err());+ // A sibling sharing the name as a string prefix is not inside it.+ let sibling = hub.join("packs-evil");+ std::fs::create_dir_all(sibling.join("snap-1")).unwrap();+ assert!(check(&sibling, &data, Some(&packs)).is_err());+ // Nor is an unrelated absolute path.+ assert!(check(Path::new("/etc"), &data, Some(&packs)).is_err());+ }++ #[test]+ fn traversal_out_of_the_pack_cache_is_refused() {+ let (_t, data, packs, _repo) = layout();+ let outside = packs.parent().unwrap().parent().unwrap().join("outside");+ std::fs::create_dir_all(outside.join("snap-1")).unwrap();+ let escape = packs+ .join("lz4")+ .join("..")+ .join("..")+ .join("..")+ .join("outside");+ let err = check(&escape, &data, Some(&packs)).expect_err("`..` refused");+ assert!(err.contains(".."), "{err}");+ }++ #[cfg(unix)]+ #[test]+ fn a_symlink_inside_the_pack_cache_cannot_point_the_restore_elsewhere() {+ let (_t, data, packs, _repo) = layout();+ let outside = packs.parent().unwrap().parent().unwrap().join("outside");+ std::fs::create_dir_all(outside.join("snap-1")).unwrap();+ let link = packs.join("evil");+ std::os::unix::fs::symlink(&outside, &link).unwrap();+ let err = check(&link, &data, Some(&packs)).expect_err("escaping link refused");+ assert!(err.contains("snapshot_repo_allowlist"), "{err}");+ }++ #[cfg(unix)]+ #[test]+ fn a_pack_cache_that_is_itself_a_symlink_is_not_trusted() {+ // `~/.xerj/hub/packs -> /` would otherwise turn the grant into the+ // whole disk; `-> <elsewhere>` would move it somewhere nobody chose.+ let tmp = tempfile::TempDir::new().unwrap();+ let root = tmp.path().canonicalize().unwrap();+ let data = root.join("data");+ std::fs::create_dir_all(&data).unwrap();+ let hub = root.join("hub");+ std::fs::create_dir_all(&hub).unwrap();+ let packs = hub.join("packs");+ std::os::unix::fs::symlink("/", &packs).unwrap();+ assert_eq!(trusted_hub_pack_root(&packs), None);+ assert!(check(Path::new("/etc"), &data, Some(&packs)).is_err());++ let real = root.join("elsewhere");+ std::fs::create_dir_all(real.join("snap-1")).unwrap();+ std::fs::remove_file(&packs).unwrap();+ std::os::unix::fs::symlink(&real, &packs).unwrap();+ assert_eq!(trusted_hub_pack_root(&packs), None);+ assert!(check(&packs, &data, Some(&packs)).is_err());+ assert!(check(&real, &data, Some(&packs)).is_err());+ }++ #[test]+ fn a_missing_relative_or_non_directory_pack_cache_is_not_trusted() {+ let tmp = tempfile::TempDir::new().unwrap();+ let root = tmp.path().canonicalize().unwrap();+ assert_eq!(trusted_hub_pack_root(&root.join("absent")), None);+ assert_eq!(trusted_hub_pack_root(Path::new("relative/packs")), None);+ let file = root.join("packs");+ std::fs::write(&file, b"not a dir").unwrap();+ assert_eq!(trusted_hub_pack_root(&file), None);+ std::fs::remove_file(&file).unwrap();+ std::fs::create_dir_all(&file).unwrap();+ assert_eq!(trusted_hub_pack_root(&file), Some(file.clone()));+ }++ #[cfg(unix)]+ #[test]+ fn the_filesystem_root_is_never_a_pack_cache() {+ assert_eq!(trusted_hub_pack_root(Path::new("/")), None);+ }+}+#[cfg(test)]mod ism_policy_persistence_validation_tests {use crate::lifecycle::{LifecyclePolicy, LifecycleState};diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rsindex bb062c7cb..b05af4472 100644--- a/engine/crates/xerj-engine/tests/product_experience.rs+++ b/engine/crates/xerj-engine/tests/product_experience.rs@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {/// Settings in a default `Config`, measured by `count_settings`. Quoted in/// `xerj-common/src/config.rs`; keep the two in step.-const EXPECTED_SETTINGS: usize = 134;+const EXPECTED_SETTINGS: usize = 135;#[tokio::test]async fn journey_zero_config() {diff --git a/engine/crates/xerj-hub/src/cli.rs b/engine/crates/xerj-hub/src/cli.rsindex f0b1be9a4..518f48c68 100644--- a/engine/crates/xerj-hub/src/cli.rs+++ b/engine/crates/xerj-hub/src/cli.rs@@ -909,7 +909,12 @@ pub(crate) fn pull_with(),Some(&json!({ "indices": format!("{prefix}-*") })),)- .map_err(|e| PullError::new(E_INFRA, format!("restore snapshot: {e:#}")))?;+ .map_err(|e| {+ PullError::new(+ E_INFRA,+ restore_error_message(&format!("{e:#}"), url, &hub_home.join("packs")),+ )+ })?;}// Only now — with the full set provably present — stamp the ledger.@@ -935,6 +940,33 @@ pub(crate) fn pull_with()))}+/// The message for a failed restore. A server that refuses the pack's location+/// (`... not in limits.snapshot_repo_allowlist`) gets the fix spelled out: by+/// default a server restores from the pack cache of the user it runs as+/// (`limits.hub_pack_restore`), so this only happens when the server does not+/// share this cache — another user, a container, another machine — or has the+/// grant switched off. The exact config line names THIS client's cache.+fn restore_error_message(err: &str, url: &str, packs: &Path) -> String {+ if !err.contains("snapshot_repo_allowlist") {+ return format!("restore snapshot: {err}");+ }+ let dir = packs.canonicalize().unwrap_or_else(|_| packs.to_path_buf());+ let dir_toml = serde_json::to_string(&dir.to_string_lossy()).unwrap_or_default();+ format!(+ "restore snapshot: the XERJ server at {url} refused to read the pack from {dir}.\n\+ A server restores pulled packs from the hub pack cache of the user it runs as \+ (limits.hub_pack_restore, on by default); this server does not share yours — it runs \+ as another user, in a container or on another machine, or has hub_pack_restore = false.\n\+ Fix, either:\n \+ - add this line under [limits] in the server's config TOML and restart it (a container \+ must also mount the directory at the same path):\n \+ snapshot_repo_allowlist = [{dir_toml}]\n \+ - or let xerj start its own node: unset XERJ_URL (or point it at a free port).\n\+ server said: {err}",+ dir = dir.display()+ )+}+/// Put the verified, fully extracted `staging` directory at `dest` by renaming it —/// one step, so `dest` is either absent or complete. A pack already at `dest`/// (from an earlier pull of the same pin) is moved into `scratch` first and put@@ -1531,6 +1563,44 @@ fn now_rfc3339() -> String {mod tests {use super::*;+ #[test]+ fn a_refused_restore_names_the_exact_config_line_that_fixes_it() {+ let hub = tempfile::tempdir().unwrap();+ let packs = hub.path().join("packs");+ std::fs::create_dir_all(&packs).unwrap();+ let canon = packs.canonicalize().unwrap();+ let server = "invalid_mapping: snapshot repository location [/x] is outside data_dir \+ and not in limits.snapshot_repo_allowlist; refusing";+ let msg = restore_error_message(server, "http://127.0.0.1:9200", &packs);+ let line = format!(+ "snapshot_repo_allowlist = [{}]",+ serde_json::to_string(&canon.to_string_lossy()).unwrap()+ );+ assert!(msg.contains(&line), "{msg}");+ assert!(msg.contains("[limits]"), "{msg}");+ assert!(msg.contains("http://127.0.0.1:9200"), "{msg}");+ assert!(msg.contains("XERJ_URL"), "{msg}");+ assert!(+ msg.contains(server),+ "the server's own words are kept: {msg}"+ );+ // The line it prints is one the real config loader accepts, and it+ // allowlists exactly the cache.+ let cfg_path = hub.path().join("fix.toml");+ std::fs::write(&cfg_path, format!("[limits]\n{line}\n")).unwrap();+ let cfg = xerj_common::config::Config::load(&cfg_path).expect("the printed fix loads");+ assert_eq!(+ cfg.limits.snapshot_repo_allowlist,+ [canon.to_string_lossy().into_owned()]+ );+ }++ #[test]+ fn any_other_restore_failure_is_reported_as_it_was() {+ let msg = restore_error_message("boom", "http://h:1", Path::new("/p"));+ assert_eq!(msg, "restore snapshot: boom");+ }+#[test]fn pin8_takes_eight_lowercase_hex() {assert_eq!(diff --git a/engine/crates/xerj-hub/src/lockfile.rs b/engine/crates/xerj-hub/src/lockfile.rsindex 901698549..cbc1f2c0e 100644--- a/engine/crates/xerj-hub/src/lockfile.rs+++ b/engine/crates/xerj-hub/src/lockfile.rs@@ -238,7 +238,7 @@ fn question_asks_for_current(question: &str) -> bool {// xerj hub status// ----------------------------------------------------------------------------pub(crate) const PACKS_DIR: &str = "packs";+pub(crate) const PACKS_DIR: &str = xerj_common::hub::HUB_PACKS_DIR;pub(crate) const PACK_MANIFEST_FILE: &str = "pack.json";pub(crate) const LAST_USED_FILE: &str = "last_used";const NODE_PID_FILE: &str = "node.pid";@@ -262,12 +262,11 @@ pub fn default_hub_root() -> PathBuf {/// [`default_hub_root`] over an injectable environment, so the rule is testable/// without mutating the process-global one.pub(crate) fn hub_root_from(env: &dyn Fn(&str) -> Option<std::ffi::OsString>) -> PathBuf {- if let Some(h) = env("XERJ_HUB_HOME").filter(|h| !h.is_empty()) {- return PathBuf::from(h);- }- // `HOME`, else `USERPROFILE` — native Windows usually has no `HOME`.- let home = xerj_common::home::home_dir_from(env).unwrap_or_else(|| PathBuf::from("."));- home.join(".xerj").join("hub")+ // One rule with the server's hub-pack restore root (`xerj_common::hub`):+ // `XERJ_HUB_HOME`, else `HOME`/`USERPROFILE` + `.xerj/hub`. With no home at+ // all the CLI falls back to the current directory; the server allows nothing.+ xerj_common::hub::hub_root_from(env)+ .unwrap_or_else(|| PathBuf::from(".").join(".xerj").join("hub"))}/// One cached pack on disk (`<hub>/packs/<slug>/<pin8>/`).diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex 8c548bf27..a1368ea05 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -1,6 +1,6 @@# xerj Default Configuration# ============================================================-# This file sets 56 of XERJ's 134 user-facing settings to their+# This file sets 57 of XERJ's 135 user-facing settings to their# production-ready defaults, and describes the rest in comments.# Every value here IS the default, so copying this file changes# nothing until you edit it (a test enforces that). The complete@@ -470,7 +470,7 @@ timeout_ms = 5000# Range: 1 – 64.# neural_window_concurrency = 1-# ── Resource Limits (6 of 14 settings) ────────────────────────────────────────+# ── Resource Limits (7 of 15 settings) ────────────────────────────────────────[limits]@@ -513,6 +513,17 @@ max_actions_per_bulk = 50000# Example: snapshot_repo_allowlist = ["/mnt/backups"]snapshot_repo_allowlist = []+# Let a snapshot RESTORE (never a create) read from the Corpus Hub pack cache of+# the user this server runs as: $XERJ_HUB_HOME/packs, else ~/.xerj/hub/packs.+# That is where `xerj hub pull` / `xerj ask` extract a verified, signed pack+# before asking the server to restore it, so this is what lets a pull work+# against a server you started yourself. Honoured only while that directory is+# a real directory (not a symlink) owned by this server's user; symlinks inside+# it that point elsewhere are refused. false = only data_dir and the allowlist.+# A server running as another user, in a container or on another machine does+# not share your cache: add the directory to snapshot_repo_allowlist there.+hub_pack_restore = true+# Process-wide retained-payload budget for the seven immutable segment# hydration caches (MiB). 0 derives 20% of the effective cgroup/system memory# limit with no minimum floor. This is not an RSS limit: query-owned results,diff --git a/tools/hub-packs/consumer_e2e.sh b/tools/hub-packs/consumer_e2e.shindex 550c19855..1d4dd716c 100755--- a/tools/hub-packs/consumer_e2e.sh+++ b/tools/hub-packs/consumer_e2e.sh@@ -65,17 +65,16 @@ python3 -m http.server "$HUB_PORT" --bind 127.0.0.1 --directory "$WORK/site" \> "$WORK/http.log" 2>&1 &PIDS+=("$!")-# 2. A fresh node and an empty HOME. The node allows snapshot repositories only-# under the hub's pack cache, as the automatic node does (node.rs).+# 2. A fresh node and an empty HOME. The node is started the way a user starts+# one — no config file, no snapshot allowlist: a server restores from the pack+# cache of the user it runs as by default (limits.hub_pack_restore), and this+# gate is what proves `xerj hub pull` works against such a node.export HOME="$WORK/home"# XERJ_HUB_PUBKEY is deliberately NOT unset: CI never sets it, so the key compiled# into <xerj> governs there; a local rehearsal with a throwaway key sets it.unset XERJ_CODE_HOME XERJ_HUB_HOME XERJ_API_KEY-mkdir -p "$HOME/.xerj/hub/packs"-printf '[limits]\nsnapshot_repo_allowlist = ["%s"]\n' "$(realpath "$HOME/.xerj/hub/packs")" \- > "$WORK/node/config.toml""$BIN" --insecure --bind 127.0.0.1 --port "$NODE_PORT" --data-dir "$WORK/node/data" \- --config "$WORK/node/config.toml" > "$WORK/node/node.log" 2>&1 &+ > "$WORK/node/node.log" 2>&1 &PIDS+=("$!")export XERJ_URL="http://127.0.0.1:$NODE_PORT"export XERJ_HUB_URL="http://127.0.0.1:$HUB_PORT"