Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(hub): let `xerj hub pull` restore into a user-started node by default (#1734)

xerj-orga568f7cc5d35
diff --git a/docs/CORPUS_HUB_V2_IMPLEMENTATION.md b/docs/CORPUS_HUB_V2_IMPLEMENTATION.md
index 1cd25498c..0c8d5eb8c 100644
--- a/docs/CORPUS_HUB_V2_IMPLEMENTATION.md
+++ b/docs/CORPUS_HUB_V2_IMPLEMENTATION.md
@@ -425,6 +425,12 @@ health check fails, start `current_exe() --insecure --port <free port ≥ 9300>
detached, PID to `~/.xerj/hub/node.pid`, wait for green (max 60 s), remember the port
in `~/.xerj/hub/node.port`. Never start a node on a port that answers. `xerj hub status`
shows it; `xerj hub stop` stops only that PID.
+A node the user started themselves needs no config for `xerj hub pull` to restore
+into it: by default a server allows snapshot **restores** (never creates) from the
+pack cache of the user it runs as (`limits.hub_pack_restore`, `xerj_common::hub::hub_packs_dir`,
+refused if that directory is a symlink or owned by another user). A server that does
+not share the cache (other user, container, other machine) refuses, and the pull
+error prints the exact `snapshot_repo_allowlist` line to add.
**Tests** (`xerj-hub/tests/routing.rs`, `xerj-hub/tests/ask_flow.rs`):
diff --git a/docs/SNAPSHOT_AND_RESTORE.md b/docs/SNAPSHOT_AND_RESTORE.md
index 49c65f746..907a1b820 100644
--- a/docs/SNAPSHOT_AND_RESTORE.md
+++ b/docs/SNAPSHOT_AND_RESTORE.md
@@ -53,6 +53,19 @@ to empty, which means only `data_dir` is permitted. This is checked when a
snapshot is created or restored, not when the repository is registered, so a
bad location is accepted by `PUT /_snapshot/{repo}` and only fails later.
+One more root is allowed for **restore only**: the Corpus Hub pack cache of the
+user the server runs as (`$XERJ_HUB_HOME/packs`, else `~/.xerj/hub/packs`),
+which is where `xerj hub pull` and `xerj ask` extract a verified, signed pack
+before asking the server to restore it. It is controlled by
+`limits.hub_pack_restore` (default `true`; `false` turns it off). The grant is
+narrower than an allowlist entry: a snapshot can never be *created* there, the
+directory must be a real directory (not a symlink) owned by the server's user,
+and the repository location is still canonicalized, so a symlink inside the
+cache that points elsewhere is refused. A server running as another user, in a
+container or on another machine does not share the cache; there, add the
+directory to `limits.snapshot_repo_allowlist` (the `xerj hub pull` error prints
+the exact line).
+
If `settings.location` is absent the handler falls back to
`/tmp/xerj-snapshots` (`es_compat.rs:23201`). That fallback is outside the
default `data_dir` and will therefore be refused by the location check, so
diff --git a/engine/README.md b/engine/README.md
index 3dade6255..36fe1da49 100644
--- a/engine/README.md
+++ b/engine/README.md
@@ -53,7 +53,7 @@ HTTP wire payloads, single-node, tmpfs data dir, both freshly started
| Per-batch (5K) bulk p99 | **34.8 ms** | 80.3 ms | ES 2.3× |
| Data loss on restart | 0% | 0% | tied |
| GC pauses | 20–40 ms (JVM) | **0** (no GC) | **xerj** |
-| Config surface | 3,000+ knobs | **134**, all optional | **xerj** |
+| Config surface | 3,000+ knobs | **135**, all optional | **xerj** |
xerj wins on every operational and read-path metric. ES still wins
bulk-ingest throughput on warm Lucene `IndexWriter` (perf-backlog item
@@ -86,7 +86,7 @@ The resulting binary is at `target/release/xerj`.
## Configuration
See [`xerj.default.toml`](xerj.default.toml) for the commonly-tuned settings, and
-[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 134
+[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 135
— every one of them optional.
The minimal config to get started:
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 6331904ef..30d36b6b6 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -1,6 +1,6 @@
//! xerj configuration system.
//!
-//! Configuration is intentionally minimal: **134 settings** versus
+//! Configuration is intentionally minimal: **135 settings** versus
//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible
//! production-ready default. The format is TOML, loaded from a single file.
//!
@@ -108,7 +108,7 @@ pub struct Config {
pub hub: HubConfig,
}
-// 24 sub-configs, 134 leaf settings in total. Do not maintain that sum by hand
+// 24 sub-configs, 135 leaf settings in total. Do not maintain that sum by hand
// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a
// serialised `Config::default()` and fails if this comment and the module
// header stop matching. `Default` is derived: every field is a sub-config that
@@ -563,7 +563,7 @@ impl Config {
}
// ═════════════════════════════════════════════════════════════════════════════
-// Sub-configs (134 user-facing settings total; counted by
+// Sub-configs (135 user-facing settings total; counted by
// `journey_zero_config`, not by hand)
// ═════════════════════════════════════════════════════════════════════════════
@@ -1703,7 +1703,7 @@ pub const AUTO_PROCESS_MEMORY_MB: u64 = u64::MAX;
/// Resource limits.
///
-/// **14 settings.**
+/// **15 settings.**
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct LimitsConfig {
@@ -1733,6 +1733,22 @@ pub struct LimitsConfig {
/// path inside `data_dir` or inside one of these bases.
#[serde(default)]
pub snapshot_repo_allowlist: Vec<String>,
+ /// Let a snapshot **restore** (never a create) read from the Corpus Hub pack
+ /// cache of the user the server runs as — `$XERJ_HUB_HOME/packs`, else
+ /// `~/.xerj/hub/packs` (`xerj_common::hub::hub_packs_dir`) — without listing
+ /// it in `snapshot_repo_allowlist` (default: `true`).
+ ///
+ /// `xerj hub pull` / `xerj ask` download a signed pack, verify it, extract it
+ /// under that directory and ask the server to restore it. Before this, only
+ /// the node XERJ starts for itself allowlisted the directory, so a server the
+ /// user started by hand refused every pull. The grant is deliberately
+ /// narrower than an allowlist entry: restore only (a client cannot write a
+ /// snapshot into the user's home), honoured only while the directory is a
+ /// real directory — not a symlink — owned by the server's own user, and the
+ /// repo location is still canonicalized, so a symlink inside the cache that
+ /// points elsewhere is refused. Set `false` to allow `data_dir` and the
+ /// allowlist only.
+ pub hub_pack_restore: bool,
/// Maximum HTTP request body size in bytes (default: `100 * 1024 * 1024`,
/// i.e. 100 MiB). Caps NDJSON bulk payloads, large mget bodies, etc.
/// Raise this only if your client routinely sends bigger requests; the
@@ -1838,6 +1854,7 @@ impl Default for LimitsConfig {
max_fields_per_index: 500,
max_actions_per_bulk: 50_000,
snapshot_repo_allowlist: Vec::new(),
+ hub_pack_restore: true,
max_body_bytes: 100 * 1024 * 1024,
max_result_window: 10_000,
max_mget_docs: 10_000,
@@ -3702,7 +3719,7 @@ mod tests {
("vector", 6),
("logs", 2),
("embedding", 20),
- ("limits", 14),
+ ("limits", 15),
("indexing", 3),
("engine", 4),
("cluster", 5),
@@ -3757,7 +3774,7 @@ mod tests {
"the section table must sum to the whole config"
);
assert_eq!(
- total, 134,
+ total, 135,
"the total settings count changed. It is quoted in this module's \
header, in xerj-common/src/lib.rs, in engine/README.md, in \
xerj.default.toml and in EXPECTED_SETTINGS in \
diff --git a/engine/crates/xerj-common/src/hub.rs b/engine/crates/xerj-common/src/hub.rs
index f9b0959e3..89bd7342b 100644
--- a/engine/crates/xerj-common/src/hub.rs
+++ b/engine/crates/xerj-common/src/hub.rs
@@ -29,6 +29,33 @@ pub const PACK_FORMAT_VERSION: u32 = 1;
/// Where the catalogue and the hosted query API live.
pub const DEFAULT_HUB_URL: &str = "https://hub.xerj.org";
+/// The hub's local root: `XERJ_HUB_HOME` when set and non-empty, else
+/// `<home>/.xerj/hub` ([`crate::home::home_dir_from`]: `HOME`, else
+/// `USERPROFILE`). `None` when neither is set.
+///
+/// This is THE rule. The CLI (`xerj hub pull`, which extracts packs under
+/// `<root>/packs`) and the server (which, by default, lets a snapshot restore
+/// read from that same `<root>/packs` — `limits.hub_pack_restore`) both resolve
+/// it here, so a server started by the same user agrees with the CLI on where a
+/// pulled pack lives.
+pub fn hub_root_from(
+ env: &dyn Fn(&str) -> Option<std::ffi::OsString>,
+) -> Option<std::path::PathBuf> {
+ if let Some(h) = env("XERJ_HUB_HOME").filter(|h| !h.is_empty()) {
+ return Some(std::path::PathBuf::from(h));
+ }
+ crate::home::home_dir_from(env).map(|home| home.join(".xerj").join("hub"))
+}
+
+/// The directory `xerj hub pull` extracts verified packs into:
+/// [`hub_root_from`]`/packs`, over the process environment.
+pub fn hub_packs_dir() -> Option<std::path::PathBuf> {
+ hub_root_from(&|key| std::env::var_os(key)).map(|root| root.join(HUB_PACKS_DIR))
+}
+
+/// The pack-cache directory name under the hub root.
+pub const HUB_PACKS_DIR: &str = "packs";
+
/// Packs at or below this size are pulled automatically; larger ones are
/// answered through the hosted route until the user runs `xerj hub pull`.
pub const AUTO_PULL_MAX_BYTES: u64 = 50 * 1024 * 1024;
@@ -338,6 +365,38 @@ pub fn render(answer: &Answer) -> String {
#[cfg(test)]
mod tests {
+ fn hub_env<'a>(
+ vars: &'a [(&'a str, &'a str)],
+ ) -> impl Fn(&str) -> Option<std::ffi::OsString> + 'a {
+ move |key| {
+ vars.iter()
+ .find(|(k, _)| *k == key)
+ .map(|(_, v)| std::ffi::OsString::from(v))
+ }
+ }
+
+ #[test]
+ fn the_hub_root_rule_is_xerj_hub_home_then_home_then_nothing() {
+ use std::path::PathBuf;
+ assert_eq!(
+ hub_root_from(&hub_env(&[
+ ("XERJ_HUB_HOME", "/srv/hub"),
+ ("HOME", "/home/u")
+ ])),
+ Some(PathBuf::from("/srv/hub"))
+ );
+ assert_eq!(
+ hub_root_from(&hub_env(&[("XERJ_HUB_HOME", ""), ("HOME", "/home/u")])),
+ Some(PathBuf::from("/home/u/.xerj/hub"))
+ );
+ assert_eq!(
+ hub_root_from(&hub_env(&[("USERPROFILE", "/Users/u")])),
+ Some(PathBuf::from("/Users/u/.xerj/hub"))
+ );
+ // No home at all: the server grants nothing (never `./.xerj/hub`).
+ assert_eq!(hub_root_from(&hub_env(&[])), None);
+ }
+
use super::*;
use serde_json::json;
diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rs
index 7aa309d5a..da8b7d4ab 100644
--- a/engine/crates/xerj-common/src/lib.rs
+++ b/engine/crates/xerj-common/src/lib.rs
@@ -7,13 +7,13 @@
//! ## Design philosophy
//!
//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes
-//! **134 settings**, each meaningful and production-tested. Every default
+//! **135 settings**, each meaningful and production-tested. Every default
//! is chosen so that a fresh deployment with zero configuration changes performs
//! well for the majority of workloads.
//!
//! ## Modules
//!
-//! - [`config`] — TOML-based configuration (134 settings)
+//! - [`config`] — TOML-based configuration (135 settings)
//! - [`code_query`] — query shaping shared by `xerj search` and the
//! `xerj_search` MCP tool (multi-word glue-token stripping, #1364)
//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index 047b9f13e..af70af5d1 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -4418,12 +4418,15 @@ impl Engine {
) -> Result<Value> {
self.ensure_cluster_state_writable()?;
let snap_dir = std::path::Path::new(repo_path).join(name);
+ // No hub-pack root here: the pack cache is a restore-only source, so a
+ // client can never write a snapshot into the user's home directory.
let snap_dir = validate_snapshot_path(
repo_path,
name,
&snap_dir,
&self.data_dir,
&self.config.limits.snapshot_repo_allowlist,
+ None,
)?;
std::fs::create_dir_all(&snap_dir).map_err(EngineError::Io)?;
@@ -4517,12 +4520,22 @@ impl Engine {
) -> Result<Vec<String>> {
self.ensure_cluster_state_writable()?;
let snap_dir = std::path::Path::new(repo_path).join(name);
+ // `limits.hub_pack_restore`: the Corpus Hub pack cache of the user this
+ // server runs as is a restore source by default, so `xerj hub pull`
+ // works against a server the user started by hand (not only against the
+ // node `xerj ask` starts, which allowlists it in its own config).
+ let hub_packs = if self.config.limits.hub_pack_restore {
+ xerj_common::hub::hub_packs_dir()
+ } else {
+ None
+ };
let snap_dir = validate_snapshot_path(
repo_path,
name,
&snap_dir,
&self.data_dir,
&self.config.limits.snapshot_repo_allowlist,
+ hub_packs.as_deref(),
)?;
let manifest_path = snap_dir.join("manifest.json");
@@ -4740,6 +4753,7 @@ fn validate_snapshot_path(
snap_dir: &std::path::Path,
data_dir: &std::path::Path,
allowlist: &[String],
+ hub_packs: Option<&std::path::Path>,
) -> Result<std::path::PathBuf> {
if repo_path.is_empty() {
return Err(EngineError::Common(
@@ -4780,6 +4794,10 @@ fn validate_snapshot_path(
.iter()
.map(|b| base_canon(std::path::Path::new(b))),
);
+ // The hub pack cache (restore only — create passes `None`). Unlike an
+ // operator's allowlist entry it is implicit, so it is trusted only in the
+ // shape `xerj hub pull` makes it; see `trusted_hub_pack_root`.
+ allowed_bases.extend(hub_packs.and_then(trusted_hub_pack_root));
if !allowed_bases.iter().any(|base| repo_base.starts_with(base)) {
return Err(EngineError::Common(
xerj_common::XerjError::invalid_mapping(format!(
@@ -4833,6 +4851,50 @@ fn validate_snapshot_path(
}
}
+/// The canonical hub pack-cache root a restore may read from, or `None` when
+/// `packs` is not safe to trust implicitly.
+///
+/// `limits.snapshot_repo_allowlist` entries are canonicalized as given: an
+/// operator who allowlists a symlink means its target. The hub root is NOT an
+/// operator's choice — it is derived from the environment — so it is held to
+/// the shape `xerj hub pull` creates (`create_dir_all`, by the same user):
+///
+/// - it exists and is a real directory, not a symlink (`symlink_metadata`),
+/// so `~/.xerj/hub/packs -> /` cannot widen the grant to the whole disk;
+/// - canonicalizing it changes nothing but its parent's resolution (its own
+/// last component is not a link);
+/// - it is not a filesystem root;
+/// - on Unix it is owned by the server's effective user, so a shared or
+/// pre-created directory (a `HOME` under a world-writable dir) is not
+/// honoured.
+///
+/// What is inside it is still checked per request: the caller canonicalizes the
+/// repo location, so `packs/x -> /etc` resolves outside this root and is
+/// refused, and `..` components are rejected before this is consulted.
+fn trusted_hub_pack_root(packs: &std::path::Path) -> Option<std::path::PathBuf> {
+ if !packs.is_absolute() {
+ return None;
+ }
+ let meta = std::fs::symlink_metadata(packs).ok()?;
+ if meta.file_type().is_symlink() || !meta.is_dir() {
+ return None;
+ }
+ #[cfg(unix)]
+ {
+ use std::os::unix::fs::MetadataExt;
+ // SAFETY: geteuid has no preconditions and cannot fail.
+ if meta.uid() != unsafe { libc::geteuid() } {
+ return None;
+ }
+ }
+ let canon = packs.canonicalize().ok()?;
+ let parent = packs.parent()?.canonicalize().ok()?;
+ if canon != parent.join(packs.file_name()?) || canon.parent().is_none() {
+ return None;
+ }
+ Some(canon)
+}
+
/// Recursively copy all files from `src` to `dst`, recording relative paths in `files`.
fn copy_dir_recursive(
src: &std::path::Path,
@@ -4911,7 +4973,7 @@ mod snapshot_path_security_tests {
let outside = tempfile::TempDir::new().unwrap();
let repo = outside.path().to_str().unwrap();
let snap = std::path::Path::new(repo).join("s1");
- let err = validate_snapshot_path(repo, "s1", &snap, data.path(), &[])
+ let err = validate_snapshot_path(repo, "s1", &snap, data.path(), &[], None)
.expect_err("external repo location must be rejected");
assert!(err.to_string().contains("outside data_dir"), "{err}");
}
@@ -4923,7 +4985,7 @@ mod snapshot_path_security_tests {
std::fs::create_dir_all(&repo_dir).unwrap();
let repo = repo_dir.to_str().unwrap();
let snap = repo_dir.join("s1");
- validate_snapshot_path(repo, "s1", &snap, data.path(), &[])
+ validate_snapshot_path(repo, "s1", &snap, data.path(), &[], None)
.expect("a repo under data_dir must be allowed");
}
@@ -4933,8 +4995,15 @@ mod snapshot_path_security_tests {
let ext = tempfile::TempDir::new().unwrap();
let repo = ext.path().to_str().unwrap().to_string();
let snap = std::path::Path::new(&repo).join("s1");
- validate_snapshot_path(&repo, "s1", &snap, data.path(), std::slice::from_ref(&repo))
- .expect("an allowlisted external root must be permitted");
+ validate_snapshot_path(
+ &repo,
+ "s1",
+ &snap,
+ data.path(),
+ std::slice::from_ref(&repo),
+ None,
+ )
+ .expect("an allowlisted external root must be permitted");
}
#[test]
@@ -4942,7 +5011,7 @@ mod snapshot_path_security_tests {
let data = tempfile::TempDir::new().unwrap();
let snap = data.path().join("..");
let repo = data.path().to_str().unwrap();
- assert!(validate_snapshot_path(repo, "..", &snap, data.path(), &[]).is_err());
+ assert!(validate_snapshot_path(repo, "..", &snap, data.path(), &[], None).is_err());
}
#[test]
@@ -4955,18 +5024,149 @@ mod snapshot_path_security_tests {
let data = tempfile::TempDir::new().unwrap();
let repo = format!("{}/../../xerj-escape", data.path().display());
let snap = std::path::Path::new(&repo).join("s1");
- let err = validate_snapshot_path(&repo, "s1", &snap, data.path(), &[])
+ let err = validate_snapshot_path(&repo, "s1", &snap, data.path(), &[], None)
.expect_err("a location containing `..` must be rejected");
assert!(err.to_string().contains(".."), "unexpected error: {err}");
// And it stays rejected even if the operator allowlists a broad root:
// the `..` guard runs before containment, so a parent-ref cannot be
// laundered through the allowlist either.
- let err2 = validate_snapshot_path(&repo, "s1", &snap, data.path(), &["/".to_string()])
- .expect_err("`..` must be rejected even with a permissive allowlist");
+ let err2 =
+ validate_snapshot_path(&repo, "s1", &snap, data.path(), &["/".to_string()], None)
+ .expect_err("`..` must be rejected even with a permissive allowlist");
assert!(err2.to_string().contains(".."), "unexpected error: {err2}");
}
}
+#[cfg(test)]
+mod hub_pack_restore_root_tests {
+ //! `limits.hub_pack_restore`: the Corpus Hub pack cache is a restore-only
+ //! snapshot root, trusted only in the shape `xerj hub pull` creates it.
+ use super::{trusted_hub_pack_root, validate_snapshot_path};
+ use std::path::{Path, PathBuf};
+
+ /// `<tmp>/hub/packs/lz4/0774d055/snapshot/<snap>` as `xerj hub pull`
+ /// extracts it, plus a data_dir elsewhere. Returns (tmp, data, packs, repo).
+ fn layout() -> (tempfile::TempDir, PathBuf, PathBuf, PathBuf) {
+ let tmp = tempfile::TempDir::new().unwrap();
+ let root = tmp.path().canonicalize().unwrap();
+ let data = root.join("data");
+ let packs = root.join("hub").join("packs");
+ let repo = packs.join("lz4").join("0774d055").join("snapshot");
+ std::fs::create_dir_all(&data).unwrap();
+ std::fs::create_dir_all(repo.join("snap-1")).unwrap();
+ (tmp, data, packs, repo)
+ }
+
+ fn check(repo: &Path, data: &Path, hub: Option<&Path>) -> Result<PathBuf, String> {
+ let repo_s = repo.to_str().unwrap();
+ validate_snapshot_path(repo_s, "snap-1", &repo.join("snap-1"), data, &[], hub)
+ .map_err(|e| e.to_string())
+ }
+
+ #[test]
+ fn a_pulled_pack_is_restorable_from_the_hub_cache_without_an_allowlist() {
+ let (_t, data, packs, repo) = layout();
+ let snap = check(&repo, &data, Some(&packs)).expect("restore from the pack cache");
+ assert!(snap.starts_with(&packs), "{}", snap.display());
+ }
+
+ #[test]
+ fn without_the_hub_root_the_same_location_is_refused_as_before() {
+ // Create passes `None`, and so does `hub_pack_restore = false`: the
+ // pack cache is then exactly as foreign as any other path.
+ let (_t, data, _packs, repo) = layout();
+ let err = check(&repo, &data, None).expect_err("no hub root, no grant");
+ assert!(err.contains("snapshot_repo_allowlist"), "{err}");
+ }
+
+ #[test]
+ fn the_grant_covers_the_pack_cache_only_not_its_parent_or_a_sibling() {
+ let (_t, data, packs, _repo) = layout();
+ let hub = packs.parent().unwrap();
+ // The hub root itself (catalogue, node data, config) is not a restore source.
+ std::fs::create_dir_all(hub.join("node").join("snap-1")).unwrap();
+ assert!(check(&hub.join("node"), &data, Some(&packs)).is_err());
+ // A sibling sharing the name as a string prefix is not inside it.
+ let sibling = hub.join("packs-evil");
+ std::fs::create_dir_all(sibling.join("snap-1")).unwrap();
+ assert!(check(&sibling, &data, Some(&packs)).is_err());
+ // Nor is an unrelated absolute path.
+ assert!(check(Path::new("/etc"), &data, Some(&packs)).is_err());
+ }
+
+ #[test]
+ fn traversal_out_of_the_pack_cache_is_refused() {
+ let (_t, data, packs, _repo) = layout();
+ let outside = packs.parent().unwrap().parent().unwrap().join("outside");
+ std::fs::create_dir_all(outside.join("snap-1")).unwrap();
+ let escape = packs
+ .join("lz4")
+ .join("..")
+ .join("..")
+ .join("..")
+ .join("outside");
+ let err = check(&escape, &data, Some(&packs)).expect_err("`..` refused");
+ assert!(err.contains(".."), "{err}");
+ }
+
+ #[cfg(unix)]
+ #[test]
+ fn a_symlink_inside_the_pack_cache_cannot_point_the_restore_elsewhere() {
+ let (_t, data, packs, _repo) = layout();
+ let outside = packs.parent().unwrap().parent().unwrap().join("outside");
+ std::fs::create_dir_all(outside.join("snap-1")).unwrap();
+ let link = packs.join("evil");
+ std::os::unix::fs::symlink(&outside, &link).unwrap();
+ let err = check(&link, &data, Some(&packs)).expect_err("escaping link refused");
+ assert!(err.contains("snapshot_repo_allowlist"), "{err}");
+ }
+
+ #[cfg(unix)]
+ #[test]
+ fn a_pack_cache_that_is_itself_a_symlink_is_not_trusted() {
+ // `~/.xerj/hub/packs -> /` would otherwise turn the grant into the
+ // whole disk; `-> <elsewhere>` would move it somewhere nobody chose.
+ let tmp = tempfile::TempDir::new().unwrap();
+ let root = tmp.path().canonicalize().unwrap();
+ let data = root.join("data");
+ std::fs::create_dir_all(&data).unwrap();
+ let hub = root.join("hub");
+ std::fs::create_dir_all(&hub).unwrap();
+ let packs = hub.join("packs");
+ std::os::unix::fs::symlink("/", &packs).unwrap();
+ assert_eq!(trusted_hub_pack_root(&packs), None);
+ assert!(check(Path::new("/etc"), &data, Some(&packs)).is_err());
+
+ let real = root.join("elsewhere");
+ std::fs::create_dir_all(real.join("snap-1")).unwrap();
+ std::fs::remove_file(&packs).unwrap();
+ std::os::unix::fs::symlink(&real, &packs).unwrap();
+ assert_eq!(trusted_hub_pack_root(&packs), None);
+ assert!(check(&packs, &data, Some(&packs)).is_err());
+ assert!(check(&real, &data, Some(&packs)).is_err());
+ }
+
+ #[test]
+ fn a_missing_relative_or_non_directory_pack_cache_is_not_trusted() {
+ let tmp = tempfile::TempDir::new().unwrap();
+ let root = tmp.path().canonicalize().unwrap();
+ assert_eq!(trusted_hub_pack_root(&root.join("absent")), None);
+ assert_eq!(trusted_hub_pack_root(Path::new("relative/packs")), None);
+ let file = root.join("packs");
+ std::fs::write(&file, b"not a dir").unwrap();
+ assert_eq!(trusted_hub_pack_root(&file), None);
+ std::fs::remove_file(&file).unwrap();
+ std::fs::create_dir_all(&file).unwrap();
+ assert_eq!(trusted_hub_pack_root(&file), Some(file.clone()));
+ }
+
+ #[cfg(unix)]
+ #[test]
+ fn the_filesystem_root_is_never_a_pack_cache() {
+ assert_eq!(trusted_hub_pack_root(Path::new("/")), None);
+ }
+}
+
#[cfg(test)]
mod ism_policy_persistence_validation_tests {
use crate::lifecycle::{LifecyclePolicy, LifecycleState};
diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rs
index bb062c7cb..b05af4472 100644
--- a/engine/crates/xerj-engine/tests/product_experience.rs
+++ b/engine/crates/xerj-engine/tests/product_experience.rs
@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {
/// Settings in a default `Config`, measured by `count_settings`. Quoted in
/// `xerj-common/src/config.rs`; keep the two in step.
-const EXPECTED_SETTINGS: usize = 134;
+const EXPECTED_SETTINGS: usize = 135;
#[tokio::test]
async fn journey_zero_config() {
diff --git a/engine/crates/xerj-hub/src/cli.rs b/engine/crates/xerj-hub/src/cli.rs
index f0b1be9a4..518f48c68 100644
--- a/engine/crates/xerj-hub/src/cli.rs
+++ b/engine/crates/xerj-hub/src/cli.rs
@@ -909,7 +909,12 @@ pub(crate) fn pull_with(
),
Some(&json!({ "indices": format!("{prefix}-*") })),
)
- .map_err(|e| PullError::new(E_INFRA, format!("restore snapshot: {e:#}")))?;
+ .map_err(|e| {
+ PullError::new(
+ E_INFRA,
+ restore_error_message(&format!("{e:#}"), url, &hub_home.join("packs")),
+ )
+ })?;
}
// Only now — with the full set provably present — stamp the ledger.
@@ -935,6 +940,33 @@ pub(crate) fn pull_with(
)))
}
+/// The message for a failed restore. A server that refuses the pack's location
+/// (`... not in limits.snapshot_repo_allowlist`) gets the fix spelled out: by
+/// default a server restores from the pack cache of the user it runs as
+/// (`limits.hub_pack_restore`), so this only happens when the server does not
+/// share this cache — another user, a container, another machine — or has the
+/// grant switched off. The exact config line names THIS client's cache.
+fn restore_error_message(err: &str, url: &str, packs: &Path) -> String {
+ if !err.contains("snapshot_repo_allowlist") {
+ return format!("restore snapshot: {err}");
+ }
+ let dir = packs.canonicalize().unwrap_or_else(|_| packs.to_path_buf());
+ let dir_toml = serde_json::to_string(&dir.to_string_lossy()).unwrap_or_default();
+ format!(
+ "restore snapshot: the XERJ server at {url} refused to read the pack from {dir}.\n\
+ A server restores pulled packs from the hub pack cache of the user it runs as \
+ (limits.hub_pack_restore, on by default); this server does not share yours — it runs \
+ as another user, in a container or on another machine, or has hub_pack_restore = false.\n\
+ Fix, either:\n \
+ - add this line under [limits] in the server's config TOML and restart it (a container \
+ must also mount the directory at the same path):\n \
+ snapshot_repo_allowlist = [{dir_toml}]\n \
+ - or let xerj start its own node: unset XERJ_URL (or point it at a free port).\n\
+ server said: {err}",
+ dir = dir.display()
+ )
+}
+
/// Put the verified, fully extracted `staging` directory at `dest` by renaming it —
/// one step, so `dest` is either absent or complete. A pack already at `dest`
/// (from an earlier pull of the same pin) is moved into `scratch` first and put
@@ -1531,6 +1563,44 @@ fn now_rfc3339() -> String {
mod tests {
use super::*;
+ #[test]
+ fn a_refused_restore_names_the_exact_config_line_that_fixes_it() {
+ let hub = tempfile::tempdir().unwrap();
+ let packs = hub.path().join("packs");
+ std::fs::create_dir_all(&packs).unwrap();
+ let canon = packs.canonicalize().unwrap();
+ let server = "invalid_mapping: snapshot repository location [/x] is outside data_dir \
+ and not in limits.snapshot_repo_allowlist; refusing";
+ let msg = restore_error_message(server, "http://127.0.0.1:9200", &packs);
+ let line = format!(
+ "snapshot_repo_allowlist = [{}]",
+ serde_json::to_string(&canon.to_string_lossy()).unwrap()
+ );
+ assert!(msg.contains(&line), "{msg}");
+ assert!(msg.contains("[limits]"), "{msg}");
+ assert!(msg.contains("http://127.0.0.1:9200"), "{msg}");
+ assert!(msg.contains("XERJ_URL"), "{msg}");
+ assert!(
+ msg.contains(server),
+ "the server's own words are kept: {msg}"
+ );
+ // The line it prints is one the real config loader accepts, and it
+ // allowlists exactly the cache.
+ let cfg_path = hub.path().join("fix.toml");
+ std::fs::write(&cfg_path, format!("[limits]\n{line}\n")).unwrap();
+ let cfg = xerj_common::config::Config::load(&cfg_path).expect("the printed fix loads");
+ assert_eq!(
+ cfg.limits.snapshot_repo_allowlist,
+ [canon.to_string_lossy().into_owned()]
+ );
+ }
+
+ #[test]
+ fn any_other_restore_failure_is_reported_as_it_was() {
+ let msg = restore_error_message("boom", "http://h:1", Path::new("/p"));
+ assert_eq!(msg, "restore snapshot: boom");
+ }
+
#[test]
fn pin8_takes_eight_lowercase_hex() {
assert_eq!(
diff --git a/engine/crates/xerj-hub/src/lockfile.rs b/engine/crates/xerj-hub/src/lockfile.rs
index 901698549..cbc1f2c0e 100644
--- a/engine/crates/xerj-hub/src/lockfile.rs
+++ b/engine/crates/xerj-hub/src/lockfile.rs
@@ -238,7 +238,7 @@ fn question_asks_for_current(question: &str) -> bool {
// xerj hub status
// ---------------------------------------------------------------------------
-pub(crate) const PACKS_DIR: &str = "packs";
+pub(crate) const PACKS_DIR: &str = xerj_common::hub::HUB_PACKS_DIR;
pub(crate) const PACK_MANIFEST_FILE: &str = "pack.json";
pub(crate) const LAST_USED_FILE: &str = "last_used";
const NODE_PID_FILE: &str = "node.pid";
@@ -262,12 +262,11 @@ pub fn default_hub_root() -> PathBuf {
/// [`default_hub_root`] over an injectable environment, so the rule is testable
/// without mutating the process-global one.
pub(crate) fn hub_root_from(env: &dyn Fn(&str) -> Option<std::ffi::OsString>) -> PathBuf {
- if let Some(h) = env("XERJ_HUB_HOME").filter(|h| !h.is_empty()) {
- return PathBuf::from(h);
- }
- // `HOME`, else `USERPROFILE` — native Windows usually has no `HOME`.
- let home = xerj_common::home::home_dir_from(env).unwrap_or_else(|| PathBuf::from("."));
- home.join(".xerj").join("hub")
+ // One rule with the server's hub-pack restore root (`xerj_common::hub`):
+ // `XERJ_HUB_HOME`, else `HOME`/`USERPROFILE` + `.xerj/hub`. With no home at
+ // all the CLI falls back to the current directory; the server allows nothing.
+ xerj_common::hub::hub_root_from(env)
+ .unwrap_or_else(|| PathBuf::from(".").join(".xerj").join("hub"))
}
/// One cached pack on disk (`<hub>/packs/<slug>/<pin8>/`).
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index 8c548bf27..a1368ea05 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -1,6 +1,6 @@
# xerj Default Configuration
# ============================================================
-# This file sets 56 of XERJ's 134 user-facing settings to their
+# This file sets 57 of XERJ's 135 user-facing settings to their
# production-ready defaults, and describes the rest in comments.
# Every value here IS the default, so copying this file changes
# nothing until you edit it (a test enforces that). The complete
@@ -470,7 +470,7 @@ timeout_ms = 5000
# Range: 1 – 64.
# neural_window_concurrency = 1
-# ── Resource Limits (6 of 14 settings) ────────────────────────────────────────
+# ── Resource Limits (7 of 15 settings) ────────────────────────────────────────
[limits]
@@ -513,6 +513,17 @@ max_actions_per_bulk = 50000
# Example: snapshot_repo_allowlist = ["/mnt/backups"]
snapshot_repo_allowlist = []
+# Let a snapshot RESTORE (never a create) read from the Corpus Hub pack cache of
+# the user this server runs as: $XERJ_HUB_HOME/packs, else ~/.xerj/hub/packs.
+# That is where `xerj hub pull` / `xerj ask` extract a verified, signed pack
+# before asking the server to restore it, so this is what lets a pull work
+# against a server you started yourself. Honoured only while that directory is
+# a real directory (not a symlink) owned by this server's user; symlinks inside
+# it that point elsewhere are refused. false = only data_dir and the allowlist.
+# A server running as another user, in a container or on another machine does
+# not share your cache: add the directory to snapshot_repo_allowlist there.
+hub_pack_restore = true
+
# Process-wide retained-payload budget for the seven immutable segment
# hydration caches (MiB). 0 derives 20% of the effective cgroup/system memory
# limit with no minimum floor. This is not an RSS limit: query-owned results,
diff --git a/tools/hub-packs/consumer_e2e.sh b/tools/hub-packs/consumer_e2e.sh
index 550c19855..1d4dd716c 100755
--- a/tools/hub-packs/consumer_e2e.sh
+++ b/tools/hub-packs/consumer_e2e.sh
@@ -65,17 +65,16 @@ python3 -m http.server "$HUB_PORT" --bind 127.0.0.1 --directory "$WORK/site" \
> "$WORK/http.log" 2>&1 &
PIDS+=("$!")
-# 2. A fresh node and an empty HOME. The node allows snapshot repositories only
-# under the hub's pack cache, as the automatic node does (node.rs).
+# 2. A fresh node and an empty HOME. The node is started the way a user starts
+# one — no config file, no snapshot allowlist: a server restores from the pack
+# cache of the user it runs as by default (limits.hub_pack_restore), and this
+# gate is what proves `xerj hub pull` works against such a node.
export HOME="$WORK/home"
# XERJ_HUB_PUBKEY is deliberately NOT unset: CI never sets it, so the key compiled
# into <xerj> governs there; a local rehearsal with a throwaway key sets it.
unset XERJ_CODE_HOME XERJ_HUB_HOME XERJ_API_KEY
-mkdir -p "$HOME/.xerj/hub/packs"
-printf '[limits]\nsnapshot_repo_allowlist = ["%s"]\n' "$(realpath "$HOME/.xerj/hub/packs")" \
- > "$WORK/node/config.toml"
"$BIN" --insecure --bind 127.0.0.1 --port "$NODE_PORT" --data-dir "$WORK/node/data" \
- --config "$WORK/node/config.toml" > "$WORK/node/node.log" 2>&1 &
+ > "$WORK/node/node.log" 2>&1 &
PIDS+=("$!")
export XERJ_URL="http://127.0.0.1:$NODE_PORT"
export XERJ_HUB_URL="http://127.0.0.1:$HUB_PORT"