AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(hub): xerj ask / xerj "<question>" answer flow + automatic local node (plan §5)
xerj-orgf955cfcc25be
diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex cf32b18e5..0e68f1629 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -7652,6 +7652,7 @@ dependencies = ["axum","base64 0.23.1","chrono",+ "libc","reqwest","serde","serde_json",diff --git a/engine/crates/xerj-hub/Cargo.toml b/engine/crates/xerj-hub/Cargo.tomlindex 955a0c4bb..209ab8fc3 100644--- a/engine/crates/xerj-hub/Cargo.toml+++ b/engine/crates/xerj-hub/Cargo.toml@@ -20,6 +20,10 @@ toml.workspace = truesha2 = "0.10"tar = "0.4"+# `xerj hub stop` signals exactly the PID it recorded (plan §5, §11).+[target.'cfg(unix)'.dependencies]+libc = "0.2"+[dev-dependencies]tempfile.workspace = true# The pack roundtrip test boots a real in-process node (ephemeral TCP port) sodiff --git a/engine/crates/xerj-hub/src/ask.rs b/engine/crates/xerj-hub/src/ask.rsnew file mode 100644index 000000000..cfe3ec407--- /dev/null+++ b/engine/crates/xerj-hub/src/ask.rs@@ -0,0 +1,2524 @@+//! Corpus Hub v2 — `xerj ask "<question>"`: route a question to a hub corpus and+//! answer it in one step (`docs/CORPUS_HUB_V2_IMPLEMENTATION.md` §5, "Answer+//! flow").+//!+//! The flow, in the plan's own numbering:+//!+//! 1. **Mode.** `--def SYM`, a question that is one identifier, or one matching+//! `what is the (exact )?value of (\w+)` is a [`AnswerMode::Def`] question;+//! anything else is [`AnswerMode::How`]; `--files` forces+//! [`AnswerMode::Files`] ([`detect_mode`]).+//! 2. **Route** ([`crate::router::route`]). `--corpus` skips routing.+//! Ambiguous → the candidates are printed and the exit code is **4**;+//! nothing credible → exit **1** with the llms.txt pointer.+//! 3. **Cached pack** → answer locally ([`AnsweredFrom::LocalPack`]); the+//! network is not touched.+//! 4. **No pack, hosted allowed** → answer through the hosted hub now, then (unless+//! `--no-fetch`) start a detached `xerj hub pull` for next time — subject to the+//! pull policy ([`pull_policy`]).+//! 5. **No pack, no hosted** → `--offline` (or `--no-fetch`) exits 1 naming the+//! pack to pull; otherwise pull synchronously (same policy) and answer+//! locally.+//! 6. **Disk budget** — before any pull, least-recently-used packs are evicted+//! until the pack cache fits ([`evict_to_budget`], called from `hub pull`).+//! 7. The §1 renderer prints the [`Answer`]; `--json` prints the struct.+//!+//! # Seams+//!+//! [`ask`] holds only the decisions. Everything with side effects sits behind a+//! trait so each decision is testable without a node or a network:+//! [`Backend`] (local pack, node, pull) and [`HostedHub`] (the hosted node,+//! plan §6). The CLI wires [`RealBackend`]; no [`HostedHub`] implementation ships+//! with this module — the hosted endpoints are plan step 6, and until a caller+//! passes one in, "hosted" is simply unavailable and the flow falls back to the+//! local pack / synchronous pull exactly as if the switch were off.+//!+//! # Not here+//!+//! Plan §8 is not wired into this flow: no `xerj.lock` is written, `--locked` does+//! not exist, and the freshness warning is not added (`lockfile.rs` has the+//! pieces; `ask` is where they plug in). `xerj hub sync | update | rm` do not+//! exist. The hosted client is plan step 6/7.+//!+//! # Exit codes+//!+//! | code | meaning |+//! |------|---------|+//! | 0 | answered, at least one hit |+//! | 1 | no match; no corpus; offline/no-fetch with no pack; pull or node failure |+//! | 2 | usage error; unknown `--corpus`; a pull rejected the slug |+//! | 3 | a pulled pack failed verification (sha256 / signature / format) |+//! | 4 | ambiguous — re-run with `--corpus` (the code `xerj autoindex` uses for "needs a decision") |++use std::cell::RefCell;+use std::collections::{HashMap, HashSet};+use std::fs;+use std::path::{Path, PathBuf};+use std::rc::Rc;+use std::time::SystemTime;++use anyhow::Result;+use serde_json::Value;+use xerj_common::hub::{+ render, Answer, AnswerMode, AnsweredFrom, Catalogue, CatalogueEntry, Hit, LicenceClass,+ PackManifest, AUTO_PULL_MAX_BYTES, DEFAULT_HUB_URL,+};+use xerj_common::xccode::{self, licence, passage, state, CodeParams, XcHttp};++use crate::catalogue::load_catalogue;+use crate::node::{self, HubNode, NodeLauncher, ResolvedNode};+use crate::router::{route, HostedSearch, Routing};+use crate::{cli, def, lockfile};++/// Answered with at least one hit.+pub const EXIT_OK: i32 = 0;+/// No match / no corpus / offline with no pack / a pull or node failure.+pub const EXIT_NO_ANSWER: i32 = 1;+/// Usage error, or an unknown corpus slug.+pub const EXIT_USAGE: i32 = 2;+/// A pulled pack failed verification.+pub const EXIT_UNVERIFIED: i32 = 3;+/// More than one corpus matches: the caller must choose (`--corpus`).+pub const EXIT_NEEDS_DECISION: i32 = 4;++/// Printed (stderr) when the router finds nothing credible.+pub const NO_CORPUS_MESSAGE: &str = "no hub corpus matches; see xerj.org/llms.txt";+/// Printed once (stderr) the first time `--hosted auto` resolves to off inside a+/// git work tree.+pub const HOSTED_OFF_NOTICE: &str =+ "hosted answers are off in git repositories; xerj hub config hosted on";+/// Printed (stderr) when `--hosted on` is asked for but no hosted client exists.+const NO_HOSTED_CLIENT: &str =+ "hosted answers were requested but this build has no hosted client; answering from a local pack";++const DEFAULT_K: usize = def::DEFAULT_K;+/// Characters of a passage a How answer keeps (the `xerj code` default).+const PASSAGE_WIDTH: usize = 800;+/// Distinct files a Files answer lists at most.+const FILES_K: usize = 10;++// ── request / outcome types ──────────────────────────────────────────────────++/// What kind of question the caller says this is (`--def` / `--how` /+/// `--files`); [`ModeRequest::Auto`] lets [`detect_mode`] decide.+#[derive(Debug, Clone, PartialEq, Eq)]+pub enum ModeRequest {+ Auto,+ Def(String),+ How,+ Files,+}++/// `--hosted on|off|auto`.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum HostedSwitch {+ Auto,+ On,+ Off,+}++/// Everything `xerj ask` takes from the command line.+#[derive(Debug, Clone)]+pub struct AskArgs {+ pub question: String,+ pub corpus: Option<String>,+ pub mode: ModeRequest,+ pub offline: bool,+ pub no_fetch: bool,+ pub hosted: HostedSwitch,+ pub max_tokens: Option<usize>,+ pub json: bool,+ pub url: Option<String>,+}++impl AskArgs {+ /// Defaults for everything but the question.+ pub fn new(question: impl Into<String>) -> Self {+ AskArgs {+ question: question.into(),+ corpus: None,+ mode: ModeRequest::Auto,+ offline: false,+ no_fetch: false,+ hosted: HostedSwitch::Auto,+ max_tokens: None,+ json: false,+ url: None,+ }+ }+}++/// What the flow decided: an exit code, the text for stdout, notices for stderr+/// and — when there was one — the [`Answer`] itself (for a caller that is not+/// the CLI, e.g. the MCP tool).+#[derive(Debug, Default, Clone)]+pub struct AskOutcome {+ pub exit: i32,+ /// The §1 rendering, the `--json` form, or the Ambiguous candidate lines.+ /// No trailing newline.+ pub stdout: String,+ /// Notices, one per line, in the order they arose.+ pub stderr: Vec<String>,+ pub answer: Option<Answer>,+}++impl AskOutcome {+ fn fail(exit: i32, message: impl Into<String>) -> Self {+ AskOutcome {+ exit,+ stderr: vec![message.into()],+ ..Default::default()+ }+ }+}++/// One question to one corpus, as the backends see it.+#[derive(Debug, Clone)]+pub struct Request<'a> {+ pub slug: &'a str,+ pub question: &'a str,+ pub mode: AnswerMode,+ /// Set exactly when `mode` is [`AnswerMode::Def`].+ pub symbol: Option<&'a str>,+ pub max_tokens: Option<usize>,+}++// ── seams ────────────────────────────────────────────────────────────────────++/// The hosted hub (plan §6), behind a trait so [`ask`] is network-free in tests+/// and so a build with no hosted client degrades cleanly. It extends+/// [`HostedSearch`], which the router uses for federated corpus search.+pub trait HostedHub: HostedSearch {+ /// Answer `req` through the hosted node (`/_hub/v1/def/…` or+ /// `/_hub/v1/code/…`).+ fn answer(&self, req: &Request<'_>) -> Result<Answer>;+}++/// Lets a `&dyn HostedHub` be handed to the router as a `&dyn HostedSearch`+/// without relying on trait-object upcasting.+struct SearchOnly<'a>(&'a dyn HostedHub);++impl HostedSearch for SearchOnly<'_> {+ fn search(&self, question: &str) -> Result<Vec<(String, f32)>> {+ self.0.search(question)+ }+}++/// A pull that did not complete, with the exit code it maps to.+#[derive(Debug, Clone)]+pub struct PullFailure {+ pub code: i32,+ pub message: String,+}++/// What [`Backend::pull_in_background`] did.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum BackgroundPull {+ Started {+ pid: u32,+ },+ /// A pull of this corpus was already running; none was started.+ AlreadyRunning {+ pid: u32,+ },+}++/// The side-effecting half of the flow: the local pack, the node it lives on,+/// and pulling. [`RealBackend`] is the production implementation.+pub trait Backend {+ /// Is a pack for `slug` on this machine AND fully loaded on the node queries+ /// run against? `Ok(false)` is the ordinary "not cached"; `Err` is a node+ /// that could not be reached or started. A backend must not start a node when+ /// no pack exists locally.+ fn local_pack_ready(&self, slug: &str) -> Result<bool, String>;+ /// Answer from the local pack. Called after [`Backend::local_pack_ready`]+ /// said yes, or after a successful pull.+ fn answer_local(&self, req: &Request<'_>) -> Result<Answer, String>;+ /// Download, verify and restore the pack, waiting for it to finish.+ fn pull_now(&self, entry: &CatalogueEntry) -> Result<(), PullFailure>;+ /// Start a detached `xerj hub pull` and return at once.+ fn pull_in_background(&self, slug: &str) -> Result<BackgroundPull, String>;+}++/// Everything [`ask`] consults.+pub struct AskDeps<'a> {+ pub hub_home: &'a Path,+ pub cwd: &'a Path,+ pub catalogue: &'a Catalogue,+ /// `None` = no hosted client is configured.+ pub hosted: Option<&'a dyn HostedHub>,+ pub backend: &'a dyn Backend,+}++// ── step 1: mode ─────────────────────────────────────────────────────────────++/// Decide the answer mode and the symbol (for `Def`). `--def` / `--how` /+/// `--files` win; otherwise a question that is one identifier, or that matches+/// `what is the (exact )?value of (\w+)`, is a definition lookup, and anything+/// else is a how-does-it-work question.+pub fn detect_mode(question: &str, requested: &ModeRequest) -> (AnswerMode, Option<String>) {+ match requested {+ ModeRequest::Def(sym) => (AnswerMode::Def, Some(sym.clone())),+ ModeRequest::How => (AnswerMode::How, None),+ ModeRequest::Files => (AnswerMode::Files, None),+ ModeRequest::Auto => {+ let q = question.trim();+ if is_identifier(q) {+ return (AnswerMode::Def, Some(q.to_string()));+ }+ match value_of_symbol(q) {+ Some(sym) => (AnswerMode::Def, Some(sym)),+ None => (AnswerMode::How, None),+ }+ }+ }+}++/// `[A-Za-z_][A-Za-z0-9_]*` — the whole string is one identifier.+fn is_identifier(s: &str) -> bool {+ let mut chars = s.chars();+ match chars.next() {+ Some(c) if c.is_ascii_alphabetic() || c == '_' => {}+ _ => return false,+ }+ chars.all(|c| c.is_ascii_alphanumeric() || c == '_')+}++/// The symbol in `what is the (exact )?value of (\w+)`, matched anywhere in the+/// question and case-insensitively on the fixed words (a sentence starts with a+/// capital); the symbol keeps its own case.+fn value_of_symbol(q: &str) -> Option<String> {+ // ASCII lowercasing keeps every byte offset, so `pos` indexes `q` too.+ let lower = q.to_ascii_lowercase();+ for lead in ["what is the exact value of ", "what is the value of "] {+ if let Some(pos) = lower.find(lead) {+ let sym: String = q[pos + lead.len()..]+ .chars()+ .take_while(|c| c.is_alphanumeric() || *c == '_')+ .collect();+ if !sym.is_empty() {+ return Some(sym);+ }+ }+ }+ None+}++// ── step 4/5: the pull policy ────────────────────────────────────────────────++/// Whether the hub may pull a corpus on the caller's behalf.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum PullPolicy {+ Allow,+ /// `pack_bytes` is above [`AUTO_PULL_MAX_BYTES`]: the user must ask.+ TooLarge,+ /// The licence class is `excluded`: never pulled.+ Excluded,+}++/// `Excluded` is never pulled; above the limit is not pulled automatically;+/// anything else (including `approach-only`, which carries a warning instead) is.+pub fn pull_policy(entry: &CatalogueEntry) -> PullPolicy {+ if entry.licence_class == LicenceClass::Excluded {+ PullPolicy::Excluded+ } else if entry.pack_bytes > AUTO_PULL_MAX_BYTES {+ PullPolicy::TooLarge+ } else {+ PullPolicy::Allow+ }+}++/// `12.3` for a pack of 12.3 MiB.+pub fn megabytes(bytes: u64) -> String {+ format!("{:.1}", bytes as f64 / (1024.0 * 1024.0))+}++/// `run: xerj hub pull <slug> (<MB> MB)` — what to type for a pack too large to+/// pull automatically.+pub fn pull_hint(entry: &CatalogueEntry) -> String {+ format!(+ "run: xerj hub pull {} ({} MB)",+ entry.slug,+ megabytes(entry.pack_bytes)+ )+}++/// `approach-only licence (<spdx>): read, do not copy`.+pub fn approach_only_warning(spdx: &str) -> String {+ format!(+ "approach-only licence ({}): read, do not copy",+ if spdx.is_empty() { "unknown" } else { spdx }+ )+}++fn push_unique(list: &mut Vec<String>, line: String) {+ if !list.contains(&line) {+ list.push(line);+ }+}++// ── the hosted switch ────────────────────────────────────────────────────────++/// Where `--hosted auto` landed, and the one-time notice (if this run owes it).+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct HostedResolution {+ pub on: bool,+ pub notice: Option<String>,+}++const CONFIG_FILE: &str = "config.toml";+const NOTICE_MARKER: &str = "hosted_notice_shown";++/// Resolve `--hosted`. `on` / `off` are final. `auto` (the default) follows+/// `<hub>/config.toml`'s `hosted = "on" | "off"` when set, and otherwise is OFF+/// inside a git work tree and ON outside one (decision 1 in the discussion). The+/// first time it resolves to off inside a repo it returns [`HOSTED_OFF_NOTICE`]+/// and drops a marker so the notice is shown once.+pub fn resolve_hosted(flag: HostedSwitch, hub_home: &Path, cwd: &Path) -> HostedResolution {+ let decided = |on| HostedResolution { on, notice: None };+ match flag {+ HostedSwitch::On => decided(true),+ HostedSwitch::Off => decided(false),+ HostedSwitch::Auto => match config_hosted(hub_home).as_deref() {+ Some("on") => decided(true),+ Some("off") => decided(false),+ _ => {+ if !inside_git_worktree(cwd) {+ return decided(true);+ }+ let marker = hub_home.join(NOTICE_MARKER);+ let first_time = !marker.exists();+ if first_time {+ let _ = fs::create_dir_all(hub_home);+ let _ = fs::write(&marker, "shown\n");+ }+ HostedResolution {+ on: false,+ notice: first_time.then(|| HOSTED_OFF_NOTICE.to_string()),+ }+ }+ },+ }+}++/// The `hosted` value in `<hub>/config.toml`, if the file parses and sets one.+fn config_hosted(hub_home: &Path) -> Option<String> {+ let text = fs::read_to_string(hub_home.join(CONFIG_FILE)).ok()?;+ let table: toml::Table = text.parse().ok()?;+ table+ .get("hosted")+ .and_then(toml::Value::as_str)+ .map(str::to_string)+}++/// Write `hosted = "on" | "off"` into `<hub>/config.toml`, keeping any other+/// keys; `None` removes the key (back to `auto`).+pub fn set_config_hosted(hub_home: &Path, value: Option<&str>) -> Result<()> {+ let path = hub_home.join(CONFIG_FILE);+ let mut table: toml::Table = match fs::read_to_string(&path) {+ Ok(text) => text.parse()?,+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => toml::Table::new(),+ Err(e) => return Err(e.into()),+ };+ match value {+ Some(v) => {+ table.insert("hosted".into(), toml::Value::String(v.to_string()));+ }+ None => {+ table.remove("hosted");+ }+ }+ fs::create_dir_all(hub_home)?;+ fs::write(&path, toml::to_string(&table)?)?;+ Ok(())+}++/// True when `cwd` or an ancestor holds a `.git` entry (directory or worktree+/// gitlink file).+fn inside_git_worktree(cwd: &Path) -> bool {+ lockfile::project_root(cwd).join(".git").exists()+}++// ── the flow ─────────────────────────────────────────────────────────────────++/// Run the answer flow (plan §5, steps 1-7). Pure over [`AskDeps`]; the only+/// state it touches directly is the hosted-notice marker in `hub_home`.+pub fn ask(args: &AskArgs, deps: &AskDeps<'_>) -> AskOutcome {+ let mut out = AskOutcome::default();+ let (mode, symbol) = detect_mode(&args.question, &args.mode);++ // `--offline` never reaches the network, so there is nothing to resolve; a+ // resolved-off switch must never send the question to the hosted node.+ let hosted: Option<&dyn HostedHub> = if args.offline {+ None+ } else {+ let res = resolve_hosted(args.hosted, deps.hub_home, deps.cwd);+ if let Some(notice) = res.notice {+ out.stderr.push(notice);+ }+ if res.on {+ if deps.hosted.is_none() && args.hosted == HostedSwitch::On {+ out.stderr.push(NO_HOSTED_CLIENT.to_string());+ }+ deps.hosted+ } else {+ None+ }+ };++ // Step 2: route.+ let Some(slug) = select_corpus(args, deps, hosted, &mut out) else {+ return out;+ };+ let entry = deps.catalogue.corpora.iter().find(|e| e.slug == slug);+ let req = Request {+ slug: &slug,+ question: &args.question,+ mode,+ symbol: symbol.as_deref(),+ max_tokens: args.max_tokens,+ };++ // Step 3: a pack on this machine answers locally, with no network.+ match deps.backend.local_pack_ready(&slug) {+ Ok(true) => return answer_locally(args, deps, &req, entry, out),+ Ok(false) => {}+ Err(e) => out.stderr.push(format!("local pack unavailable: {e}")),+ }++ // Step 4: hosted answers now; the pull happens behind it.+ if let Some(h) = hosted {+ match h.answer(&req) {+ Ok(answer) => return finish_hosted(args, deps, entry, answer, out),+ Err(e) => out.stderr.push(format!("hosted answer failed: {e:#}")),+ }+ }++ // Step 5: neither a pack nor a hosted answer.+ let Some(entry) = entry else {+ out.stderr.push(format!(+ "corpus '{slug}' is not in the catalogue, so it cannot be pulled, and no hosted \+ answer is available"+ ));+ out.exit = EXIT_NO_ANSWER;+ return out;+ };+ if args.offline || args.no_fetch {+ let flag = if args.offline {+ "--offline"+ } else {+ "--no-fetch"+ };+ out.stderr.push(format!(+ "no local pack for '{slug}' and {flag} forbids downloading it; {}",+ pull_hint(entry)+ ));+ out.exit = EXIT_NO_ANSWER;+ return out;+ }+ match pull_policy(entry) {+ PullPolicy::Excluded => {+ out.stderr.push(format!(+ "'{slug}' has an excluded licence class ({}) and is never pulled; no hosted \+ answer is available",+ entry.licence_spdx+ ));+ out.exit = EXIT_NO_ANSWER;+ out+ }+ PullPolicy::TooLarge => {+ out.stderr.push(format!(+ "'{slug}' is {} MB, over the {} MB automatic pull limit, and no hosted answer \+ is available; {}",+ megabytes(entry.pack_bytes),+ megabytes(AUTO_PULL_MAX_BYTES),+ pull_hint(entry)+ ));+ out.exit = EXIT_NO_ANSWER;+ out+ }+ PullPolicy::Allow => match deps.backend.pull_now(entry) {+ Ok(()) => answer_locally(args, deps, &req, Some(entry), out),+ Err(f) => {+ out.stderr.push(f.message);+ out.exit = f.code;+ out+ }+ },+ }+}++/// Step 2. `None` means `out` already holds the verdict (an exit code and its+/// message / candidate lines).+fn select_corpus(+ args: &AskArgs,+ deps: &AskDeps<'_>,+ hosted: Option<&dyn HostedHub>,+ out: &mut AskOutcome,+) -> Option<String> {+ if let Some(c) = &args.corpus {+ if deps.catalogue.corpora.iter().any(|e| &e.slug == c) {+ return Some(c.clone());+ }+ out.stderr.push(format!(+ "unknown corpus '{c}' (catalogue generated {})",+ deps.catalogue.generated_at+ ));+ out.exit = EXIT_USAGE;+ return None;+ }++ let cached = cached_slugs(deps.hub_home);+ // Route on the catalogue alone first. The hosted node is asked only when+ // that is not decisive, so a question a local pack can answer never leaves+ // the machine.+ let mut routing = route(&args.question, deps.cwd, deps.catalogue, None, &cached);+ if !matches!(routing, Routing::Confident(_)) {+ if let Some(h) = hosted {+ routing = route(+ &args.question,+ deps.cwd,+ deps.catalogue,+ Some(&SearchOnly(h)),+ &cached,+ );+ }+ }+ match routing {+ Routing::Confident(slug) => Some(slug),+ Routing::Ambiguous(slugs) => {+ let lines: Vec<String> = slugs+ .iter()+ .map(|s| {+ let ask_for = deps+ .catalogue+ .corpora+ .iter()+ .find(|e| &e.slug == s)+ .map(|e| e.ask_for.replace(['\n', '\r'], " "))+ .unwrap_or_default();+ format!("{s} — {ask_for}")+ })+ .collect();+ out.stdout = lines.join("\n");+ out.stderr+ .push("more than one corpus matches; re-run with --corpus <slug>".to_string());+ out.exit = EXIT_NEEDS_DECISION;+ None+ }+ Routing::NoCorpus => {+ out.stderr.push(NO_CORPUS_MESSAGE.to_string());+ out.exit = EXIT_NO_ANSWER;+ None+ }+ }+}++/// Answer from the local pack and finish.+fn answer_locally(+ args: &AskArgs,+ deps: &AskDeps<'_>,+ req: &Request<'_>,+ entry: Option<&CatalogueEntry>,+ mut out: AskOutcome,+) -> AskOutcome {+ match deps.backend.answer_local(req) {+ Ok(answer) => {+ finalize(args, entry, answer, Vec::new(), &mut out);+ out+ }+ Err(e) => {+ out.stderr.push(e);+ out.exit = EXIT_NO_ANSWER;+ out+ }+ }+}++/// A hosted answer was produced: apply the pull policy (spawning the background+/// pull when allowed) and finish.+fn finish_hosted(+ args: &AskArgs,+ deps: &AskDeps<'_>,+ entry: Option<&CatalogueEntry>,+ mut answer: Answer,+ mut out: AskOutcome,+) -> AskOutcome {+ answer.answered_from = AnsweredFrom::Hosted;+ let mut notes: Vec<String> = Vec::new();+ match entry {+ Some(e) => match pull_policy(e) {+ PullPolicy::Allow => {+ if !args.no_fetch {+ match deps.backend.pull_in_background(&e.slug) {+ Ok(BackgroundPull::Started { pid }) => out.stderr.push(format!(+ "pulling '{}' in the background (pid {pid}); the next answer is local",+ e.slug+ )),+ Ok(BackgroundPull::AlreadyRunning { .. }) => {}+ Err(msg) => out+ .stderr+ .push(format!("could not start the background pull: {msg}")),+ }+ }+ }+ PullPolicy::TooLarge => notes.push(pull_hint(e)),+ PullPolicy::Excluded => notes.push(format!(+ "excluded licence ({}): this corpus is never pulled",+ e.licence_spdx+ )),+ },+ None => notes.push(format!(+ "corpus '{}' is not in the local catalogue; it was not pulled",+ answer.corpus+ )),+ }+ finalize(args, entry, answer, notes, &mut out);+ out+}++/// Common tail: licence warnings, the token cap, printing, the exit code.+fn finalize(+ args: &AskArgs,+ entry: Option<&CatalogueEntry>,+ mut answer: Answer,+ notes: Vec<String>,+ out: &mut AskOutcome,+) {+ for n in notes {+ push_unique(&mut answer.warnings, n);+ }+ let approach_only = answer.licence_class == LicenceClass::ApproachOnly+ || entry.is_some_and(|e| e.licence_class == LicenceClass::ApproachOnly);+ if approach_only {+ let spdx = entry+ .map(|e| e.licence_spdx.as_str())+ .filter(|s| !s.is_empty())+ .unwrap_or(&answer.licence_spdx)+ .to_string();+ let warning = approach_only_warning(&spdx);+ for hit in &mut answer.hits {+ if hit.licence_warning.is_none() {+ hit.licence_warning = Some(warning.clone());+ }+ }+ push_unique(&mut answer.warnings, warning);+ }+ if let Some(n) = args.max_tokens {+ cap_tokens(&mut answer, n);+ }+ out.stdout = if args.json {+ serde_json::to_string(&answer).unwrap_or_else(|e| format!("{{\"error\":\"{e}\"}}"))+ } else {+ render(&answer)+ };+ out.exit = if answer.hits.is_empty() {+ EXIT_NO_ANSWER+ } else {+ EXIT_OK+ };+ out.answer = Some(answer);+}++/// Keep the rendered answer within roughly `max_tokens` (4 characters a token):+/// drop trailing hits, then shorten the last one's text, marking the answer+/// `truncated`. The first hit is always kept.+pub fn cap_tokens(answer: &mut Answer, max_tokens: usize) {+ let budget = max_tokens.saturating_mul(4);+ let size = |a: &Answer| render(a).chars().count();+ while size(answer) > budget && answer.hits.len() > 1 {+ answer.hits.pop();+ answer.truncated = true;+ }+ let over = size(answer).saturating_sub(budget);+ if over > 0 {+ if let Some(hit) = answer.hits.first_mut() {+ let keep = hit.text.chars().count().saturating_sub(over + 1);+ let mut shortened: String = hit.text.chars().take(keep).collect();+ shortened.push('…');+ hit.text = shortened;+ answer.truncated = true;+ }+ }+}++// ── the pack cache on disk ───────────────────────────────────────────────────++const PACKS_DIR: &str = "packs";+const PACK_MANIFEST_FILE: &str = "pack.json";+const LAST_USED_FILE: &str = "last_used";++/// One extracted pack: `<hub>/packs/<slug>/<pin8>/`.+#[derive(Debug, Clone)]+pub struct PackDir {+ pub slug: String,+ pub pin8: String,+ pub dir: PathBuf,+ /// The mtime of the pack's `last_used` file, else of the directory itself.+ pub last_used: SystemTime,+}++/// A [`PackDir`] with its size on disk.+#[derive(Debug, Clone)]+pub struct CachedPack {+ pub pack: PackDir,+ pub bytes: u64,+}++/// Every complete pack (a directory holding `pack.json`) under `<hub>/packs`,+/// sorted by slug then pin. Cheap: no directory is walked for its size.+pub fn pack_dirs(hub_home: &Path) -> Vec<PackDir> {+ let mut out = Vec::new();+ let Ok(slugs) = fs::read_dir(hub_home.join(PACKS_DIR)) else {+ return out;+ };+ for slug_ent in slugs.flatten() {+ if !slug_ent.file_type().is_ok_and(|t| t.is_dir()) {+ continue;+ }+ let slug = slug_ent.file_name().to_string_lossy().into_owned();+ let Ok(pins) = fs::read_dir(slug_ent.path()) else {+ continue;+ };+ for pin_ent in pins.flatten() {+ let dir = pin_ent.path();+ if !pin_ent.file_type().is_ok_and(|t| t.is_dir())+ || !dir.join(PACK_MANIFEST_FILE).is_file()+ {+ continue; // a `.tmp`, a `.lock`, or a half-extracted dir+ }+ let mtime = |p: &Path| fs::metadata(p).and_then(|m| m.modified()).ok();+ let last_used = mtime(&dir.join(LAST_USED_FILE))+ .or_else(|| mtime(&dir))+ .unwrap_or(SystemTime::UNIX_EPOCH);+ out.push(PackDir {+ slug: slug.clone(),+ pin8: pin_ent.file_name().to_string_lossy().into_owned(),+ dir,+ last_used,+ });+ }+ }+ out.sort_by(|a, b| a.slug.cmp(&b.slug).then_with(|| a.pin8.cmp(&b.pin8)));+ out+}++/// [`pack_dirs`] with each pack's size.+pub fn list_cached_packs(hub_home: &Path) -> Vec<CachedPack> {+ pack_dirs(hub_home)+ .into_iter()+ .map(|pack| CachedPack {+ bytes: dir_size(&pack.dir),+ pack,+ })+ .collect()+}++/// Slugs with at least one pack on disk (what the router gives a small bonus).+pub fn cached_slugs(hub_home: &Path) -> Vec<String> {+ let mut slugs: Vec<String> = pack_dirs(hub_home).into_iter().map(|p| p.slug).collect();+ slugs.dedup(); // already sorted by slug+ slugs+}++/// Mark every pack of `slug` as just used (the `last_used` file each pack+/// directory gets on every answer). Best effort: LRU order is a courtesy, never+/// a reason to fail an answer.+pub fn touch_last_used(hub_home: &Path, slug: &str) {+ let stamp = chrono::Utc::now().timestamp().to_string();+ for p in pack_dirs(hub_home).into_iter().filter(|p| p.slug == slug) {+ let _ = fs::write(p.dir.join(LAST_USED_FILE), &stamp);+ }+}++/// Total bytes under `dir` (symlinks are not followed).+fn dir_size(dir: &Path) -> u64 {+ let Ok(rd) = fs::read_dir(dir) else { return 0 };+ rd.flatten()+ .map(|e| match e.file_type() {+ Ok(t) if t.is_dir() => dir_size(&e.path()),+ Ok(t) if t.is_file() => e.metadata().map(|m| m.len()).unwrap_or(0),+ _ => 0,+ })+ .sum()+}++/// Make room for a pack of `incoming_bytes` (plan §5 step 6): when `<hub>/packs`+/// plus the incoming pack would exceed `budget`, remove whole packs, least+/// recently used first, until it fits — never a pack of `keep_slug` (the one+/// being pulled). Returns what was removed. If everything else is gone and the+/// budget is still exceeded, the pull proceeds anyway: the budget bounds the+/// cache, it does not veto the pack being fetched.+///+/// Only the extracted pack directories are removed. The restored indices live in+/// the node's own data directory and are not touched — deleting indices from a+/// node is not this function's business.+pub fn evict_to_budget(+ hub_home: &Path,+ keep_slug: &str,+ incoming_bytes: u64,+ budget: u64,+) -> Vec<CachedPack> {+ let mut total = dir_size(&hub_home.join(PACKS_DIR)).saturating_add(incoming_bytes);+ let mut evicted = Vec::new();+ if total <= budget {+ return evicted;+ }+ let mut candidates: Vec<CachedPack> = list_cached_packs(hub_home)+ .into_iter()+ .filter(|c| c.pack.slug != keep_slug)+ .collect();+ candidates.sort_by(|a, b| {+ a.pack+ .last_used+ .cmp(&b.pack.last_used)+ .then_with(|| a.pack.slug.cmp(&b.pack.slug))+ .then_with(|| a.pack.pin8.cmp(&b.pack.pin8))+ });+ for c in candidates {+ if total <= budget {+ break;+ }+ if fs::remove_dir_all(&c.pack.dir).is_ok() {+ total = total.saturating_sub(c.bytes);+ // Drop the slug directory once its last pin is gone.+ if let Some(parent) = c.pack.dir.parent() {+ let _ = fs::remove_dir(parent);+ }+ evicted.push(c);+ }+ }+ evicted+}++// ── answers from a local pack ────────────────────────────────────────────────++/// The corpus facts an [`Answer`] carries, from the state ledger `hub pull`+/// wrote (the same source `def::lookup` uses, so the two modes agree).+struct AnswerMeta {+ corpus: String,+ pin_sha: String,+ pin_date: String,+ age_days: u32,+ licence_spdx: String,+ licence_class: LicenceClass,+ answered_from: AnsweredFrom,+}++fn answer_meta(root: &Path, slug: &str) -> AnswerMeta {+ match state::load_state(root, slug) {+ Ok(st) => {+ let pin_date = st.pin_date.clone().unwrap_or_default();+ AnswerMeta {+ corpus: st.name(slug),+ pin_sha: st.pin_sha.clone().unwrap_or_default(),+ age_days: age_days(&pin_date, st.indexed_at.as_deref()),+ pin_date,+ licence_spdx: st.licence_spdx.clone().unwrap_or_default(),+ licence_class: match st.licence_class.as_deref() {+ Some("approach-only") => LicenceClass::ApproachOnly,+ Some("excluded") => LicenceClass::Excluded,+ _ => LicenceClass::AdaptWithAttribution,+ },+ answered_from: if st.is_hubpack() {+ AnsweredFrom::LocalPack+ } else {+ AnsweredFrom::LocalIndex+ },+ }+ }+ Err(_) => AnswerMeta {+ corpus: slug.to_string(),+ pin_sha: String::new(),+ pin_date: String::new(),+ age_days: 0,+ licence_spdx: String::new(),+ licence_class: LicenceClass::AdaptWithAttribution,+ answered_from: AnsweredFrom::LocalPack,+ },+ }+}++/// Days from the pin date (`YYYY-MM-DD`) to today; else from `indexed_at`; else 0.+fn age_days(pin_date: &str, indexed_at: Option<&str>) -> u32 {+ if let Ok(d) = chrono::NaiveDate::parse_from_str(pin_date, "%Y-%m-%d") {+ return (chrono::Utc::now().date_naive() - d).num_days().max(0) as u32;+ }+ if let Some(stamp) = indexed_at {+ if let Ok(dt) = chrono::NaiveDateTime::parse_from_str(stamp, "%Y-%m-%dT%H:%M:%SZ") {+ let age = chrono::Utc::now()+ .naive_utc()+ .signed_duration_since(dt)+ .num_days();+ return age.max(0) as u32;+ }+ }+ 0+}++/// Turn a code-search response (the raw `_search` body `run_code_query` returns+/// with `as_json`) into an [`Answer`] for How or Files mode.+///+/// How: one [`Hit`] per record, the passage chosen exactly as `xerj code` chooses+/// it (the matching definition when the record lists symbols, else the densest+/// window), `kind` the definition's kind or `passage`, and `line` the line that+/// definition starts at — or the record's own `line`, or 0 when it carries none:+/// a line number is reported only when the index supplied one. Files: one hit+/// per distinct file, its licence as the text.+///+/// `licences` maps a repo (the first path segment) to its licence; a repo with a+/// restricted licence tags its hits with the read-don't-copy warning.+fn answer_from_search(+ meta: AnswerMeta,+ resp: &Value,+ question: &str,+ mode: AnswerMode,+ licences: &HashMap<String, String>,+) -> Answer {+ let rows = resp+ .pointer("/hits/hits")+ .and_then(Value::as_array)+ .cloned()+ .unwrap_or_default();+ let mut seen_files: HashSet<String> = HashSet::new();+ let mut hits = Vec::new();+ for row in &rows {+ let src = row.get("_source").cloned().unwrap_or(Value::Null);+ let locator = passage::provenance(&src);+ let (file, record_line) = split_locator(&locator);+ let score = row.get("_score").and_then(Value::as_f64).unwrap_or(0.0) as f32;+ let lic = licences+ .get(passage::locator_repo(&locator))+ .cloned()+ .unwrap_or_default();+ let licence_warning =+ (!lic.is_empty() && licence::is_restricted(&lic)).then(|| approach_only_warning(&lic));++ if mode == AnswerMode::Files {+ if !seen_files.insert(file.clone()) {+ continue;+ }+ let shown = if lic.is_empty() {+ meta.licence_spdx.as_str()+ } else {+ lic.as_str()+ };+ hits.push(Hit {+ file,+ line: record_line,+ kind: "file".to_string(),+ text: format!("licence: {shown}"),+ value: None,+ score,+ licence_warning,+ });+ continue;+ }++ let body = src+ .get("body")+ .and_then(Value::as_str)+ .or_else(|| src.get("text").and_then(Value::as_str))+ .map(str::to_string)+ .unwrap_or_else(|| longest_string_field(&src));+ let (text, kind, line) = match passage::symbol_passage(&body, &src, question, PASSAGE_WIDTH)+ {+ Some((text, label)) => {+ let (kind, line) = parse_symbol_label(&label);+ (text, kind, line.unwrap_or(record_line))+ }+ None => {+ let (window, _start, _total) = passage::best_window(&body, question, PASSAGE_WIDTH);+ (window, "passage".to_string(), record_line)+ }+ };+ hits.push(Hit {+ file,+ line,+ kind,+ text,+ value: None,+ score,+ licence_warning,+ });+ }+ Answer {+ corpus: meta.corpus,+ pin_sha: meta.pin_sha,+ pin_date: meta.pin_date,+ age_days: meta.age_days,+ licence_spdx: meta.licence_spdx,+ licence_class: meta.licence_class,+ answered_from: meta.answered_from,+ mode,+ hits,+ truncated: false,+ warnings: Vec::new(),+ }+}++/// `path:line` → (`path`, `line`); a locator with no numeric `:line` suffix is+/// all path, line 0.+fn split_locator(locator: &str) -> (String, u32) {+ if let Some((path, tail)) = locator.rsplit_once(':') {+ if let Ok(n) = tail.parse::<u32>() {+ return (path.to_string(), n);+ }+ }+ (locator.to_string(), 0)+}++/// `function foo @ line 12` → (`function`, Some(12)).+fn parse_symbol_label(label: &str) -> (String, Option<u32>) {+ let kind = label.split_whitespace().next().unwrap_or("passage");+ let line = label+ .rsplit_once("@ line ")+ .and_then(|(_, n)| n.trim().parse::<u32>().ok());+ (kind.to_string(), line)+}++/// The longest non-empty string field of a record, "" when there is none — the+/// content of a raw-JSON record that has neither `body` nor `text` (the same+/// rule `xerj code`'s renderer applies).+fn longest_string_field(src: &Value) -> String {+ let mut best: Option<(usize, &str)> = None;+ if let Some(obj) = src.as_object() {+ let mut keys: Vec<&String> = obj.keys().collect();+ keys.sort();+ for key in keys {+ if let Some(s) = obj[key].as_str() {+ let n = s.chars().count();+ if !s.trim().is_empty() && best.is_none_or(|(len, _)| n > len) {+ best = Some((n, s));+ }+ }+ }+ }+ best.map(|(_, s)| s.to_string()).unwrap_or_default()+}++/// repo → licence from a pack's own `pack.json` (`sources[]`): the licence data+/// a machine that never cloned the corpus still has.+fn pack_licences(manifest: &PackManifest) -> HashMap<String, String> {+ manifest+ .sources+ .iter()+ .map(|s| (s.repo.clone(), s.licence.clone()))+ .collect()+}++/// The pack manifest the ledger's pin names, else the most recently used pack+/// of `slug` on disk.+fn read_pack_manifest(hub_home: &Path, slug: &str, pin_sha: &str) -> Option<PackManifest> {+ let packs: Vec<PackDir> = pack_dirs(hub_home)+ .into_iter()+ .filter(|p| p.slug == slug)+ .collect();+ let want: String = pin_sha.chars().take(8).collect::<String>().to_lowercase();+ let pack = packs+ .iter()+ .find(|p| !want.is_empty() && p.pin8 == want)+ .or_else(|| packs.iter().max_by_key(|p| p.last_used))?;+ let text = fs::read_to_string(pack.dir.join(PACK_MANIFEST_FILE)).ok()?;+ serde_json::from_str(&text).ok()+}++// ── the production backend ───────────────────────────────────────────────────++/// [`Backend`] over a real node and the real `hub pull`.+pub struct RealBackend {+ hub_home: PathBuf,+ url_flag: Option<String>,+ launcher: NodeLauncher,+ /// The node, resolved once per run: the outcome — success or failure — is+ /// kept, so a node that could not be started is not retried (and waited on+ /// for another minute) by the next step of the same question.+ node: RefCell<Option<Result<SharedNode, String>>>,+}++/// The resolved node and a client connected to it, shared within one run.+type SharedNode = Rc<(ResolvedNode, HubNode)>;++impl RealBackend {+ pub fn new(hub_home: PathBuf, url_flag: Option<String>, launcher: NodeLauncher) -> Self {+ RealBackend {+ hub_home,+ url_flag,+ launcher,+ node: RefCell::new(None),+ }+ }++ /// The node queries run against, resolved (and started if need be) on first+ /// use and then reused for the rest of the run.+ fn node(&self) -> Result<SharedNode, String> {+ if let Some(done) = self.node.borrow().as_ref() {+ return done.clone();+ }+ let resolved = node::ensure_node(&self.hub_home, self.url_flag.as_deref(), &self.launcher)+ .map_err(|e| format!("{e:#}"))+ .and_then(|r| {+ let client = r.connect().map_err(|e| format!("{e:#}"))?;+ Ok(Rc::new((r, client)))+ });+ *self.node.borrow_mut() = Some(resolved.clone());+ resolved+ }++ /// The How / Files path: the existing retrieval pipeline, asked for its raw+ /// response, converted to an [`Answer`].+ fn answer_search(+ &self,+ node: &HubNode,+ url: &str,+ req: &Request<'_>,+ ) -> Result<Answer, String> {+ let root = xerj_autoindex::xc::code_root();+ let mut params = CodeParams::new(req.slug, req.question);+ params.k = if req.mode == AnswerMode::Files {+ FILES_K+ } else {+ DEFAULT_K+ };+ params.full = PASSAGE_WIDTH;+ params.as_json = true;+ let outcome = xccode::run_code_query(&root, node, url, ¶ms, "`--stale-ok`");+ let resp = match outcome.json {+ Some(v) => v,+ None => return Err(outcome.text.trim().to_string()),+ };+ let meta = answer_meta(&root, req.slug);+ let licences = read_pack_manifest(&self.hub_home, req.slug, &meta.pin_sha)+ .map(|m| pack_licences(&m))+ .unwrap_or_default();+ let mut answer = answer_from_search(meta, &resp, req.question, req.mode, &licences);+ answer.warnings.extend(outcome.warnings);+ Ok(answer)+ }+}++impl Backend for RealBackend {+ fn local_pack_ready(&self, slug: &str) -> Result<bool, String> {+ // No pack on disk: not cached, and no reason to start a node to say so.+ if !pack_dirs(&self.hub_home).iter().any(|p| p.slug == slug) {+ return Ok(false);+ }+ let root = xerj_autoindex::xc::code_root();+ let Ok(st) = state::load_state(&root, slug) else {+ return Ok(false); // pack files but no ledger: re-pulling rewrites it+ };+ let Some(manifest) = read_pack_manifest(+ &self.hub_home,+ slug,+ st.pin_sha.as_deref().unwrap_or_default(),+ ) else {+ return Ok(false);+ };+ let node = self.node()?;+ let live = node+ .1+ .cat_indices_json(&format!("{}*", manifest.index_prefix))+ .unwrap_or_default();+ // Every index of the pack must be live: a partial restore would answer a+ // definition lookup "no match" for the wrong reason.+ Ok(!manifest.indices.is_empty()+ && manifest+ .indices+ .iter()+ .all(|i| live.iter().any(|n| n == &i.name)))+ }++ fn answer_local(&self, req: &Request<'_>) -> Result<Answer, String> {+ let node = self.node()?;+ let answer = match (req.mode, req.symbol) {+ (AnswerMode::Def, Some(symbol)) => def::lookup(&node.1, req.slug, symbol, DEFAULT_K),+ _ => self.answer_search(&node.1, &node.0.url, req)?,+ };+ touch_last_used(&self.hub_home, req.slug);+ Ok(answer)+ }++ fn pull_now(&self, entry: &CatalogueEntry) -> Result<(), PullFailure> {+ let node = self.node().map_err(|message| PullFailure {+ code: EXIT_NO_ANSWER,+ message,+ })?;+ eprintln!(+ "xerj hub: pulling '{}' ({} MB) — one-time download, later questions answer locally",+ entry.slug,+ megabytes(entry.pack_bytes)+ );+ match cli::pull(&entry.slug, &node.0.url, node.0.managed, false) {+ Ok(cli::Pulled::Already) => Ok(()),+ Ok(cli::Pulled::Done(line)) => {+ eprintln!("xerj hub: {line}");+ Ok(())+ }+ Err(e) => Err(PullFailure {+ code: e.code,+ message: format!("pull of '{}' failed: {}", entry.slug, e.message),+ }),+ }+ }++ fn pull_in_background(&self, slug: &str) -> Result<BackgroundPull, String> {+ // The slug names files below (`pulls/<slug>.pid`) and is an argument of+ // the child: refuse anything `hub pull` itself would refuse, before it+ // touches a path. (The catalogue may be unverified until the hub key+ // ships.)+ if !cli::valid_slug(slug) {+ return Err(format!("'{slug}' is not a valid corpus slug"));+ }+ let pulls = self.hub_home.join("pulls");+ let pid_file = pulls.join(format!("{slug}.pid"));+ let needle = format!("hub pull {slug}");+ if let Some(pid) = fs::read_to_string(&pid_file)+ .ok()+ .and_then(|s| s.trim().parse::<u32>().ok())+ {+ if node::identify(pid, &[&needle]) == node::Identity::Ours {+ return Ok(BackgroundPull::AlreadyRunning { pid });+ }+ }+ let mut args = vec![+ "hub".to_string(),+ "pull".to_string(),+ slug.to_string(),+ "--quiet".to_string(),+ ];+ if let Some(u) = &self.url_flag {+ args.push("--url".to_string());+ args.push(u.clone());+ }+ let child = node::spawn_detached(&self.launcher, &args, &pulls.join(format!("{slug}.log")))+ .map_err(|e| format!("{e:#}"))?;+ let pid = child.id();+ fs::write(&pid_file, format!("{pid}\n")).map_err(|e| e.to_string())?;+ // The child outlives this process; dropping the handle neither kills+ // nor waits for it.+ drop(child);+ Ok(BackgroundPull::Started { pid })+ }+}++// ── the library entry point ──────────────────────────────────────────────────++/// Load the catalogue, build the production backend, and run [`ask`]. The CLI+/// prints the outcome; another caller (the MCP tool, plan §7) can use+/// [`AskOutcome::answer`] directly. `hosted` is the hosted client to use, or+/// `None` when none is configured.+pub fn run_ask(args: &AskArgs, hosted: Option<&dyn HostedHub>) -> AskOutcome {+ let hub_home = cli::hub_home();+ let hub_url = std::env::var("XERJ_HUB_URL")+ .ok()+ .filter(|u| !u.is_empty())+ .unwrap_or_else(|| DEFAULT_HUB_URL.to_string());+ let catalogue = match load_catalogue(&hub_url, &hub_home, args.offline) {+ Ok(c) => c,+ Err(e) => {+ return AskOutcome::fail(+ EXIT_NO_ANSWER,+ format!("cannot load the hub catalogue: {e:#}"),+ )+ }+ };+ let launcher = match NodeLauncher::current() {+ Ok(l) => l,+ Err(e) => return AskOutcome::fail(EXIT_NO_ANSWER, format!("{e:#}")),+ };+ let cwd = std::env::current_dir().unwrap_or_else(|_| PathBuf::from("."));+ let backend = RealBackend::new(hub_home.clone(), args.url.clone(), launcher);+ ask(+ args,+ &AskDeps {+ hub_home: &hub_home,+ cwd: &cwd,+ catalogue: &catalogue,+ hosted,+ backend: &backend,+ },+ )+}++#[cfg(test)]+mod tests {+ use super::*;+ use std::cell::{Cell, RefCell};+ use std::time::Duration;++ use serde_json::json;+ use xerj_common::hub::Sketch;++ // ── fixtures ────────────────────────────────────────────────────────────++ fn sketch(terms: &[&str]) -> Sketch {+ crate::sketch::build(terms.iter().map(|t| t.to_string()))+ }++ fn entry(slug: &str, bytes: u64, class: LicenceClass, terms: &[&str]) -> CatalogueEntry {+ CatalogueEntry {+ slug: slug.to_string(),+ domain: "TEST".to_string(),+ topics: vec![],+ ask_for: format!("an agent working on {slug} queries this for its internals"),+ pin_sha: "0".repeat(40),+ pin_date: "2026-10-01".to_string(),+ refresh_cadence_days: 7,+ licence_spdx: if class == LicenceClass::ApproachOnly {+ "GPL-3.0".to_string()+ } else {+ "MIT".to_string()+ },+ licence_class: class,+ pack_bytes: bytes,+ pack_url: String::new(),+ pack_sha256: String::new(),+ sig_url: String::new(),+ dependency_names: vec![],+ quality_grade: None,+ term_sketch: sketch(terms),+ }+ }++ fn lz4(bytes: u64, class: LicenceClass) -> CatalogueEntry {+ entry(+ "lz4",+ bytes,+ class,+ &["lz4", "block", "decoder", "compression"],+ )+ }++ /// Four unrelated corpora, so that a term found in only one of the others+ /// is not "common vocabulary" (the router halves terms present in more than+ /// a quarter of all sketches) and a sketch match can be decisive.+ fn with_fillers(mut entries: Vec<CatalogueEntry>) -> Vec<CatalogueEntry> {+ for (slug, terms) in [+ ("filler-a", ["alpha", "bravo", "charlie"]),+ ("filler-b", ["delta", "echo", "foxtrot"]),+ ("filler-c", ["golf", "hotel", "india"]),+ ("filler-d", ["juliet", "kilo", "lima"]),+ ] {+ entries.push(entry(slug, 1, LicenceClass::AdaptWithAttribution, &terms));+ }+ entries+ }++ fn catalogue(entries: Vec<CatalogueEntry>) -> Catalogue {+ Catalogue {+ format_version: 1,+ generated_at: "2026-10-14T00:00:00Z".to_string(),+ corpora: entries,+ }+ }++ fn canned(slug: &str, from: AnsweredFrom, mode: AnswerMode) -> Answer {+ Answer {+ corpus: slug.to_string(),+ pin_sha: "0".repeat(40),+ pin_date: "2026-10-01".to_string(),+ age_days: 13,+ licence_spdx: "MIT".to_string(),+ licence_class: LicenceClass::AdaptWithAttribution,+ answered_from: from,+ mode,+ hits: vec![Hit {+ file: "lz4/lib/lz4.h".to_string(),+ line: 100,+ kind: "define".to_string(),+ text: "#define LZ4_VERSION_MAJOR 1".to_string(),+ value: Some("1".to_string()),+ score: 1.0,+ licence_warning: None,+ }],+ truncated: false,+ warnings: vec![],+ }+ }++ /// A backend that records every call and never touches a node.+ struct FakeBackend {+ ready: Cell<bool>,+ ready_err: Option<String>,+ pull_fail: RefCell<Option<PullFailure>>,+ log: RefCell<Vec<String>>,+ }++ impl FakeBackend {+ fn new(ready: bool) -> Self {+ FakeBackend {+ ready: Cell::new(ready),+ ready_err: None,+ pull_fail: RefCell::new(None),+ log: RefCell::new(vec![]),+ }+ }+ fn calls(&self) -> Vec<String> {+ self.log.borrow().clone()+ }+ fn called(&self, prefix: &str) -> bool {+ self.log.borrow().iter().any(|c| c.starts_with(prefix))+ }+ }++ impl Backend for FakeBackend {+ fn local_pack_ready(&self, slug: &str) -> Result<bool, String> {+ self.log.borrow_mut().push(format!("ready:{slug}"));+ match &self.ready_err {+ Some(e) => Err(e.clone()),+ None => Ok(self.ready.get()),+ }+ }+ fn answer_local(&self, req: &Request<'_>) -> Result<Answer, String> {+ self.log+ .borrow_mut()+ .push(format!("answer_local:{}", req.slug));+ Ok(canned(req.slug, AnsweredFrom::LocalPack, req.mode))+ }+ fn pull_now(&self, entry: &CatalogueEntry) -> Result<(), PullFailure> {+ self.log+ .borrow_mut()+ .push(format!("pull_now:{}", entry.slug));+ if let Some(f) = self.pull_fail.borrow().clone() {+ return Err(f);+ }+ self.ready.set(true);+ Ok(())+ }+ fn pull_in_background(&self, slug: &str) -> Result<BackgroundPull, String> {+ self.log.borrow_mut().push(format!("pull_bg:{slug}"));+ Ok(BackgroundPull::Started { pid: 4242 })+ }+ }++ /// A hosted hub that fails the test if it is ever contacted.+ struct NoNetwork;+ impl HostedSearch for NoNetwork {+ fn search(&self, _q: &str) -> Result<Vec<(String, f32)>> {+ panic!("hosted search must not be called");+ }+ }+ impl HostedHub for NoNetwork {+ fn answer(&self, _req: &Request<'_>) -> Result<Answer> {+ panic!("hosted answer must not be called");+ }+ }++ /// A hosted hub that answers from a canned response and counts its calls.+ struct StubHosted {+ found: Vec<(String, f32)>,+ fail: bool,+ searches: Cell<usize>,+ answers: Cell<usize>,+ }+ impl StubHosted {+ fn new() -> Self {+ StubHosted {+ found: vec![],+ fail: false,+ searches: Cell::new(0),+ answers: Cell::new(0),+ }+ }+ }+ impl HostedSearch for StubHosted {+ fn search(&self, _q: &str) -> Result<Vec<(String, f32)>> {+ self.searches.set(self.searches.get() + 1);+ Ok(self.found.clone())+ }+ }+ impl HostedHub for StubHosted {+ fn answer(&self, req: &Request<'_>) -> Result<Answer> {+ self.answers.set(self.answers.get() + 1);+ if self.fail {+ anyhow::bail!("hosted node unreachable");+ }+ // Deliberately mislabelled: `ask` must stamp it `Hosted` itself.+ Ok(canned(req.slug, AnsweredFrom::LocalPack, req.mode))+ }+ }++ /// Everything an `ask` run needs, in throwaway directories.+ struct Rig {+ hub: tempfile::TempDir,+ cwd: tempfile::TempDir,+ }+ impl Rig {+ fn new() -> Self {+ Rig {+ hub: tempfile::tempdir().unwrap(),+ cwd: tempfile::tempdir().unwrap(),+ }+ }+ /// Make `cwd` look like a git work tree.+ fn in_git_repo(self) -> Self {+ fs::create_dir(self.cwd.path().join(".git")).unwrap();+ self+ }+ fn run(+ &self,+ args: &AskArgs,+ cat: &Catalogue,+ hosted: Option<&dyn HostedHub>,+ backend: &dyn Backend,+ ) -> AskOutcome {+ ask(+ args,+ &AskDeps {+ hub_home: self.hub.path(),+ cwd: self.cwd.path(),+ catalogue: cat,+ hosted,+ backend,+ },+ )+ }+ }++ fn args(question: &str, corpus: Option<&str>) -> AskArgs {+ let mut a = AskArgs::new(question);+ a.corpus = corpus.map(str::to_string);+ a+ }++ // ── step 1: mode ────────────────────────────────────────────────────────++ #[test]+ fn mode_is_def_for_the_flag_a_lone_identifier_and_the_value_of_phrase() {+ let auto = ModeRequest::Auto;+ assert_eq!(+ detect_mode("x", &ModeRequest::Def("LZ4_VERSION".into())),+ (AnswerMode::Def, Some("LZ4_VERSION".to_string()))+ );+ assert_eq!(+ detect_mode("LZ4_VERSION_RELEASE", &auto),+ (AnswerMode::Def, Some("LZ4_VERSION_RELEASE".to_string()))+ );+ assert_eq!(+ detect_mode(" mdb_maxkeysize\n", &auto),+ (AnswerMode::Def, Some("mdb_maxkeysize".to_string()))+ );+ // The phrase, matched anywhere and on its fixed words case-insensitively.+ for q in [+ "what is the value of MDB_MAXKEYSIZE",+ "What is the value of MDB_MAXKEYSIZE?",+ "in lmdb, what is the exact value of MDB_MAXKEYSIZE in a release build",+ "WHAT IS THE EXACT VALUE OF MDB_MAXKEYSIZE",+ ] {+ assert_eq!(+ detect_mode(q, &auto),+ (AnswerMode::Def, Some("MDB_MAXKEYSIZE".to_string())),+ "{q}"+ );+ }+ }++ #[test]+ fn mode_is_how_for_prose_and_flags_force_the_rest() {+ let auto = ModeRequest::Auto;+ for q in [+ "how does the lz4 frame decoder work",+ "lz4 decoder", // two words: not one identifier+ "what is the value of", // no symbol after the phrase+ "what is the value of ???", // no word character after the phrase+ "LZ4-VERSION", // a hyphen is not an identifier character+ "",+ ] {+ assert_eq!(detect_mode(q, &auto), (AnswerMode::How, None), "{q:?}");+ }+ // A forced mode ignores what the question looks like.+ assert_eq!(+ detect_mode("LZ4_VERSION_RELEASE", &ModeRequest::How),+ (AnswerMode::How, None)+ );+ assert_eq!(+ detect_mode("LZ4_VERSION_RELEASE", &ModeRequest::Files),+ (AnswerMode::Files, None)+ );+ }++ // ── step 3: a cached pack never touches the network ─────────────────────++ #[test]+ fn a_cached_pack_answers_locally_and_never_contacts_the_hosted_hub() {+ let rig = Rig::new();+ let cat = catalogue(with_fillers(vec![+ lz4(1024, LicenceClass::AdaptWithAttribution),+ entry(+ "zstd",+ 1024,+ LicenceClass::AdaptWithAttribution,+ &["zstd", "frame"],+ ),+ ]));+ let backend = FakeBackend::new(true);++ // Routed by the sketch (no --corpus): hosted is on (cwd is not a repo)+ // but the sketch is decisive, so the hosted node is never asked.+ let out = rig.run(+ &args("lz4 block decoder", None),+ &cat,+ Some(&NoNetwork),+ &backend,+ );+ assert_eq!(out.exit, EXIT_OK, "{:?}", out.stderr);+ let ans = out.answer.expect("an answer");+ assert_eq!(ans.answered_from, AnsweredFrom::LocalPack);+ assert!(+ out.stdout.contains("answered from: local pack"),+ "{}",+ out.stdout+ );+ assert_eq!(backend.calls(), ["ready:lz4", "answer_local:lz4"]);++ // And with --corpus (routing skipped altogether).+ let backend = FakeBackend::new(true);+ let out = rig.run(+ &args("anything", Some("lz4")),+ &cat,+ Some(&NoNetwork),+ &backend,+ );+ assert_eq!(out.exit, EXIT_OK);+ assert_eq!(backend.calls(), ["ready:lz4", "answer_local:lz4"]);+ }++ #[test]+ fn hosted_is_consulted_for_routing_only_when_the_sketch_is_not_decisive() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ // No word of the question is in any sketch: sketch routing finds nothing.+ let q = "completely unrelated wording here";++ let none = rig.run(&args(q, None), &cat, None, &FakeBackend::new(true));+ assert_eq!(none.exit, EXIT_NO_ANSWER);+ assert!(+ none.stderr.contains(&NO_CORPUS_MESSAGE.to_string()),+ "{:?}",+ none.stderr+ );++ // With a hosted node that recognises it, routing becomes confident.+ let mut hosted = StubHosted::new();+ hosted.found = vec![("lz4".to_string(), 7.0)];+ let backend = FakeBackend::new(true);+ let out = rig.run(&args(q, None), &cat, Some(&hosted), &backend);+ assert_eq!(out.exit, EXIT_OK, "{:?}", out.stderr);+ assert_eq!(hosted.searches.get(), 1, "one federated search");+ assert_eq!(+ hosted.answers.get(),+ 0,+ "the pack was cached: no hosted answer"+ );+ assert_eq!(backend.calls(), ["ready:lz4", "answer_local:lz4"]);+ }++ // ── step 2: routing outcomes ────────────────────────────────────────────++ #[test]+ fn ambiguous_prints_one_candidate_per_line_and_exits_4() {+ let rig = Rig::new();+ // Two corpora that both hold every word of the question.+ let cat = catalogue(vec![+ entry(+ "alpha",+ 1,+ LicenceClass::AdaptWithAttribution,+ &["tokio", "runtime"],+ ),+ entry(+ "beta",+ 1,+ LicenceClass::AdaptWithAttribution,+ &["tokio", "runtime"],+ ),+ ]);+ let backend = FakeBackend::new(true);+ let out = rig.run(&args("tokio runtime", None), &cat, None, &backend);+ assert_eq!(out.exit, EXIT_NEEDS_DECISION, "{:?}", out);+ let lines: Vec<&str> = out.stdout.lines().collect();+ assert_eq!(lines.len(), 2, "{}", out.stdout);+ assert!(+ lines[0].starts_with("alpha — an agent working on alpha"),+ "{}",+ lines[0]+ );+ assert!(lines[1].starts_with("beta — "), "{}", lines[1]);+ assert!(out.stderr.iter().any(|l| l.contains("--corpus")));+ assert!(backend.calls().is_empty(), "nothing is answered or pulled");+ }++ #[test]+ fn no_corpus_exits_1_with_the_llms_txt_pointer() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1, LicenceClass::AdaptWithAttribution)]);+ let out = rig.run(+ &args("gardening tips for tomatoes", None),+ &cat,+ None,+ &FakeBackend::new(false),+ );+ assert_eq!(out.exit, EXIT_NO_ANSWER);+ assert_eq!(out.stderr, [NO_CORPUS_MESSAGE]);+ assert_eq!(+ NO_CORPUS_MESSAGE,+ "no hub corpus matches; see xerj.org/llms.txt"+ );+ }++ #[test]+ fn an_unknown_corpus_flag_is_a_usage_error_naming_the_catalogue_date() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1, LicenceClass::AdaptWithAttribution)]);+ let backend = FakeBackend::new(true);+ let out = rig.run(&args("q", Some("nope")), &cat, None, &backend);+ assert_eq!(out.exit, EXIT_USAGE);+ assert!(+ out.stderr[0].contains("unknown corpus 'nope'"),+ "{:?}",+ out.stderr+ );+ assert!(out.stderr[0].contains("2026-10-14"), "{:?}", out.stderr);+ assert!(backend.calls().is_empty());+ }++ // ── step 4: hosted answers now, the pull follows ────────────────────────++ #[test]+ fn hosted_answers_now_and_starts_the_pull_behind_it() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let hosted = StubHosted::new();+ let backend = FakeBackend::new(false);++ let out = rig.run(&args("q", Some("lz4")), &cat, Some(&hosted), &backend);+ assert_eq!(out.exit, EXIT_OK, "{:?}", out.stderr);+ assert_eq!(out.answer.unwrap().answered_from, AnsweredFrom::Hosted);+ assert!(+ out.stdout.contains("answered from: hosted hub"),+ "{}",+ out.stdout+ );+ assert_eq!(backend.calls(), ["ready:lz4", "pull_bg:lz4"]);+ assert!(+ out.stderr.iter().any(|l| l.contains("background")),+ "{:?}",+ out.stderr+ );+ assert_eq!(hosted.answers.get(), 1);+ }++ #[test]+ fn no_fetch_answers_hosted_without_starting_a_pull() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let backend = FakeBackend::new(false);+ let mut a = args("q", Some("lz4"));+ a.no_fetch = true;+ let out = rig.run(&a, &cat, Some(&StubHosted::new()), &backend);+ assert_eq!(out.exit, EXIT_OK);+ assert!(out.stdout.contains("answered from: hosted hub"));+ assert!(!backend.called("pull_"), "{:?}", backend.calls());+ }++ #[test]+ fn a_pack_over_the_limit_is_not_pulled_and_the_message_names_its_size() {+ let rig = Rig::new();+ let sixty_mib = 60 * 1024 * 1024;+ let cat = catalogue(vec![lz4(sixty_mib, LicenceClass::AdaptWithAttribution)]);+ let backend = FakeBackend::new(false);+ let out = rig.run(+ &args("q", Some("lz4")),+ &cat,+ Some(&StubHosted::new()),+ &backend,+ );+ assert_eq!(out.exit, EXIT_OK);+ assert!(!backend.called("pull_"), "{:?}", backend.calls());+ assert!(+ out.stdout.contains("run: xerj hub pull lz4 (60.0 MB)"),+ "{}",+ out.stdout+ );+ // The same hint is in the structured answer.+ let ans = out.answer.unwrap();+ assert!(ans+ .warnings+ .contains(&"run: xerj hub pull lz4 (60.0 MB)".to_string()));+ }++ #[test]+ fn an_excluded_corpus_is_never_pulled() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::Excluded)]);+ // Hosted answers, but no pull is started.+ let backend = FakeBackend::new(false);+ let out = rig.run(+ &args("q", Some("lz4")),+ &cat,+ Some(&StubHosted::new()),+ &backend,+ );+ assert_eq!(out.exit, EXIT_OK);+ assert!(!backend.called("pull_"), "{:?}", backend.calls());+ assert!(+ out.stdout.contains("excluded licence (MIT)"),+ "{}",+ out.stdout+ );++ // With no hosted answer to fall back on, there is nothing to say but no.+ let backend = FakeBackend::new(false);+ let out = rig.run(&args("q", Some("lz4")), &cat, None, &backend);+ assert_eq!(out.exit, EXIT_NO_ANSWER);+ assert!(!backend.called("pull_"), "{:?}", backend.calls());+ assert!(out.stderr[0].contains("never pulled"), "{:?}", out.stderr);+ }++ #[test]+ fn an_approach_only_corpus_is_pulled_and_every_answer_carries_the_warning() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::ApproachOnly)]);+ let want = "approach-only licence (GPL-3.0): read, do not copy";++ // Hosted answer: pulled behind it, warning on the answer and on the hit.+ let backend = FakeBackend::new(false);+ let out = rig.run(+ &args("q", Some("lz4")),+ &cat,+ Some(&StubHosted::new()),+ &backend,+ );+ assert!(backend.called("pull_bg:lz4"));+ assert!(+ out.stdout.contains(&format!("warning: {want}")),+ "{}",+ out.stdout+ );+ let ans = out.answer.unwrap();+ assert_eq!(ans.hits[0].licence_warning.as_deref(), Some(want));+ assert_eq!(ans.warnings.iter().filter(|w| *w == want).count(), 1);++ // Local answer: the same.+ let backend = FakeBackend::new(true);+ let out = rig.run(&args("q", Some("lz4")), &cat, None, &backend);+ assert!(+ out.stdout.contains(&format!("warning: {want}")),+ "{}",+ out.stdout+ );+ }++ #[test]+ fn a_failing_hosted_answer_falls_back_to_the_synchronous_pull() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let mut hosted = StubHosted::new();+ hosted.fail = true;+ let backend = FakeBackend::new(false);+ let out = rig.run(&args("q", Some("lz4")), &cat, Some(&hosted), &backend);+ assert_eq!(out.exit, EXIT_OK, "{:?}", out.stderr);+ assert!(out+ .stderr+ .iter()+ .any(|l| l.contains("hosted answer failed")));+ assert_eq!(+ backend.calls(),+ ["ready:lz4", "pull_now:lz4", "answer_local:lz4"]+ );+ assert_eq!(out.answer.unwrap().answered_from, AnsweredFrom::LocalPack);+ }++ // ── step 5: no pack, no hosted ──────────────────────────────────────────++ #[test]+ fn offline_with_no_pack_exits_1_naming_the_pack_to_pull() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(+ 12 * 1024 * 1024,+ LicenceClass::AdaptWithAttribution,+ )]);+ let backend = FakeBackend::new(false);+ let mut a = args("q", Some("lz4"));+ a.offline = true;+ // --offline: the hosted hub must not be contacted even though it is on.+ let out = rig.run(&a, &cat, Some(&NoNetwork), &backend);+ assert_eq!(out.exit, EXIT_NO_ANSWER);+ assert!(out.stdout.is_empty());+ let msg = out.stderr.join("\n");+ assert!(msg.contains("--offline"), "{msg}");+ assert!(msg.contains("run: xerj hub pull lz4 (12.0 MB)"), "{msg}");+ assert_eq!(backend.calls(), ["ready:lz4"], "no pull, no answer");+ }++ #[test]+ fn offline_still_answers_from_a_cached_pack() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let backend = FakeBackend::new(true);+ let mut a = args("q", Some("lz4"));+ a.offline = true;+ let out = rig.run(&a, &cat, Some(&NoNetwork), &backend);+ assert_eq!(out.exit, EXIT_OK);+ assert_eq!(backend.calls(), ["ready:lz4", "answer_local:lz4"]);+ }++ #[test]+ fn with_no_hosted_the_first_question_pulls_synchronously_then_answers_locally() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let backend = FakeBackend::new(false);+ let out = rig.run(&args("q", Some("lz4")), &cat, None, &backend);+ assert_eq!(out.exit, EXIT_OK, "{:?}", out.stderr);+ assert_eq!(+ backend.calls(),+ ["ready:lz4", "pull_now:lz4", "answer_local:lz4"]+ );+ assert!(out.stdout.contains("answered from: local pack"));+ }++ #[test]+ fn a_failed_pull_keeps_its_exit_code() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ for code in [1, 2, 3] {+ let backend = FakeBackend::new(false);+ *backend.pull_fail.borrow_mut() = Some(PullFailure {+ code,+ message: format!("pull failed {code}"),+ });+ let out = rig.run(&args("q", Some("lz4")), &cat, None, &backend);+ assert_eq!(out.exit, code);+ assert_eq!(+ out.stderr.last().map(String::as_str),+ Some(&*format!("pull failed {code}"))+ );+ assert!(!backend.called("answer_local"), "{:?}", backend.calls());+ }+ }++ #[test]+ fn the_policy_also_binds_the_synchronous_pull() {+ let rig = Rig::new();+ // Over the limit: no pull, the message names the size and the command.+ let cat = catalogue(vec![lz4(+ 80 * 1024 * 1024,+ LicenceClass::AdaptWithAttribution,+ )]);+ let backend = FakeBackend::new(false);+ let out = rig.run(&args("q", Some("lz4")), &cat, None, &backend);+ assert_eq!(out.exit, EXIT_NO_ANSWER);+ assert!(!backend.called("pull_now"), "{:?}", backend.calls());+ assert!(out.stderr[0].contains("80.0 MB"), "{:?}", out.stderr);+ assert!(+ out.stderr[0].contains("run: xerj hub pull lz4"),+ "{:?}",+ out.stderr+ );++ // --no-fetch means no download at all, even with nothing else to answer from.+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let backend = FakeBackend::new(false);+ let mut a = args("q", Some("lz4"));+ a.no_fetch = true;+ let out = rig.run(&a, &cat, None, &backend);+ assert_eq!(out.exit, EXIT_NO_ANSWER);+ assert!(out.stderr[0].contains("--no-fetch"), "{:?}", out.stderr);+ assert!(!backend.called("pull_now"));+ }++ #[test]+ fn a_node_that_cannot_be_reached_falls_through_to_the_hosted_answer() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let mut backend = FakeBackend::new(true);+ backend.ready_err = Some("no node answers".to_string());+ let out = rig.run(+ &args("q", Some("lz4")),+ &cat,+ Some(&StubHosted::new()),+ &backend,+ );+ assert_eq!(out.exit, EXIT_OK);+ assert!(out.stderr[0].contains("local pack unavailable: no node answers"));+ assert!(out.stdout.contains("answered from: hosted hub"));+ }++ // ── the hosted switch ───────────────────────────────────────────────────++ #[test]+ fn hosted_off_never_sends_the_question_anywhere() {+ let rig = Rig::new();+ let cat = catalogue(with_fillers(vec![lz4(+ 1024,+ LicenceClass::AdaptWithAttribution,+ )]));+ let backend = FakeBackend::new(false);+ let mut a = args("lz4 block decoder", None);+ a.hosted = HostedSwitch::Off;+ // NoNetwork would panic on the routing search or on an answer.+ let out = rig.run(&a, &cat, Some(&NoNetwork), &backend);+ assert_eq!(out.exit, EXIT_OK, "{:?}", out.stderr);+ assert_eq!(+ backend.calls(),+ ["ready:lz4", "pull_now:lz4", "answer_local:lz4"]+ );+ }++ #[test]+ fn auto_is_off_inside_a_repo_with_a_one_time_notice_and_on_outside_one() {+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);++ // Inside a git work tree: off (NoNetwork would panic), notice once.+ let rig = Rig::new().in_git_repo();+ let first = rig.run(+ &args("q", Some("lz4")),+ &cat,+ Some(&NoNetwork),+ &FakeBackend::new(true),+ );+ assert_eq!(first.exit, EXIT_OK);+ assert_eq!(first.stderr, [HOSTED_OFF_NOTICE]);+ let second = rig.run(+ &args("q", Some("lz4")),+ &cat,+ Some(&NoNetwork),+ &FakeBackend::new(true),+ );+ assert!(second.stderr.is_empty(), "{:?}", second.stderr);++ // Outside one: on, no notice.+ let rig = Rig::new();+ let hosted = StubHosted::new();+ let out = rig.run(+ &args("q", Some("lz4")),+ &cat,+ Some(&hosted),+ &FakeBackend::new(false),+ );+ assert_eq!(hosted.answers.get(), 1);+ assert!(!out.stderr.iter().any(|l| l == HOSTED_OFF_NOTICE));+ }++ #[test]+ fn config_overrides_the_repo_rule_in_both_directions() {+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);++ // hosted = "on" inside a repo: used, and no notice.+ let rig = Rig::new().in_git_repo();+ set_config_hosted(rig.hub.path(), Some("on")).unwrap();+ let hosted = StubHosted::new();+ let out = rig.run(+ &args("q", Some("lz4")),+ &cat,+ Some(&hosted),+ &FakeBackend::new(false),+ );+ assert_eq!(hosted.answers.get(), 1);+ assert!(out.stderr.iter().all(|l| l != HOSTED_OFF_NOTICE));++ // hosted = "off" outside a repo: off.+ let rig = Rig::new();+ set_config_hosted(rig.hub.path(), Some("off")).unwrap();+ let backend = FakeBackend::new(false);+ let out = rig.run(&args("q", Some("lz4")), &cat, Some(&NoNetwork), &backend);+ assert_eq!(out.exit, EXIT_OK);+ assert!(backend.called("pull_now"));+ }++ #[test]+ fn an_explicit_hosted_on_without_a_client_says_so_and_still_answers() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let mut a = args("q", Some("lz4"));+ a.hosted = HostedSwitch::On;+ let out = rig.run(&a, &cat, None, &FakeBackend::new(true));+ assert_eq!(out.exit, EXIT_OK);+ assert!(+ out.stderr.iter().any(|l| l.contains("no hosted client")),+ "{:?}",+ out.stderr+ );+ }++ #[test]+ fn set_config_hosted_keeps_other_keys_and_auto_removes_the_key() {+ let hub = tempfile::tempdir().unwrap();+ fs::write(hub.path().join("config.toml"), "other = 3\n").unwrap();+ set_config_hosted(hub.path(), Some("on")).unwrap();+ assert_eq!(config_hosted(hub.path()).as_deref(), Some("on"));+ let text = fs::read_to_string(hub.path().join("config.toml")).unwrap();+ assert!(text.contains("other = 3"), "{text}");+ set_config_hosted(hub.path(), None).unwrap();+ assert_eq!(config_hosted(hub.path()), None);+ let text = fs::read_to_string(hub.path().join("config.toml")).unwrap();+ assert!(text.contains("other = 3"), "{text}");+ }++ // ── output ──────────────────────────────────────────────────────────────++ #[test]+ fn json_prints_the_answer_and_the_rendering_otherwise() {+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let mut a = args("q", Some("lz4"));+ a.json = true;+ let out = rig.run(&a, &cat, None, &FakeBackend::new(true));+ let parsed: Answer = serde_json::from_str(&out.stdout).expect("stdout is the Answer JSON");+ assert_eq!(parsed.corpus, "lz4");+ assert_eq!(parsed.answered_from, AnsweredFrom::LocalPack);++ let out = rig.run(&args("q", Some("lz4")), &cat, None, &FakeBackend::new(true));+ assert!(out+ .stdout+ .starts_with("lz4 pin 2026-10-01 (13 days old) MIT lz4/lib/lz4.h:100"));+ }++ #[test]+ fn an_answer_with_no_hits_exits_1_but_is_still_printed() {+ struct Empty;+ impl Backend for Empty {+ fn local_pack_ready(&self, _: &str) -> Result<bool, String> {+ Ok(true)+ }+ fn answer_local(&self, req: &Request<'_>) -> Result<Answer, String> {+ let mut a = canned(req.slug, AnsweredFrom::LocalPack, req.mode);+ a.hits.clear();+ Ok(a)+ }+ fn pull_now(&self, _: &CatalogueEntry) -> Result<(), PullFailure> {+ unreachable!()+ }+ fn pull_in_background(&self, _: &str) -> Result<BackgroundPull, String> {+ unreachable!()+ }+ }+ let rig = Rig::new();+ let cat = catalogue(vec![lz4(1024, LicenceClass::AdaptWithAttribution)]);+ let out = rig.run(&args("q", Some("lz4")), &cat, None, &Empty);+ assert_eq!(out.exit, EXIT_NO_ANSWER);+ assert!(+ out.stdout.contains("no match in lz4 (pin 2026-10-01)"),+ "{}",+ out.stdout+ );+ }++ #[test]+ fn max_tokens_drops_trailing_hits_then_shortens_the_last() {+ let mut a = canned("lz4", AnsweredFrom::LocalPack, AnswerMode::How);+ let hit = a.hits[0].clone();+ a.hits = (0..4)+ .map(|i| Hit {+ line: 100 + i,+ text: "x".repeat(200),+ ..hit.clone()+ })+ .collect();++ // Generous budget: nothing changes.+ let mut big = a.clone();+ cap_tokens(&mut big, 10_000);+ assert_eq!(big.hits.len(), 4);+ assert!(!big.truncated);++ // Room for about two hits: the tail is dropped and the answer says so.+ let mut mid = a.clone();+ cap_tokens(&mut mid, 120);+ assert!(+ mid.hits.len() < 4 && !mid.hits.is_empty(),+ "{}",+ mid.hits.len()+ );+ assert!(mid.truncated);+ assert!(render(&mid).chars().count() <= 120 * 4);++ // A budget smaller than one hit: the first hit survives, shortened.+ let mut tiny = a.clone();+ cap_tokens(&mut tiny, 60);+ assert_eq!(tiny.hits.len(), 1);+ assert!(tiny.hits[0].text.ends_with('…'));+ assert!(tiny.hits[0].text.chars().count() < 200);+ assert!(tiny.truncated);+ assert!(render(&tiny).chars().count() <= 60 * 4);+ }++ // ── the disk budget ─────────────────────────────────────────────────────++ /// A fake extracted pack: `pack.json` plus a payload of `payload` bytes.+ fn fake_pack(hub: &Path, slug: &str, pin8: &str, payload: usize, used_secs_ago: u64) {+ let dir = hub.join("packs").join(slug).join(pin8);+ fs::create_dir_all(dir.join("snapshot")).unwrap();+ fs::write(dir.join("pack.json"), "{}").unwrap();+ fs::write(dir.join("snapshot").join("data"), vec![7u8; payload]).unwrap();+ let used = dir.join("last_used");+ fs::write(&used, "x").unwrap();+ let when = SystemTime::now() - Duration::from_secs(used_secs_ago);+ fs::File::options()+ .write(true)+ .open(&used)+ .unwrap()+ .set_modified(when)+ .unwrap();+ }++ fn slugs_on_disk(hub: &Path) -> Vec<String> {+ pack_dirs(hub).into_iter().map(|p| p.slug).collect()+ }++ #[test]+ fn the_least_recently_used_pack_is_evicted_first() {+ let hub = tempfile::tempdir().unwrap();+ // Three packs of ~1000 bytes; "old" was used longest ago, "new" just now.+ fake_pack(hub.path(), "mid", "aaaa0001", 1000, 500);+ fake_pack(hub.path(), "old", "aaaa0002", 1000, 9000);+ fake_pack(hub.path(), "new", "aaaa0003", 1000, 5);++ // An incoming pack that needs one slot's worth of room.+ let total = dir_size(&hub.path().join("packs"));+ let budget = total; // already full: any incoming byte is over+ let evicted = evict_to_budget(hub.path(), "incoming", 900, budget);+ let names: Vec<_> = evicted.iter().map(|c| c.pack.slug.as_str()).collect();+ assert_eq!(names, ["old"], "only the oldest goes: that is enough");+ assert_eq!(slugs_on_disk(hub.path()), ["mid", "new"]);+ // The emptied slug directory went with it.+ assert!(!hub.path().join("packs").join("old").exists());++ // A second incoming pack the size of "mid": the next-oldest goes too, and+ // the newest stays.+ let total = dir_size(&hub.path().join("packs"));+ let mid_bytes = list_cached_packs(hub.path())+ .into_iter()+ .find(|c| c.pack.slug == "mid")+ .map(|c| c.bytes)+ .unwrap();+ let evicted = evict_to_budget(hub.path(), "incoming", mid_bytes, total);+ let names: Vec<_> = evicted.iter().map(|c| c.pack.slug.as_str()).collect();+ assert_eq!(names, ["mid"]);+ assert_eq!(slugs_on_disk(hub.path()), ["new"]);+ }++ #[test]+ fn eviction_never_removes_the_pack_being_pulled_and_stops_when_it_fits() {+ let hub = tempfile::tempdir().unwrap();+ // The slug being pulled is the OLDEST of all.+ fake_pack(hub.path(), "target", "bbbb0001", 1000, 99_999);+ fake_pack(hub.path(), "other", "bbbb0002", 1000, 100);+ let total = dir_size(&hub.path().join("packs"));++ // Over budget: "other" must go, "target" must stay although it is oldest.+ let evicted = evict_to_budget(hub.path(), "target", 5000, total);+ assert_eq!(evicted.len(), 1);+ assert_eq!(evicted[0].pack.slug, "other");+ assert_eq!(slugs_on_disk(hub.path()), ["target"]);++ // Even when nothing else is left and the budget is still exceeded, the+ // pull proceeds and the target survives.+ let evicted = evict_to_budget(hub.path(), "target", u64::MAX / 2, 10);+ assert!(evicted.is_empty());+ assert_eq!(slugs_on_disk(hub.path()), ["target"]);+ }++ #[test]+ fn under_budget_evicts_nothing() {+ let hub = tempfile::tempdir().unwrap();+ fake_pack(hub.path(), "a", "cccc0001", 1000, 1);+ fake_pack(hub.path(), "b", "cccc0002", 1000, 2);+ let evicted = evict_to_budget(hub.path(), "x", 100, 1_000_000);+ assert!(evicted.is_empty());+ assert_eq!(slugs_on_disk(hub.path()), ["a", "b"]);+ }++ #[test]+ fn the_real_budget_is_four_gib_and_a_pack_dir_needs_pack_json_to_count() {+ assert_eq!(xerj_common::hub::DISK_BUDGET_BYTES, 4 * 1024 * 1024 * 1024);+ let hub = tempfile::tempdir().unwrap();+ fake_pack(hub.path(), "real", "dddd0001", 10, 1);+ // A half-extracted dir, a temp dir and a lock dir are not packs.+ for junk in ["dddd0002", "dddd0003.tmp", "dddd0003.lock"] {+ fs::create_dir_all(hub.path().join("packs").join("real").join(junk)).unwrap();+ }+ assert_eq!(pack_dirs(hub.path()).len(), 1);+ assert_eq!(cached_slugs(hub.path()), ["real"]);+ }++ #[test]+ fn answering_touches_last_used_so_the_pack_becomes_the_newest() {+ let hub = tempfile::tempdir().unwrap();+ fake_pack(hub.path(), "a", "eeee0001", 10, 9000);+ fake_pack(hub.path(), "b", "eeee0002", 10, 500);+ let before = pack_dirs(hub.path());+ assert!(+ before[0].last_used < before[1].last_used,+ "a is older than b"+ );+ touch_last_used(hub.path(), "a");+ let after = pack_dirs(hub.path());+ assert!(+ after[0].last_used > after[1].last_used,+ "a is now newer than b"+ );+ }++ // ── converting a search response ────────────────────────────────────────++ fn meta() -> AnswerMeta {+ AnswerMeta {+ corpus: "lz4".to_string(),+ pin_sha: "0".repeat(40),+ pin_date: "2026-10-01".to_string(),+ age_days: 13,+ licence_spdx: "BSD-2-Clause".to_string(),+ licence_class: LicenceClass::AdaptWithAttribution,+ answered_from: AnsweredFrom::LocalPack,+ }+ }++ fn search_response() -> Value {+ json!({"hits": {"hits": [+ {"_index": "xc-lz4-b1-docs", "_score": 7.5, "_source": {+ "ax_path": "lz4/lib/lz4.c",+ "body": "int other(void) {\n return 0;\n}\nint LZ4_compress(const char* s) {\n return 1;\n}\n",+ "symbols": [+ {"name": "other", "kind": "function", "line": 1},+ {"name": "LZ4_compress", "kind": "function", "line": 4}+ ]+ }},+ {"_index": "xc-lz4-b1-docs", "_score": 5.0, "_source": {+ "ax_path": "lz4/lib/lz4.c", "line": 40,+ "body": "second record of the same file about compress"+ }},+ {"_index": "xc-lz4-b1-docs", "_score": 3.0, "_source": {+ "ax_path": "gplrepo/src/x.c",+ "body": "static int compress_gpl(void) { return 2; }"+ }}+ ]}})+ }++ fn licences() -> HashMap<String, String> {+ HashMap::from([+ ("lz4".to_string(), "BSD-2-Clause".to_string()),+ ("gplrepo".to_string(), "GPL-3.0".to_string()),+ ])+ }++ #[test]+ fn a_how_answer_picks_the_matching_definition_with_its_real_line() {+ let a = answer_from_search(+ meta(),+ &search_response(),+ "how does LZ4_compress work",+ AnswerMode::How,+ &licences(),+ );+ assert_eq!(a.mode, AnswerMode::How);+ assert_eq!(a.hits.len(), 3);+ let h = &a.hits[0];+ assert_eq!(h.file, "lz4/lib/lz4.c");+ assert_eq!(h.kind, "function");+ assert_eq!(h.line, 4, "the definition's own start line");+ assert!(h.text.contains("LZ4_compress"), "{}", h.text);+ assert!((h.score - 7.5).abs() < 1e-6);+ assert!(h.licence_warning.is_none());+ // No symbols: a window, the record's own line number.+ assert_eq!(a.hits[1].kind, "passage");+ assert_eq!(a.hits[1].line, 40);+ // A record with no line at all reports 0 rather than inventing one.+ assert_eq!(a.hits[2].line, 0);+ // A restricted licence tags its hit.+ assert_eq!(+ a.hits[2].licence_warning.as_deref(),+ Some("approach-only licence (GPL-3.0): read, do not copy")+ );+ }++ #[test]+ fn a_files_answer_lists_each_file_once_with_its_licence() {+ let a = answer_from_search(+ meta(),+ &search_response(),+ "compress",+ AnswerMode::Files,+ &licences(),+ );+ assert_eq!(a.mode, AnswerMode::Files);+ let files: Vec<&str> = a.hits.iter().map(|h| h.file.as_str()).collect();+ assert_eq!(+ files,+ ["lz4/lib/lz4.c", "gplrepo/src/x.c"],+ "deduplicated, rank order"+ );+ assert_eq!(a.hits[0].kind, "file");+ assert_eq!(a.hits[0].text, "licence: BSD-2-Clause");+ assert_eq!(a.hits[1].text, "licence: GPL-3.0");+ assert!(a.hits[1].licence_warning.is_some());+ }++ #[test]+ fn locators_and_labels_parse() {+ assert_eq!(split_locator("a/b.c:12"), ("a/b.c".to_string(), 12));+ assert_eq!(split_locator("a/b.c"), ("a/b.c".to_string(), 0));+ assert_eq!(split_locator("a/b:c"), ("a/b:c".to_string(), 0));+ assert_eq!(+ parse_symbol_label("function foo @ line 12"),+ ("function".to_string(), Some(12))+ );+ assert_eq!(+ parse_symbol_label("struct bar"),+ ("struct".to_string(), None)+ );+ }++ #[test]+ fn a_raw_json_record_without_body_uses_its_longest_string_field() {+ let src = json!({"id": "x", "summary": "the longest field wins here", "n": 3});+ assert_eq!(longest_string_field(&src), "the longest field wins here");+ assert_eq!(longest_string_field(&json!({"n": 1})), "");+ }++ // ── policy helpers ──────────────────────────────────────────────────────++ #[test]+ fn pull_policy_orders_excluded_before_size() {+ let mib = 1024 * 1024;+ assert_eq!(+ pull_policy(&lz4(+ AUTO_PULL_MAX_BYTES,+ LicenceClass::AdaptWithAttribution+ )),+ PullPolicy::Allow,+ "exactly at the limit is allowed"+ );+ assert_eq!(+ pull_policy(&lz4(+ AUTO_PULL_MAX_BYTES + 1,+ LicenceClass::AdaptWithAttribution+ )),+ PullPolicy::TooLarge+ );+ assert_eq!(+ pull_policy(&lz4(1, LicenceClass::ApproachOnly)),+ PullPolicy::Allow,+ "approach-only is pulled (with a warning), not refused"+ );+ assert_eq!(+ pull_policy(&lz4(500 * mib, LicenceClass::Excluded)),+ PullPolicy::Excluded,+ "excluded wins over size"+ );+ assert_eq!(megabytes(52_428_800), "50.0");+ assert_eq!(megabytes(1_572_864), "1.5");+ }++ #[test]+ fn approach_only_warning_names_the_licence_or_says_unknown() {+ assert_eq!(+ approach_only_warning("AGPL-3.0"),+ "approach-only licence (AGPL-3.0): read, do not copy"+ );+ assert_eq!(+ approach_only_warning(""),+ "approach-only licence (unknown): read, do not copy"+ );+ }+}diff --git a/engine/crates/xerj-hub/src/cli.rs b/engine/crates/xerj-hub/src/cli.rsindex 6f4d5022f..a704330db 100644--- a/engine/crates/xerj-hub/src/cli.rs+++ b/engine/crates/xerj-hub/src/cli.rs@@ -1,11 +1,15 @@-//! Corpus Hub v2 — the `xerj hub …` subcommands and the `--def` code path-//! (plan §2 builder + client, §3 `def`).+//! Corpus Hub v2 — the `xerj hub …` subcommands, `xerj ask`, and the `--def` code+//! path (plan §2 builder + client, §3 `def`, §5 `ask` / node, §8 `status`).//!-//! Three entry points, all returning a process exit code:+//! Entry points, all returning a process exit code://!//! - [`run_hub_cli`] dispatches `xerj hub build-pack` (CI builds a signed,-//! pre-indexed pack) and `xerj hub pull` (a client downloads, verifies and-//! restores one). Future hub subcommands (status, sync, …) are plan §5/§8.+//! pre-indexed pack), `pull` (a client downloads, verifies and restores one),+//! `status`, `stop` (stop the automatic node) and `config hosted on|off`.+//! `sync`, `update` and `rm` are plan §8 and are not here.+//! - [`run_ask`] is `xerj ask "<question>"` (and the bare `xerj "<question>"`+//! form, see [`is_bare_question`]): the flags are parsed here, the answer flow+//! itself is [`crate::ask`].//! - [`run_code`] is the `xerj code` front door: `--def <SYMBOL>` answers from//! the corpus's `-defs` index via [`crate::def::lookup`] and the shared//! renderer; everything else delegates unchanged to the existing@@ -28,14 +32,16 @@ use serde_json::{json, Value};use xerj_common::hub::{render, Answer, LicenceClass, PackIndex, PackManifest, PackSource, AUTO_PULL_MAX_BYTES,- DEFAULT_HUB_URL, HUB_PUBKEY_HEX, PACK_FORMAT_VERSION,+ DEFAULT_HUB_URL, DISK_BUDGET_BYTES, HUB_PUBKEY_HEX, PACK_FORMAT_VERSION,};use xerj_common::xccode::manifest::read_hub_manifest;use xerj_common::xccode::state;use xerj_common::xccode::XcHttp;+use crate::ask::{self, AskArgs, HostedSwitch, ModeRequest};use crate::catalogue::{build_catalogue, load_catalogue, CorpusBuildInput};-use crate::node::{resolve_url, HubNode};+use crate::lockfile;+use crate::node::{self, resolve_url, HubNode, NodeLauncher, StopOutcome};use crate::pack;const HUB_USAGE: &str = "usage: xerj hub <command>@@ -44,6 +50,15 @@ const HUB_USAGE: &str = "usage: xerj hub <command>Build a signed, pre-indexed pack from a registry manifest (CI).pull <slug> [--quiet] [--url <node>]Download, verify and restore a pack, then `xerj code <slug>` works.+ status+ Cached packs, the automatic node, the hosted setting, catalogue age.+ stop+ Stop the automatic node `xerj ask` started (only that process).+ config hosted on|off|auto+ Whether questions may be sent to the hosted hub (default auto: off+ inside a git repository, on elsewhere).++Ask a question with `xerj ask \"<question>\"` (see `xerj ask --help`).Packs are pulled on demand and never cloned or indexed on the user's machine(docs/CORPUS_HUB_V2_IMPLEMENTATION.md).";@@ -58,6 +73,9 @@ pub fn run_hub_cli(args: &[String]) -> i32 {match sub.as_str() {"build-pack" => run_build_pack(rest),"pull" => run_pull(rest),+ "status" => run_status(rest),+ "stop" => run_stop(rest),+ "config" => run_config(rest),"-h" | "--help" | "help" => {println!("{HUB_USAGE}");0@@ -460,11 +478,24 @@ fn run_pull(args: &[String]) -> i32 {eprintln!("xerj hub pull: <slug> is required\n\n{HUB_USAGE}");return 2;};- // DEFER: the client's localhost:9200 default is mandated by plan §2 ("Node- // for the client ... localhost:9200"); step 5's auto-spawned ~/.xerj/hub/node- // will own client node resolution. Unlike build-pack, pull's --url is optional.- let url = resolve_url(url.as_deref());- match pull(&slug, &url, quiet) {+ // Unlike build-pack, pull's --url is optional: `--url` > `XERJ_URL` >+ // `localhost:9200`, and when none of those answers, the automatic node+ // (plan §5) is started or reused.+ let launcher = match NodeLauncher::current() {+ Ok(l) => l,+ Err(e) => {+ eprintln!("xerj hub pull: {e:#}");+ return 1;+ }+ };+ let resolved = match node::ensure_node(&hub_home(), url.as_deref(), &launcher) {+ Ok(r) => r,+ Err(e) => {+ eprintln!("xerj hub pull: {e:#}");+ return 1;+ }+ };+ match pull(&slug, &resolved.url, resolved.managed, quiet) {Ok(Pulled::Already) => {if !quiet {println!("{slug}: already present");@@ -484,7 +515,7 @@ fn run_pull(args: &[String]) -> i32 {}}-enum Pulled {+pub(crate) enum Pulled {Already,Done(String),}@@ -496,9 +527,9 @@ enum Pulled {/// a pull already in progress, the download, the restore, or writing the/// ledger). Keeping 2 and 3 exact lets a caller tell "wrong name" and "untrusted/// bytes" apart from "the machine or network got in the way".-struct PullError {- code: i32,- message: String,+pub(crate) struct PullError {+ pub(crate) code: i32,+ pub(crate) message: String,}impl PullError {fn new(code: i32, message: impl Into<String>) -> Self {@@ -562,7 +593,10 @@ impl Drop for PullLock {}}-fn pull(slug: &str, url: &str, _quiet: bool) -> Result<Pulled, PullError> {+/// Pull `slug` into the node at `url`. `managed` is true for the automatic node+/// (started `--insecure`, so no credential is sent); `quiet` suppresses the+/// eviction notices.+pub(crate) fn pull(slug: &str, url: &str, managed: bool, quiet: bool) -> Result<Pulled, PullError> {// 0. Validate the slug BEFORE it is ever joined into a filesystem path. The// catalogue is accepted UNVERIFIED when no hub pubkey is configured, so a// hostile `slug` ("../../etc") must be rejected here, ahead of every fs op.@@ -622,10 +656,30 @@ fn pull(slug: &str, url: &str, _quiet: bool) -> Result<Pulled, PullError> {// key is a verification failure that must not leave a `.tmp` to re-fetch.let pubkey = hub_pubkey().map_err(|e| PullError::new(E_VERIFY, format!("{e:#}")))?;+ // 3b. Disk budget (plan §5 step 6): make room BEFORE any download, evicting+ // least-recently-used packs — never this slug's. Pack directories only;+ // the node's restored indices are not touched.+ for c in ask::evict_to_budget(&hub_home, slug, entry.pack_bytes, DISK_BUDGET_BYTES) {+ if !quiet {+ eprintln!(+ "xerj hub: evicted '{}' {} ({} MB, least recently used) to stay under the {} MB \+ disk budget",+ c.pack.slug,+ c.pack.pin8,+ ask::megabytes(c.bytes),+ ask::megabytes(DISK_BUDGET_BYTES)+ );+ }+ }+// 4. The node (infra if unreachable).let root = xerj_autoindex::xc::code_root();- let node = HubNode::connect_with_key(url, None)- .map_err(|e| PullError::new(E_INFRA, format!("connect {url}: {e:#}")))?;+ let node = if managed {+ HubNode::connect_insecure(url)+ } else {+ HubNode::connect_with_key(url, None)+ }+ .map_err(|e| PullError::new(E_INFRA, format!("connect {url}: {e:#}")))?;// 5. Take the per-pin lock so a concurrent pull does not race the tmp/extract// dirs; released on every return below (Drop).@@ -732,7 +786,7 @@ fn pull(slug: &str, url: &str, _quiet: bool) -> Result<Pulled, PullError> {/// `..`, no `/`, non-empty, no leading/trailing separator. This is what stops a/// catalogue (which may be unverified) from steering a pull's writes out of/// `~/.xerj/hub/packs/<slug>/`.-fn valid_slug(s: &str) -> bool {+pub(crate) fn valid_slug(s: &str) -> bool {let b = s.as_bytes();if b.is_empty() {return false;@@ -779,6 +833,262 @@ fn fetch_to(url: &str, dest: &Path, max_bytes: u64) -> Result<u64> {Ok(buf.len() as u64)}+// ── xerj hub status / stop / config ──────────────────────────────────────────++/// `xerj hub status`: cached packs, the node, the hosted setting, last sync and+/// catalogue age — plan §8's report, one line per item. A node record whose+/// process is gone is cleared first so it is not reported as running.+fn run_status(args: &[String]) -> i32 {+ if let Some(code) = no_extra_args("status", args) {+ return code;+ }+ status_in(&hub_home())+}++fn status_in(home: &Path) -> i32 {+ node::reap_stale(home);+ println!(+ "{}",+ lockfile::render_status(&lockfile::gather_status(home))+ );+ 0+}++/// `xerj hub stop`: stop the node `xerj ask` started — exactly the PID recorded+/// in `~/.xerj/hub/node.pid`, after confirming it is still that node.+fn run_stop(args: &[String]) -> i32 {+ if let Some(code) = no_extra_args("stop", args) {+ return code;+ }+ stop_in(&hub_home())+}++fn stop_in(home: &Path) -> i32 {+ match node::stop_node(home) {+ Ok(StopOutcome::NothingRecorded) => {+ println!("no hub node was started by xerj; nothing to stop");+ 0+ }+ Ok(StopOutcome::StaleRemoved { pid }) => {+ println!("the recorded hub node (pid {pid}) is no longer running; removed its record");+ 0+ }+ Ok(StopOutcome::Stopped { pid }) => {+ println!("stopped the hub node (pid {pid})");+ 0+ }+ Err(e) => {+ eprintln!("xerj hub stop: {e:#}");+ 1+ }+ }+}++/// `-h` prints the usage; any other argument to a no-argument command is a+/// usage error. `None` means "carry on".+fn no_extra_args(cmd: &str, args: &[String]) -> Option<i32> {+ match args.first().map(String::as_str) {+ None => None,+ Some("-h" | "--help") => {+ println!("{HUB_USAGE}");+ Some(0)+ }+ Some(other) => {+ eprintln!("xerj hub {cmd}: unexpected argument '{other}'\n\n{HUB_USAGE}");+ Some(2)+ }+ }+}++/// `xerj hub config hosted on|off|auto`.+fn run_config(args: &[String]) -> i32 {+ config_in(&hub_home(), args)+}++fn config_in(home: &Path, args: &[String]) -> i32 {+ let usage = |msg: &str| {+ eprintln!("xerj hub config: {msg}\n\n{HUB_USAGE}");+ 2+ };+ let value = match args {+ [key, value] if key == "hosted" => value.as_str(),+ [first, ..] if first == "-h" || first == "--help" => {+ println!("{HUB_USAGE}");+ return 0;+ }+ _ => return usage("expected `hosted on|off|auto`"),+ };+ let stored = match value {+ "on" | "off" => Some(value),+ "auto" => None,+ other => return usage(&format!("hosted must be on, off or auto, not '{other}'")),+ };+ match ask::set_config_hosted(home, stored) {+ Ok(()) => {+ println!("hosted = {value}");+ 0+ }+ Err(e) => {+ eprintln!("xerj hub config: {e:#}");+ 1+ }+ }+}++// ── xerj ask ─────────────────────────────────────────────────────────────────++const ASK_USAGE: &str = "usage: xerj ask \"<question>\" [options]+ xerj \"<question>\" (the same, when the question contains a space)++Answers from a hub corpus - pulling its pre-indexed pack the first time - and+always names the corpus, the pinned revision and the licence.++options:+ --corpus <slug> skip routing and ask this corpus+ --def <SYMBOL> exact definition lookup (file:line, declaration, value)+ --how a how-does-it-work question (ranked passages)+ --files list the relevant files with their licences+ --offline never touch the network (cached catalogue and packs only)+ --no-fetch do not download a pack for next time+ --hosted on|off|auto may the question be sent to the hosted hub? (default+ auto: off inside a git repository, on elsewhere)+ --max-tokens N cap the answer at about N tokens+ --json print the answer as JSON+ --url <node> the node that holds the packs (default: XERJ_URL, then+ localhost:9200, else a local node is started)++exit codes: 0 answered; 1 no match or nothing to answer from; 2 usage error or+unknown corpus; 3 a pack failed verification; 4 more than one corpus matches+(candidates are printed; re-run with --corpus <slug>)";++/// The first argument of every real `xerj` subcommand.+pub const KNOWN_SUBCOMMANDS: &[&str] = &[+ "index",+ "search",+ "def",+ "init",+ "gain",+ "autoindex",+ "brain",+ "share",+ "feedback",+ "code",+ "corpus",+ "hub",+ "ask",+ "mcp",+];++/// True for `xerj "<question>"`: `argv[1]` is not a known subcommand, does not+/// start with `-`, and contains a space (plan §5).+pub fn is_bare_question(argv1: Option<&str>) -> bool {+ match argv1 {+ Some(a) => !KNOWN_SUBCOMMANDS.contains(&a) && !a.starts_with('-') && a.contains(' '),+ None => false,+ }+}++/// `xerj ask "<question>" [options]` — also the target of the bare+/// `xerj "<question>"` form, whose arguments are the same minus the word `ask`.+pub fn run_ask(args: &[String]) -> i32 {+ let parsed = match parse_ask_args(args) {+ Ok(Some(a)) => a,+ Ok(None) => {+ println!("{ASK_USAGE}");+ return 0;+ }+ Err(msg) => {+ eprintln!("xerj ask: {msg}\n\n{ASK_USAGE}");+ return ask::EXIT_USAGE;+ }+ };+ let outcome = ask::run_ask(&parsed, None);+ for line in &outcome.stderr {+ eprintln!("{line}");+ }+ if !outcome.stdout.is_empty() {+ println!("{}", outcome.stdout);+ }+ outcome.exit+}++/// Parse `xerj ask`'s arguments. `Ok(None)` is `--help`. Words that are not+/// flags (or flag values) are the question, joined with spaces — so an+/// unquoted `xerj ask how does lz4 work` still asks the whole sentence.+pub fn parse_ask_args(args: &[String]) -> Result<Option<AskArgs>, String> {+ let mut a = AskArgs::new(String::new());+ let mut words: Vec<&str> = Vec::new();+ let mut mode_flags = 0usize;+ let mut i = 0;+ while i < args.len() {+ let flag = args[i].as_str();+ let mut value = |name: &str| -> Result<String, String> {+ i += 1;+ args.get(i)+ .cloned()+ .ok_or_else(|| format!("{name} needs a value"))+ };+ match flag {+ "-h" | "--help" => return Ok(None),+ "--corpus" => a.corpus = Some(value("--corpus")?),+ "--def" => {+ a.mode = ModeRequest::Def(value("--def")?);+ mode_flags += 1;+ }+ "--how" => {+ a.mode = ModeRequest::How;+ mode_flags += 1;+ }+ "--files" => {+ a.mode = ModeRequest::Files;+ mode_flags += 1;+ }+ "--offline" => a.offline = true,+ "--no-fetch" => a.no_fetch = true,+ "--json" => a.json = true,+ "--hosted" => {+ a.hosted = match value("--hosted")?.as_str() {+ "on" => HostedSwitch::On,+ "off" => HostedSwitch::Off,+ "auto" => HostedSwitch::Auto,+ other => {+ return Err(format!("--hosted must be on, off or auto, not '{other}'"))+ }+ }+ }+ "--max-tokens" => {+ let v = value("--max-tokens")?;+ match v.parse::<usize>() {+ Ok(n) if n >= 1 => a.max_tokens = Some(n),+ _ => return Err(format!("--max-tokens must be a positive number, got '{v}'")),+ }+ }+ "--url" => a.url = Some(value("--url")?),+ f if f.starts_with("--") => return Err(format!("unknown flag '{f}'")),+ word => words.push(word),+ }+ i += 1;+ }+ if mode_flags > 1 {+ return Err("--def, --how and --files are mutually exclusive".to_string());+ }+ if let ModeRequest::Def(sym) = &a.mode {+ if sym.trim().is_empty() {+ return Err("--def needs a symbol".to_string());+ }+ }+ a.question = words.join(" ");+ if a.question.trim().is_empty() {+ // `--def SYM` alone is a complete request: the symbol is the question.+ if let ModeRequest::Def(sym) = &a.mode {+ a.question = sym.clone();+ } else {+ return Err("a question is required".to_string());+ }+ }+ Ok(Some(a))+}+// ── xerj code (with --def) ───────────────────────────────────────────────────/// `xerj code …`: `--def <SYMBOL>` answers from the `-defs` index; everything@@ -902,7 +1212,7 @@ fn emit_answer(answer: &Answer, as_json: bool) {/// `~/.xerj/hub`, overridable by `XERJ_HUB_HOME` (tests). Never `/tmp`: the/// pack cache and catalogue must survive a reboot (same rule as `code_root`).-fn hub_home() -> PathBuf {+pub(crate) fn hub_home() -> PathBuf {if let Ok(h) = std::env::var("XERJ_HUB_HOME") {if !h.is_empty() {return PathBuf::from(h);@@ -1059,6 +1369,167 @@ mod tests {assert_eq!(rc, 2, "missing --url is a usage error");}+ // ── xerj ask: argument parsing ───────────────────────────────────────────++ fn strs(a: &[&str]) -> Vec<String> {+ a.iter().map(|s| s.to_string()).collect()+ }++ fn parsed(a: &[&str]) -> AskArgs {+ parse_ask_args(&strs(a)).unwrap().unwrap()+ }++ #[test]+ fn ask_args_default_to_auto_everything() {+ let a = parsed(&["how does lz4 framing work"]);+ assert_eq!(a.question, "how does lz4 framing work");+ assert_eq!(a.mode, ModeRequest::Auto);+ assert_eq!(a.hosted, HostedSwitch::Auto);+ assert!(a.corpus.is_none() && a.url.is_none() && a.max_tokens.is_none());+ assert!(!a.offline && !a.no_fetch && !a.json);+ }++ #[test]+ fn ask_args_take_every_documented_flag() {+ let a = parsed(&[+ "a question",+ "--corpus",+ "lz4",+ "--offline",+ "--no-fetch",+ "--hosted",+ "off",+ "--max-tokens",+ "80",+ "--json",+ "--url",+ "http://127.0.0.1:9300",+ "--files",+ ]);+ assert_eq!(a.corpus.as_deref(), Some("lz4"));+ assert!(a.offline && a.no_fetch && a.json);+ assert_eq!(a.hosted, HostedSwitch::Off);+ assert_eq!(a.max_tokens, Some(80));+ assert_eq!(a.url.as_deref(), Some("http://127.0.0.1:9300"));+ assert_eq!(a.mode, ModeRequest::Files);+ assert_eq!(parsed(&["q", "--how"]).mode, ModeRequest::How);+ assert_eq!(+ parsed(&["q", "--def", "LZ4_VERSION"]).mode,+ ModeRequest::Def("LZ4_VERSION".into())+ );+ }++ #[test]+ fn unquoted_words_are_one_question_and_flags_may_come_first() {+ let a = parsed(&["--json", "how", "does", "lz4", "work", "--corpus", "lz4"]);+ assert_eq!(a.question, "how does lz4 work");+ assert!(a.json);+ // `--def SYM` alone is a complete request.+ let a = parsed(&["--def", "LZ4_VERSION_RELEASE"]);+ assert_eq!(a.question, "LZ4_VERSION_RELEASE");+ }++ #[test]+ fn ask_args_reject_misuse() {+ for (bad, why) in [+ (vec![], "no question"),+ (vec!["--json"], "flags only"),+ (vec!["q", "--bogus"], "unknown flag"),+ (vec!["q", "--corpus"], "missing value"),+ (vec!["q", "--hosted", "maybe"], "bad hosted value"),+ (vec!["q", "--max-tokens", "0"], "zero tokens"),+ (vec!["q", "--max-tokens", "lots"], "non-numeric tokens"),+ (vec!["q", "--def", "A", "--how"], "two modes"),+ (vec!["q", "--how", "--files"], "two modes"),+ (vec!["q", "--def", " "], "blank symbol"),+ ] {+ assert!(parse_ask_args(&strs(&bad)).is_err(), "{why}: {bad:?}");+ }+ assert!(parse_ask_args(&strs(&["--help"])).unwrap().is_none());+ assert!(parse_ask_args(&strs(&["q", "-h"])).unwrap().is_none());+ }++ #[test]+ fn the_bare_form_needs_a_space_a_non_flag_and_a_non_subcommand() {+ assert!(is_bare_question(Some("how does lz4 frame work")));+ assert!(is_bare_question(Some("lz4 block")));+ // No space: a typo'd or unknown word is not a question.+ assert!(!is_bare_question(Some("lz4")));+ assert!(!is_bare_question(Some("LZ4_VERSION_RELEASE")));+ // A flag, with or without spaces, is not a question.+ assert!(!is_bare_question(Some("--port")));+ assert!(!is_bare_question(Some("--data-dir /x y")));+ // Subcommands are never questions, even if one ever grew a space.+ for sub in KNOWN_SUBCOMMANDS {+ assert!(!is_bare_question(Some(sub)), "{sub}");+ }+ assert!(!is_bare_question(None));+ assert!(!is_bare_question(Some("")));+ }++ #[test]+ fn known_subcommands_cover_what_main_dispatches() {+ for sub in [+ "index",+ "search",+ "def",+ "init",+ "gain",+ "autoindex",+ "brain",+ "share",+ "feedback",+ "code",+ "corpus",+ "hub",+ "ask",+ "mcp",+ ] {+ assert!(KNOWN_SUBCOMMANDS.contains(&sub), "{sub}");+ }+ }++ // ── xerj hub status / stop / config ──────────────────────────────────────++ #[test]+ fn hub_config_writes_and_clears_the_hosted_setting() {+ let home = tempfile::tempdir().unwrap();+ assert_eq!(config_in(home.path(), &strs(&["hosted", "on"])), 0);+ let text = fs::read_to_string(home.path().join("config.toml")).unwrap();+ assert!(text.contains("hosted = \"on\""), "{text}");+ assert_eq!(config_in(home.path(), &strs(&["hosted", "off"])), 0);+ assert!(fs::read_to_string(home.path().join("config.toml"))+ .unwrap()+ .contains("hosted = \"off\""));+ assert_eq!(config_in(home.path(), &strs(&["hosted", "auto"])), 0);+ assert!(!fs::read_to_string(home.path().join("config.toml"))+ .unwrap()+ .contains("hosted"));+ // Misuse is a usage error and writes nothing new.+ assert_eq!(config_in(home.path(), &strs(&["hosted", "sometimes"])), 2);+ assert_eq!(config_in(home.path(), &strs(&["hosted"])), 2);+ assert_eq!(config_in(home.path(), &strs(&["colour", "on"])), 2);+ assert_eq!(config_in(home.path(), &strs(&[])), 2);+ }++ #[test]+ fn hub_stop_and_status_with_nothing_running_succeed() {+ let home = tempfile::tempdir().unwrap();+ assert_eq!(stop_in(home.path()), 0);+ assert_eq!(status_in(home.path()), 0);+ assert!(!home.path().join("node.pid").exists());+ }++ #[test]+ fn hub_usage_lists_the_new_commands() {+ for word in ["status", "stop", "config hosted"] {+ assert!(HUB_USAGE.contains(word), "{word}");+ }+ assert_eq!(run_hub_cli(&strs(&["status", "extra"])), 2);+ assert_eq!(run_hub_cli(&strs(&["stop", "--bogus"])), 2);+ assert_eq!(run_hub_cli(&strs(&["config"])), 2);+ }+// NOTE: `hub_pubkey()` reads the process-global `XERJ_HUB_PUBKEY`, which the// `catalogue` test module already mutates under its own `ENV_LOCK`. A unit// test here that set/removed that var would race those tests (lib tests sharediff --git a/engine/crates/xerj-hub/src/lib.rs b/engine/crates/xerj-hub/src/lib.rsindex e46f16250..ea0e9e258 100644--- a/engine/crates/xerj-hub/src/lib.rs+++ b/engine/crates/xerj-hub/src/lib.rs@@ -13,6 +13,7 @@//! pack build/pull (Step 2), definitions (Step 3), catalogue + sketch (Step 4),//! router + ask (Step 5). Alongside them this file re-exports the shared types.+pub mod ask;pub mod catalogue;pub mod cli;pub mod def;diff --git a/engine/crates/xerj-hub/src/node.rs b/engine/crates/xerj-hub/src/node.rsindex 4a01d6d09..24edcf83c 100644--- a/engine/crates/xerj-hub/src/node.rs+++ b/engine/crates/xerj-hub/src/node.rs@@ -1,18 +1,38 @@-//! Corpus Hub v2 — node resolution and a blocking HTTP client to a xerj node-//! (plan §2 "Node for the client", §4 node resolution).+//! Corpus Hub v2 — node resolution, the automatic local node, and a blocking+//! HTTP client to a xerj node (plan §2 "Node for the client", §5 "Node").//!-//! `build-pack`, `pull` and `def` all talk to a running node: the builder-//! snapshots into one, `pull` restores into one, and `def` queries one. This+//! `build-pack`, `pull`, `def` and `ask` all talk to a running node: the builder+//! snapshots into one, `pull` restores into one, and `def`/`ask` query one. This//! module resolves WHICH node ([`resolve_url`], the same `--url` / `XERJ_URL` /-//! `localhost:9200` order `xc.rs` uses) and gives a small blocking client-//! ([`HubNode`]) over it.+//! `localhost:9200` order `xc.rs` uses), gives a small blocking client+//! ([`HubNode`]) over it, and — plan §5 — starts a node of its own when none+//! answers ([`ensure_node`]) and stops exactly that node ([`stop_node`]).//!-//! The AUTOMATIC node spawn (start a node when none answers) is plan §5 and is-//! NOT done here — this module only resolves an explicit or defaulted URL and-//! connects to it. [`HubNode`] implements [`XcHttp`] so `def::lookup` runs over-//! it unchanged, and it adds the extra verbs (`PUT`/`POST` with an arbitrary-//! path, a readiness poll) the builder and the pull path need.+//! The automatic node follows four rules, each of them a line in the plan or in+//! its "do not" list:+//!+//! 1. **Never start a node on a port that answers.** The resolved URL is probed+//! first; only a silent loopback URL leads to a spawn, and the port is chosen+//! from ones that are bindable *now* (`xerj --port P` also claims `P+1` and+//! `P+2`, so all three are checked).+//! 2. **Record what was started.** The PID goes to `<hub>/node.pid` the moment+//! the process exists and the port to `<hub>/node.port` once it is healthy.+//! 3. **Stop only what was recorded.** [`stop_node`] signals the recorded PID and+//! nothing else, and first checks that the process still looks like the node+//! this tool started (its command line names `<hub>/node` as its data dir) —+//! so a recycled PID, or a second process that merely shares the binary's+//! name, is never signalled.+//! 4. **One starter at a time.** A lock directory serialises concurrent+//! `xerj ask` runs so two of them cannot both launch a node on one data dir.+//!+//! [`HubNode`] implements [`XcHttp`] so `def::lookup` runs over it unchanged, and+//! it adds the extra verbs (`PUT`/`POST` with an arbitrary path, a readiness+//! poll) the builder and the pull path need.+use std::fs;+use std::net::TcpListener;+use std::path::{Path, PathBuf};+use std::process::{Child, Command, Stdio};use std::time::{Duration, Instant};use anyhow::{anyhow, bail, Context, Result};@@ -77,6 +97,22 @@ impl HubNode {})}+ /// Connect to a node this tool manages itself ([`ensure_node`] starts it+ /// with `--insecure`), sending no credential. Unlike [`HubNode::connect`] it+ /// never reaches for a locally discoverable admin key: that key belongs to+ /// some other node, and announcing it on every `xerj ask` would be noise.+ pub fn connect_insecure(url: &str) -> Result<Self> {+ let client = reqwest::blocking::Client::builder()+ .timeout(Duration::from_secs(300))+ .build()+ .context("build HTTP client")?;+ Ok(HubNode {+ base: url.trim_end_matches('/').to_string(),+ auth: None,+ client,+ })+ }+/// The resolved base URL (no trailing slash) — for diagnostics and the/// state ledger's `url` field.pub fn url(&self) -> &str {@@ -146,9 +182,10 @@ impl HubNode {/// Poll `GET /_cluster/health` until the node answers (any HTTP status that/// is not a transport error means it is up), up to `timeout`. Returns the- /// last health body. The automatic node spawn (plan §5) will reuse this to- /// wait for a node it started; here it lets `pull`/`build-pack` fail fast- /// with a clear message when the node is not reachable.+ /// last health body. It lets `build-pack` fail fast with a clear message+ /// when the node is not reachable. (The automatic node waits on its own+ /// stricter check, `wait_until_healthy`: HTTP 200 and a green/yellow+ /// status, and it stops early if the process it started has exited.)pub fn wait_until_reachable(&self, timeout: Duration) -> Result<Value> {let deadline = Instant::now() + timeout;loop {@@ -202,6 +239,796 @@ impl XcHttp for HubNode {}}+// ── The automatic local node (plan §5 "Node") ────────────────────────────────++/// The first ES-compat port an automatic node may take. `xerj --port P` also+/// claims `P+1` (native REST) and `P+2` (gRPC), so a candidate needs all three.+pub const FIRST_AUTO_PORT: u16 = 9300;+/// Candidate base ports tried (stepping by 3) before giving up.+const AUTO_PORT_TRIES: u16 = 100;+/// How long a freshly started node gets to report a healthy cluster (plan §5:+/// "wait for green (max 60 s)").+pub const NODE_BOOT_TIMEOUT: Duration = Duration::from_secs(60);+/// How long a second `xerj ask` waits for the first one to finish starting the+/// node before it gives up (boot timeout plus slack).+const NODE_LOCK_WAIT: Duration = Duration::from_secs(90);+/// A start lock older than this was left by a crashed starter and is stolen.+const NODE_LOCK_STALE: Duration = Duration::from_secs(180);++const NODE_PID_FILE: &str = "node.pid";+const NODE_PORT_FILE: &str = "node.port";+const NODE_LOG_FILE: &str = "node.log";+const NODE_CONFIG_FILE: &str = "node.toml";+const NODE_DATA_DIR: &str = "node";+const NODE_LOCK_DIR: &str = "node.lock";++/// How to start "the xerj binary": the executable and any arguments that come+/// before the node's own flags. Production uses [`NodeLauncher::current`]+/// (`current_exe()`, no prefix); a test supplies its own pair so a cargo test+/// binary is never started with server flags it would reject.+#[derive(Debug, Clone)]+pub struct NodeLauncher {+ pub exe: PathBuf,+ pub pre_args: Vec<String>,+}++impl NodeLauncher {+ /// The running binary, i.e. the `xerj` the user invoked.+ pub fn current() -> Result<Self> {+ Ok(NodeLauncher {+ exe: std::env::current_exe().context("locate the xerj binary")?,+ pre_args: Vec::new(),+ })+ }++ /// A [`Command`] for the executable with the prefix arguments applied.+ pub fn command(&self) -> Command {+ let mut c = Command::new(&self.exe);+ c.args(&self.pre_args);+ c+ }+}++/// The node `ask`/`pull` will talk to.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct ResolvedNode {+ pub url: String,+ /// True when the node is the one this tool starts and records+ /// (`--insecure`, so no credential is sent); false for a user-supplied or+ /// default-URL node.+ pub managed: bool,+ /// True when THIS call started it (as opposed to finding it running).+ pub started_now: bool,+}++impl ResolvedNode {+ /// Connect with the right credential policy for this node.+ pub fn connect(&self) -> Result<HubNode> {+ if self.managed {+ HubNode::connect_insecure(&self.url)+ } else {+ HubNode::connect(&self.url)+ }+ }+}++/// A health probe's answer.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct Health {+ pub http_status: u16,+ /// `status` from the `/_cluster/health` body (`green`/`yellow`/`red`), when+ /// the body was JSON and carried one.+ pub status: Option<String>,+}++/// `GET {url}/_cluster/health` with a short timeout. `None` means nothing+/// answered (refused, timed out, not HTTP). ANY HTTP response counts as a live+/// node — a key-protected node answers 401 and must still not be started over.+pub fn probe(url: &str, timeout: Duration) -> Option<Health> {+ let client = reqwest::blocking::Client::builder()+ .timeout(timeout)+ .connect_timeout(timeout)+ .build()+ .ok()?;+ let resp = client+ .get(format!("{}/_cluster/health", url.trim_end_matches('/')))+ .send()+ .ok()?;+ let http_status = resp.status().as_u16();+ let status = resp+ .text()+ .ok()+ .and_then(|t| serde_json::from_str::<Value>(&t).ok())+ .and_then(|v| v.get("status").and_then(Value::as_str).map(str::to_string));+ Some(Health {+ http_status,+ status,+ })+}++const PROBE_TIMEOUT: Duration = Duration::from_secs(2);++/// The node data directory this tool starts a node on: `<hub>/node`.+pub fn node_data_dir(hub_home: &Path) -> PathBuf {+ hub_home.join(NODE_DATA_DIR)+}++/// Resolve the node to use, starting one if none answers (plan §5 "Node").+///+/// 1. `explicit` (`--url`) > `XERJ_URL` > `http://localhost:9200`. If that+/// answers, it is used — a node that answers is never started over.+/// 2. If it is silent and is NOT a loopback address, error: starting a local+/// node cannot satisfy a request for a remote one, and silently substituting+/// would hide the outage.+/// 3. Otherwise, a node this tool started earlier (`node.pid` + `node.port`) is+/// reused when it still answers.+/// 4. Otherwise start `exe --insecure --bind 127.0.0.1 --port <free ≥ 9300>+/// --data-dir <hub>/node --config <hub>/node.toml`, detached, record its PID,+/// and wait for a healthy cluster (≤ 60 s), then record the port.+///+/// `--bind 127.0.0.1` and `--config` are additions to the plan's command line:+/// the node runs with auth off, so the bind must not be steerable from the+/// environment (`XERJ_BIND_ADDRESS`); and the snapshot-repo allowlist in+/// `node.toml` is what lets the node accept `<hub>/packs/…` as a restore+/// location — without it the engine refuses any repository outside the node's+/// own data dir and `xerj hub pull` could never restore into this node.+pub fn ensure_node(+ hub_home: &Path,+ explicit: Option<&str>,+ launcher: &NodeLauncher,+) -> Result<ResolvedNode> {+ ensure_node_with(hub_home, explicit, launcher, NODE_BOOT_TIMEOUT)+}++/// [`ensure_node`] with the boot timeout spelled out (tests shorten it).+fn ensure_node_with(+ hub_home: &Path,+ explicit: Option<&str>,+ launcher: &NodeLauncher,+ boot_timeout: Duration,+) -> Result<ResolvedNode> {+ let url = resolve_url(explicit);+ if probe(&url, PROBE_TIMEOUT).is_some() {+ return Ok(ResolvedNode {+ url,+ managed: false,+ started_now: false,+ });+ }+ if !url_is_loopback(&url) {+ bail!(+ "no node answers at {url}; it is not a local address, so no local node was started \+ (pass --url for a running node, or unset XERJ_URL to use the automatic one)"+ );+ }+ if let Some(found) = recorded_node(hub_home) {+ return Ok(found);+ }++ // Start one — serialised, so concurrent callers cannot both launch a node+ // on the same data directory.+ let _lock = NodeLock::acquire(hub_home, NODE_LOCK_WAIT)?;+ if let Some(found) = recorded_node(hub_home) {+ // Another process finished starting it while we waited for the lock.+ return Ok(found);+ }+ start_node(hub_home, launcher, &url, boot_timeout)+}++/// The node this tool started earlier, if it is still running and answering.+/// Files left by a node that is gone (or whose PID now belongs to something+/// else) are removed here, so `xerj hub status` never reports a dead node.+fn recorded_node(hub_home: &Path) -> Option<ResolvedNode> {+ reap_stale(hub_home);+ let (_pid, port) = read_recorded(hub_home)?;+ let url = format!("http://127.0.0.1:{port}");+ probe(&url, PROBE_TIMEOUT)?;+ Some(ResolvedNode {+ url,+ managed: true,+ started_now: false,+ })+}++fn start_node(+ hub_home: &Path,+ launcher: &NodeLauncher,+ silent_url: &str,+ boot_timeout: Duration,+) -> Result<ResolvedNode> {+ let data_dir = node_data_dir(hub_home);+ fs::create_dir_all(&data_dir).with_context(|| format!("create {}", data_dir.display()))?;+ let config = write_node_config(hub_home)?;+ let port = pick_free_port(FIRST_AUTO_PORT, AUTO_PORT_TRIES).ok_or_else(|| {+ anyhow!(+ "no free port for a local node: tried {} blocks of three from {FIRST_AUTO_PORT}",+ AUTO_PORT_TRIES+ )+ })?;+ let log = hub_home.join(NODE_LOG_FILE);++ eprintln!(+ "xerj hub: no node answers at {silent_url}; starting a local node on port {port} \+ (data: {}, log: {})",+ data_dir.display(),+ log.display()+ );+ let args = vec![+ "--insecure".to_string(),+ "--bind".to_string(),+ "127.0.0.1".to_string(),+ "--port".to_string(),+ port.to_string(),+ "--data-dir".to_string(),+ data_dir.to_string_lossy().into_owned(),+ "--config".to_string(),+ config.to_string_lossy().into_owned(),+ ];+ let mut child = spawn_detached(launcher, &args, &log)+ .with_context(|| format!("start the local node ({})", launcher.exe.display()))?;+ let pid = child.id();+ // Recorded the moment the process exists: a crash during boot must still+ // leave a PID that `xerj hub stop` and `status` can see.+ write_file(&hub_home.join(NODE_PID_FILE), &format!("{pid}\n"))?;+ let _ = fs::remove_file(hub_home.join(NODE_PORT_FILE));++ let url = format!("http://127.0.0.1:{port}");+ match wait_until_healthy(&mut child, &url, boot_timeout, &log) {+ Ok(()) => {+ write_file(&hub_home.join(NODE_PORT_FILE), &format!("{port}\n"))?;+ eprintln!("xerj hub: node ready on port {port} (pid {pid})");+ Ok(ResolvedNode {+ url,+ managed: true,+ started_now: true,+ })+ }+ Err(e) => {+ // We started this process and recorded it, so cleaning it up is+ // stopping a PID we own — not a process we did not record.+ let _ = child.kill();+ let _ = child.wait();+ let _ = fs::remove_file(hub_home.join(NODE_PID_FILE));+ let _ = fs::remove_file(hub_home.join(NODE_PORT_FILE));+ Err(e)+ }+ }+}++/// Poll `/_cluster/health` until it is 200 with status green (yellow is+/// accepted too: a single node with an unassigned replica is usable), the child+/// exits, or `timeout` passes.+fn wait_until_healthy(child: &mut Child, url: &str, timeout: Duration, log: &Path) -> Result<()> {+ let deadline = Instant::now() + timeout;+ loop {+ if let Some(status) = child.try_wait().context("check the local node process")? {+ bail!(+ "the local node exited ({status}) before becoming healthy; its log ({}):\n{}",+ log.display(),+ log_tail(log, 20)+ );+ }+ if let Some(h) = probe(url, PROBE_TIMEOUT) {+ if h.http_status == 200 && matches!(h.status.as_deref(), Some("green" | "yellow")) {+ return Ok(());+ }+ }+ if Instant::now() >= deadline {+ bail!(+ "the local node did not report a healthy cluster within {}s; its log ({}):\n{}",+ timeout.as_secs(),+ log.display(),+ log_tail(log, 20)+ );+ }+ std::thread::sleep(Duration::from_millis(200));+ }+}++fn log_tail(path: &Path, lines: usize) -> String {+ match fs::read_to_string(path) {+ Ok(text) => {+ let all: Vec<&str> = text.lines().collect();+ all[all.len().saturating_sub(lines)..].join("\n")+ }+ Err(e) => format!("(log unreadable: {e})"),+ }+}++/// Write `<hub>/node.toml`: the one setting the node needs beyond its flags — a+/// snapshot-repo allowlist naming `<hub>/packs`, where `xerj hub pull` extracts+/// the snapshot it asks the node to restore. Rewritten on every start.+fn write_node_config(hub_home: &Path) -> Result<PathBuf> {+ let packs = hub_home.join("packs");+ fs::create_dir_all(&packs).with_context(|| format!("create {}", packs.display()))?;+ let path = hub_home.join(NODE_CONFIG_FILE);+ // serde_json string escaping is valid TOML basic-string escaping for the+ // characters a path can contain (the same trick `xerj brain` uses).+ let packs_toml = serde_json::to_string(&packs.to_string_lossy())?;+ write_file(+ &path,+ &format!(+ "# written by `xerj ask` each time it starts the hub node\n\+ [limits]\n\+ snapshot_repo_allowlist = [{packs_toml}]\n"+ ),+ )?;+ Ok(path)+}++fn write_file(path: &Path, body: &str) -> Result<()> {+ if let Some(parent) = path.parent() {+ fs::create_dir_all(parent).with_context(|| format!("create {}", parent.display()))?;+ }+ fs::write(path, body).with_context(|| format!("write {}", path.display()))+}++/// The first base port ≥ `start` (stepping by 3) whose three consecutive ports+/// are all bindable on loopback right now.+fn pick_free_port(start: u16, tries: u16) -> Option<u16> {+ let mut base = start;+ for _ in 0..tries {+ if base > u16::MAX - 2 {+ return None;+ }+ // Hold all three at once so the check is not three independent races.+ let held: Vec<_> = (0..3u16)+ .filter_map(|off| TcpListener::bind(("127.0.0.1", base + off)).ok())+ .collect();+ if held.len() == 3 {+ return Some(base);+ }+ base += 3;+ }+ None+}++/// Spawn `launcher` with `args`, detached: own process group (own session on+/// Windows), stdin closed, stdout and stderr appended to `log`. The child+/// outlives this process and never receives its Ctrl-C. Shared with the+/// background pull in `ask`.+pub(crate) fn spawn_detached(+ launcher: &NodeLauncher,+ args: &[String],+ log: &Path,+) -> Result<Child> {+ if let Some(parent) = log.parent() {+ fs::create_dir_all(parent).with_context(|| format!("create {}", parent.display()))?;+ }+ let out = fs::OpenOptions::new()+ .create(true)+ .append(true)+ .open(log)+ .with_context(|| format!("open {}", log.display()))?;+ let err = out.try_clone()?;+ let mut cmd = launcher.command();+ cmd.args(args)+ .stdin(Stdio::null())+ .stdout(Stdio::from(out))+ .stderr(Stdio::from(err));+ detach(&mut cmd);+ cmd.spawn()+ .with_context(|| format!("spawn {}", launcher.exe.display()))+}++#[cfg(unix)]+fn detach(cmd: &mut Command) {+ use std::os::unix::process::CommandExt;+ cmd.process_group(0);+}++#[cfg(windows)]+fn detach(cmd: &mut Command) {+ use std::os::windows::process::CommandExt;+ // DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP+ cmd.creation_flags(0x0000_0008 | 0x0000_0200);+}++#[cfg(not(any(unix, windows)))]+fn detach(_cmd: &mut Command) {}++/// A best-effort cross-process mutex for starting the node: an atomically+/// created `<hub>/node.lock` directory, removed on drop. A lock left by a+/// crashed starter is stolen once older than [`NODE_LOCK_STALE`].+struct NodeLock {+ path: PathBuf,+}++impl NodeLock {+ fn acquire(hub_home: &Path, wait: Duration) -> Result<NodeLock> {+ fs::create_dir_all(hub_home).with_context(|| format!("create {}", hub_home.display()))?;+ let path = hub_home.join(NODE_LOCK_DIR);+ let deadline = Instant::now() + wait;+ loop {+ match fs::create_dir(&path) {+ Ok(()) => return Ok(NodeLock { path }),+ Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {+ let stale = fs::metadata(&path)+ .ok()+ .and_then(|m| m.modified().ok())+ .and_then(|m| m.elapsed().ok())+ .is_some_and(|age| age > NODE_LOCK_STALE);+ if stale {+ let _ = fs::remove_dir(&path);+ continue;+ }+ if Instant::now() >= deadline {+ bail!(+ "another xerj process has been starting the hub node for over {}s \+ (lock {}); retry, or remove the lock if no such process exists",+ wait.as_secs(),+ path.display()+ );+ }+ std::thread::sleep(Duration::from_millis(100));+ }+ Err(e) => return Err(e).with_context(|| format!("create lock {}", path.display())),+ }+ }+ }+}++impl Drop for NodeLock {+ fn drop(&mut self) {+ let _ = fs::remove_dir(&self.path);+ }+}++// ── recorded PID / port ──────────────────────────────────────────────────────++/// `(pid, port)` from `<hub>/node.pid` and `<hub>/node.port`; `None` unless both+/// parse. (A pid file alone means the node is still booting or died booting.)+fn read_recorded(hub_home: &Path) -> Option<(u32, u16)> {+ let pid = read_pid(hub_home)?;+ let port = fs::read_to_string(hub_home.join(NODE_PORT_FILE))+ .ok()?+ .trim()+ .parse::<u16>()+ .ok()?;+ Some((pid, port))+}++fn read_pid(hub_home: &Path) -> Option<u32> {+ fs::read_to_string(hub_home.join(NODE_PID_FILE))+ .ok()?+ .trim()+ .parse::<u32>()+ .ok()+}++/// What a PID is, relative to the node this tool started.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub(crate) enum Identity {+ /// Running, and its command line names this hub's node data directory.+ Ours,+ /// Running, but it is not the node we started (a recycled PID, a decoy).+ NotOurs,+ /// No such process (or only a zombie awaiting its parent's `wait`).+ Gone,+ /// Cannot tell on this platform / with these permissions. Never signalled.+ Unknown,+}++/// Identify `pid`: does its command line contain every needle? On Linux the+/// command line is `/proc/<pid>/cmdline`; elsewhere on unix it is `ps`; on+/// Windows only the image name is available, so the needle check is replaced by+/// "the image is a xerj binary".+pub(crate) fn identify(pid: u32, needles: &[&str]) -> Identity {+ match proc_state(pid) {+ ProcState::Gone | ProcState::Zombie => Identity::Gone,+ ProcState::Unknown => Identity::Unknown,+ ProcState::Running => {+ // A process that has only just exec'd is not reliably readable: for a+ // few milliseconds `/proc/<pid>/cmdline` is empty, shows the PARENT's+ // command line (the child still shares its address space), or is a+ // splice of both. Measured: reading it right after `spawn` returned+ // mis-identified a live, correct node as "not ours" in roughly one+ // run in ten under load. So a match is believed at once, but a+ // mismatch (or an empty read) must persist across several samples+ // before it counts.+ let mut saw_other = false;+ for attempt in 0..IDENTIFY_SAMPLES {+ match proc_cmdline(pid) {+ Some(cmd) if identity_matches(&cmd, needles) => return Identity::Ours,+ Some(_) => saw_other = true,+ None => {}+ }+ if attempt + 1 < IDENTIFY_SAMPLES {+ std::thread::sleep(IDENTIFY_GAP);+ }+ }+ if saw_other {+ Identity::NotOurs+ } else {+ Identity::Unknown+ }+ }+ }+}++/// How many times [`identify`] reads a running process's command line before+/// believing a mismatch, and the pause between reads (~175 ms in all, paid only+/// when the process is NOT the one being looked for).+const IDENTIFY_SAMPLES: u32 = 8;+const IDENTIFY_GAP: Duration = Duration::from_millis(25);++#[cfg(not(windows))]+fn identity_matches(cmdline: &str, needles: &[&str]) -> bool {+ needles.iter().all(|n| cmdline.contains(n))+}++#[cfg(windows)]+fn identity_matches(image: &str, _needles: &[&str]) -> bool {+ image.to_ascii_lowercase().contains("xerj")+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+#[cfg_attr(windows, allow(dead_code))] // `tasklist` has no zombie state+enum ProcState {+ Running,+ Zombie,+ Gone,+ Unknown,+}++#[cfg(target_os = "linux")]+fn proc_state(pid: u32) -> ProcState {+ match fs::read_to_string(format!("/proc/{pid}/stat")) {+ // `pid (comm) S …` — comm may contain spaces and parens, so the state+ // is the first character after the LAST `)`.+ Ok(s) => match s+ .rsplit_once(')')+ .and_then(|(_, rest)| rest.trim_start().chars().next())+ {+ Some('Z' | 'X') => ProcState::Zombie,+ Some(_) => ProcState::Running,+ None => ProcState::Unknown,+ },+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => ProcState::Gone,+ Err(_) => ProcState::Unknown,+ }+}++#[cfg(target_os = "linux")]+fn proc_cmdline(pid: u32) -> Option<String> {+ let raw = fs::read(format!("/proc/{pid}/cmdline")).ok()?;+ if raw.is_empty() {+ return None;+ }+ Some(+ raw.split(|b| *b == 0)+ .map(|part| String::from_utf8_lossy(part).into_owned())+ .collect::<Vec<_>>()+ .join(" "),+ )+}++#[cfg(all(unix, not(target_os = "linux")))]+fn proc_state(pid: u32) -> ProcState {+ match Command::new("ps")+ .args(["-p", &pid.to_string(), "-o", "stat="])+ .output()+ {+ Ok(out) if out.status.success() => {+ let stat = String::from_utf8_lossy(&out.stdout);+ match stat.trim().chars().next() {+ Some('Z') => ProcState::Zombie,+ Some(_) => ProcState::Running,+ None => ProcState::Gone,+ }+ }+ // `ps -p` exits non-zero when the PID does not exist.+ Ok(_) => ProcState::Gone,+ Err(_) => ProcState::Unknown,+ }+}++#[cfg(all(unix, not(target_os = "linux")))]+fn proc_cmdline(pid: u32) -> Option<String> {+ let out = Command::new("ps")+ .args(["-p", &pid.to_string(), "-o", "command="])+ .output()+ .ok()?;+ if !out.status.success() {+ return None;+ }+ let s = String::from_utf8_lossy(&out.stdout).trim().to_string();+ (!s.is_empty()).then_some(s)+}++#[cfg(windows)]+fn tasklist_row(pid: u32) -> Result<Option<String>, ()> {+ let out = Command::new("tasklist")+ .args(["/FI", &format!("PID eq {pid}"), "/FO", "CSV", "/NH"])+ .output()+ .map_err(|_| ())?;+ let text = String::from_utf8_lossy(&out.stdout).to_string();+ Ok(text+ .lines()+ .find(|l| l.starts_with('"'))+ .map(str::to_string))+}++#[cfg(windows)]+fn proc_state(pid: u32) -> ProcState {+ match tasklist_row(pid) {+ Ok(Some(_)) => ProcState::Running,+ Ok(None) => ProcState::Gone,+ Err(()) => ProcState::Unknown,+ }+}++#[cfg(windows)]+fn proc_cmdline(pid: u32) -> Option<String> {+ tasklist_row(pid).ok().flatten()+}++/// The needle that makes a PID "the hub node we started": its command line+/// carries `--data-dir <hub>/node` (the two arguments adjacent), which only the+/// node this tool launched has (on Windows only the image name is checked — see+/// [`identify`]).+fn node_needles(hub_home: &Path) -> Vec<String> {+ vec![format!(+ "--data-dir {}",+ node_data_dir(hub_home).to_string_lossy()+ )]+}++fn identify_node(hub_home: &Path, pid: u32) -> Identity {+ let needles = node_needles(hub_home);+ let refs: Vec<&str> = needles.iter().map(String::as_str).collect();+ identify(pid, &refs)+}++/// Remove `node.pid` / `node.port` when the recorded process is gone or is no+/// longer the node we started. An `Unknown` identity leaves the files alone.+pub fn reap_stale(hub_home: &Path) {+ let Some(pid) = read_pid(hub_home) else {+ // No parseable PID: any port file is an orphan.+ if hub_home.join(NODE_PORT_FILE).exists() {+ let _ = fs::remove_file(hub_home.join(NODE_PORT_FILE));+ }+ return;+ };+ if matches!(+ identify_node(hub_home, pid),+ Identity::Gone | Identity::NotOurs+ ) {+ let _ = fs::remove_file(hub_home.join(NODE_PID_FILE));+ let _ = fs::remove_file(hub_home.join(NODE_PORT_FILE));+ }+}++// ── stopping ─────────────────────────────────────────────────────────────────++/// What [`stop_node`] did.+#[derive(Debug, Clone, PartialEq, Eq)]+pub enum StopOutcome {+ /// No `node.pid` — nothing was started by this tool.+ NothingRecorded,+ /// The recorded process was already gone (or its PID now belongs to another+ /// program); the stale record was removed and nothing was signalled.+ StaleRemoved { pid: u32 },+ /// The recorded node was stopped.+ Stopped { pid: u32 },+}++/// Stop the node this tool started: ONLY the PID in `<hub>/node.pid`, and only+/// after confirming the process is still that node. Never matches by name,+/// never signals a process group, never touches a node it did not record+/// (plan §11).+pub fn stop_node(hub_home: &Path) -> Result<StopOutcome> {+ let Some(pid) = read_pid(hub_home) else {+ let _ = fs::remove_file(hub_home.join(NODE_PID_FILE));+ let _ = fs::remove_file(hub_home.join(NODE_PORT_FILE));+ return Ok(StopOutcome::NothingRecorded);+ };+ match identify_node(hub_home, pid) {+ Identity::Gone | Identity::NotOurs => {+ let _ = fs::remove_file(hub_home.join(NODE_PID_FILE));+ let _ = fs::remove_file(hub_home.join(NODE_PORT_FILE));+ Ok(StopOutcome::StaleRemoved { pid })+ }+ Identity::Unknown => bail!(+ "cannot confirm that pid {pid} is the hub node (this platform or its permissions \+ do not expose the process command line); not signalling it. If it is the node, \+ stop it yourself and delete {}",+ hub_home.join(NODE_PID_FILE).display()+ ),+ Identity::Ours => {+ terminate(pid)?;+ // Not "gone" yet? Wait, then escalate once — to the same PID.+ let gone = wait_gone(hub_home, pid, Duration::from_secs(10));+ if !gone {+ kill_hard(pid)?;+ if !wait_gone(hub_home, pid, Duration::from_secs(3)) {+ bail!("pid {pid} (the hub node) did not exit after SIGTERM and SIGKILL");+ }+ }+ let _ = fs::remove_file(hub_home.join(NODE_PID_FILE));+ let _ = fs::remove_file(hub_home.join(NODE_PORT_FILE));+ Ok(StopOutcome::Stopped { pid })+ }+ }+}++fn wait_gone(hub_home: &Path, pid: u32, timeout: Duration) -> bool {+ let deadline = Instant::now() + timeout;+ loop {+ if matches!(identify_node(hub_home, pid), Identity::Gone) {+ return true;+ }+ if Instant::now() >= deadline {+ return false;+ }+ std::thread::sleep(Duration::from_millis(100));+ }+}++#[cfg(unix)]+fn terminate(pid: u32) -> Result<()> {+ send_signal(pid, libc::SIGTERM)+}++#[cfg(unix)]+fn kill_hard(pid: u32) -> Result<()> {+ send_signal(pid, libc::SIGKILL)+}++#[cfg(unix)]+fn send_signal(pid: u32, sig: libc::c_int) -> Result<()> {+ // A single PID — never `-pid` (a whole process group).+ // SAFETY: plain syscall with integer arguments; `pid` was read from the+ // hub's own pid file and identity-checked immediately before.+ let rc = unsafe { libc::kill(pid as libc::pid_t, sig) };+ if rc == 0 {+ Ok(())+ } else {+ let err = std::io::Error::last_os_error();+ if err.raw_os_error() == Some(libc::ESRCH) {+ Ok(()) // exited between the identity check and the signal+ } else {+ Err(err).with_context(|| format!("signal pid {pid}"))+ }+ }+}++#[cfg(windows)]+fn terminate(pid: u32) -> Result<()> {+ taskkill(pid)+}++#[cfg(windows)]+fn kill_hard(pid: u32) -> Result<()> {+ taskkill(pid)+}++#[cfg(windows)]+fn taskkill(pid: u32) -> Result<()> {+ // `/F` without `/T`: exactly this PID, not its children.+ let status = Command::new("taskkill")+ .args(["/PID", &pid.to_string(), "/F"])+ .stdout(Stdio::null())+ .stderr(Stdio::null())+ .status()+ .context("run taskkill")?;+ if status.success() {+ Ok(())+ } else {+ bail!("taskkill /PID {pid} failed ({status})")+ }+}++#[cfg(not(any(unix, windows)))]+fn terminate(_pid: u32) -> Result<()> {+ bail!("stopping a process is not supported on this platform")+}++#[cfg(not(any(unix, windows)))]+fn kill_hard(_pid: u32) -> Result<()> {+ bail!("stopping a process is not supported on this platform")+}+#[cfg(test)]mod tests {use super::*;@@ -245,4 +1072,421 @@ mod tests {.unwrap();assert_eq!(n.auth.as_deref(), Some("ApiKey sekret"));}++ // ── the automatic node ──────────────────────────────────────────────────++ use std::io::{Read, Write};+ use std::net::TcpListener;++ /// A stand-in node: answers every request on an ephemeral loopback port with+ /// `status` and the given JSON body. The thread lives until the test binary+ /// exits.+ fn fake_node(status: &'static str, body: &'static str) -> u16 {+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();+ let port = listener.local_addr().unwrap().port();+ std::thread::spawn(move || {+ for conn in listener.incoming() {+ let Ok(mut conn) = conn else { continue };+ let mut buf = [0u8; 2048];+ let _ = conn.read(&mut buf);+ let reply = format!(+ "HTTP/1.1 {status}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",+ body.len()+ );+ let _ = conn.write_all(reply.as_bytes());+ }+ });+ port+ }++ /// A launcher that must never be used.+ fn forbidden_launcher() -> NodeLauncher {+ NodeLauncher {+ exe: PathBuf::from("/nonexistent/xerj-must-not-be-spawned"),+ pre_args: vec![],+ }+ }++ #[test]+ fn a_node_that_answers_is_used_and_nothing_is_started() {+ let hub = tempfile::tempdir().unwrap();+ let port = fake_node("200 OK", r#"{"status":"green"}"#);+ let url = format!("http://127.0.0.1:{port}");+ let r = ensure_node(hub.path(), Some(&url), &forbidden_launcher()).unwrap();+ assert_eq!(r.url, url);+ assert!(!r.managed && !r.started_now);+ assert!(!hub.path().join("node.pid").exists());+ }++ #[test]+ fn a_key_protected_node_that_answers_401_still_counts_as_running() {+ let hub = tempfile::tempdir().unwrap();+ let port = fake_node("401 Unauthorized", r#"{"error":"auth"}"#);+ let url = format!("http://127.0.0.1:{port}");+ let r = ensure_node(hub.path(), Some(&url), &forbidden_launcher()).unwrap();+ assert_eq!(r.url, url, "never started over");+ let h = probe(&url, Duration::from_secs(2)).unwrap();+ assert_eq!(h.http_status, 401);+ }++ #[test]+ fn probe_reads_the_cluster_status_and_reports_silence_as_none() {+ let port = fake_node("200 OK", r#"{"status":"yellow","number_of_nodes":1}"#);+ let h = probe(&format!("http://127.0.0.1:{port}/"), Duration::from_secs(2)).unwrap();+ assert_eq!(h.http_status, 200);+ assert_eq!(h.status.as_deref(), Some("yellow"));+ // A port nothing listens on: bind then drop to find one.+ let closed = TcpListener::bind("127.0.0.1:0")+ .unwrap()+ .local_addr()+ .unwrap()+ .port();+ assert!(probe(+ &format!("http://127.0.0.1:{closed}"),+ Duration::from_secs(1)+ )+ .is_none());+ }++ #[test]+ fn a_silent_remote_url_is_an_error_not_a_local_node() {+ let hub = tempfile::tempdir().unwrap();+ let err = ensure_node(+ hub.path(),+ Some("http://nonexistent.invalid:9200"),+ &forbidden_launcher(),+ )+ .unwrap_err()+ .to_string();+ assert!(err.contains("not a local address"), "{err}");+ assert!(!hub.path().join("node.pid").exists());+ }++ #[test]+ fn free_port_search_skips_a_busy_triple() {+ let held = TcpListener::bind("127.0.0.1:0").unwrap();+ let busy = held.local_addr().unwrap().port();+ let found = pick_free_port(busy, 50).expect("some free triple above");+ // Candidates step by three from the start, so a busy start is passed over.+ assert!(found > busy, "the busy base port is skipped");+ assert_eq!((found - busy) % 3, 0);+ // The search is bounded by the top of the port range and by `tries`.+ assert!(+ pick_free_port(u16::MAX - 1, 5).is_none(),+ "no room above the top"+ );+ assert!(pick_free_port(busy, 0).is_none(), "no tries, no port");+ // The default floor is the plan's: never 9200, never 9297-9299.+ assert_eq!(FIRST_AUTO_PORT, 9300);+ }++ #[test]+ fn the_node_config_is_accepted_by_the_real_config_loader_and_allows_the_packs_dir() {+ let hub = tempfile::tempdir().unwrap();+ let path = write_node_config(hub.path()).unwrap();+ let cfg = xerj_common::config::Config::load(&path).expect("node.toml loads");+ let packs = hub.path().join("packs");+ assert_eq!(+ cfg.limits.snapshot_repo_allowlist,+ [packs.to_string_lossy().into_owned()]+ );+ }++ #[test]+ fn the_start_lock_serialises_starters_and_steals_a_stale_one() {+ let hub = tempfile::tempdir().unwrap();+ let first = NodeLock::acquire(hub.path(), Duration::from_secs(1)).unwrap();+ let err = NodeLock::acquire(hub.path(), Duration::from_millis(300))+ .err()+ .expect("held lock times out")+ .to_string();+ assert!(err.contains("another xerj process"), "{err}");+ drop(first);+ let again = NodeLock::acquire(hub.path(), Duration::from_secs(1)).unwrap();+ drop(again);+ assert!(!hub.path().join(NODE_LOCK_DIR).exists());+ }++ #[test]+ fn connect_insecure_sends_no_credential() {+ let n = HubNode::connect_insecure("http://127.0.0.1:1/").unwrap();+ assert_eq!(n.url(), "http://127.0.0.1:1");+ assert!(n.auth.is_none());+ }++ // The process-level tests drive real child processes through `sh`, so they+ // are unix-only; the Windows paths are compiled but not exercised here.+ #[cfg(unix)]+ mod processes {+ use super::*;+ use std::process::Child;++ /// A long-lived `sh` whose command line carries `--data-dir <dir>` — what+ /// `identify_node` looks for. The trailing `; true` keeps `sh` itself+ /// alive (a lone command may be exec'd, replacing the command line).+ fn spawn_node_like(dir: &Path) -> Child {+ Command::new("sh")+ .args(["-c", "sleep 30; true", "node", "--data-dir"])+ .arg(dir)+ .stdin(Stdio::null())+ .stdout(Stdio::null())+ .stderr(Stdio::null())+ .spawn()+ .unwrap()+ }++ /// A process of the same program but a different command line — the+ /// "dummy process named like the binary" that must survive.+ fn spawn_decoy() -> Child {+ Command::new("sh")+ .args([+ "-c",+ "sleep 30; true",+ "decoy",+ "--data-dir",+ "/elsewhere/node",+ ])+ .stdin(Stdio::null())+ .stdout(Stdio::null())+ .stderr(Stdio::null())+ .spawn()+ .unwrap()+ }++ fn alive(child: &mut Child) -> bool {+ child.try_wait().unwrap().is_none()+ }++ fn record(hub: &Path, pid: u32, port: Option<u16>) {+ fs::write(hub.join(NODE_PID_FILE), format!("{pid}\n")).unwrap();+ if let Some(p) = port {+ fs::write(hub.join(NODE_PORT_FILE), format!("{p}\n")).unwrap();+ }+ }++ #[test]+ fn stop_signals_exactly_the_recorded_pid_and_spares_a_look_alike() {+ let hub = tempfile::tempdir().unwrap();+ let dir = node_data_dir(hub.path());+ let mut node = spawn_node_like(&dir);+ let mut decoy = spawn_decoy();+ record(hub.path(), node.id(), Some(9301));+ assert_eq!(identify_node(hub.path(), node.id()), Identity::Ours);+ assert_eq!(identify_node(hub.path(), decoy.id()), Identity::NotOurs);++ let outcome = stop_node(hub.path()).unwrap();+ assert_eq!(outcome, StopOutcome::Stopped { pid: node.id() });+ // The recorded process is dead (reap it) ...+ let status = node.wait().unwrap();+ assert!(!status.success(), "terminated by a signal: {status:?}");+ // ... the look-alike is untouched, and the records are gone.+ assert!(alive(&mut decoy), "the decoy must survive `hub stop`");+ assert!(!hub.path().join(NODE_PID_FILE).exists());+ assert!(!hub.path().join(NODE_PORT_FILE).exists());+ decoy.kill().unwrap();+ decoy.wait().unwrap();+ }++ #[test]+ fn a_recycled_pid_is_never_signalled() {+ let hub = tempfile::tempdir().unwrap();+ // The record names a live process that is NOT this hub's node.+ let mut stranger = spawn_decoy();+ record(hub.path(), stranger.id(), Some(9301));+ let outcome = stop_node(hub.path()).unwrap();+ assert_eq!(outcome, StopOutcome::StaleRemoved { pid: stranger.id() });+ assert!(+ alive(&mut stranger),+ "an unrelated process must not be killed"+ );+ assert!(+ !hub.path().join(NODE_PID_FILE).exists(),+ "the stale record goes"+ );+ stranger.kill().unwrap();+ stranger.wait().unwrap();+ }++ #[test]+ fn stop_with_nothing_recorded_or_a_dead_pid_is_a_clean_no_op() {+ let hub = tempfile::tempdir().unwrap();+ assert_eq!(stop_node(hub.path()).unwrap(), StopOutcome::NothingRecorded);++ let mut gone = Command::new("sh").args(["-c", "exit 0"]).spawn().unwrap();+ let pid = gone.id();+ gone.wait().unwrap();+ record(hub.path(), pid, Some(9301));+ assert_eq!(identify_node(hub.path(), pid), Identity::Gone);+ assert_eq!(+ stop_node(hub.path()).unwrap(),+ StopOutcome::StaleRemoved { pid }+ );+ assert!(!hub.path().join(NODE_PID_FILE).exists());+ }++ #[test]+ fn a_zombie_counts_as_gone() {+ let hub = tempfile::tempdir().unwrap();+ // Exits at once; not `wait`ed, so it stays a zombie until reaped.+ let mut child = Command::new("sh").args(["-c", "exit 0"]).spawn().unwrap();+ std::thread::sleep(Duration::from_millis(300));+ assert_eq!(identify_node(hub.path(), child.id()), Identity::Gone);+ child.wait().unwrap();+ }++ #[test]+ fn identify_matches_every_needle_in_the_command_line() {+ let me = std::process::id();+ assert_eq!(identify(me, &["xerj_hub"]), Identity::Ours);+ assert_eq!(+ identify(me, &["xerj_hub", "no-such-argument"]),+ Identity::NotOurs+ );+ }++ #[test]+ fn status_forgets_a_dead_node_but_keeps_a_live_one() {+ let hub = tempfile::tempdir().unwrap();+ let mut dead = Command::new("sh").args(["-c", "exit 0"]).spawn().unwrap();+ let dead_pid = dead.id();+ dead.wait().unwrap();+ record(hub.path(), dead_pid, Some(9301));+ reap_stale(hub.path());+ assert!(!hub.path().join(NODE_PID_FILE).exists());+ assert!(!hub.path().join(NODE_PORT_FILE).exists());++ let mut live = spawn_node_like(&node_data_dir(hub.path()));+ record(hub.path(), live.id(), Some(9301));+ reap_stale(hub.path());+ assert!(hub.path().join(NODE_PID_FILE).exists());+ live.kill().unwrap();+ live.wait().unwrap();+ }++ #[test]+ fn a_recorded_live_node_that_answers_is_reused_not_restarted() {+ let hub = tempfile::tempdir().unwrap();+ let mut live = spawn_node_like(&node_data_dir(hub.path()));+ let port = fake_node("200 OK", r#"{"status":"green"}"#);+ record(hub.path(), live.id(), Some(port));+ // The default-ish URL is silent (a closed loopback port), so the+ // recorded node is the one used.+ let silent = TcpListener::bind("127.0.0.1:0")+ .unwrap()+ .local_addr()+ .unwrap()+ .port();+ let r = ensure_node(+ hub.path(),+ Some(&format!("http://127.0.0.1:{silent}")),+ &forbidden_launcher(),+ )+ .unwrap();+ assert_eq!(r.url, format!("http://127.0.0.1:{port}"));+ assert!(r.managed && !r.started_now);+ live.kill().unwrap();+ live.wait().unwrap();+ }++ #[test]+ fn a_node_that_dies_while_booting_leaves_no_record_and_shows_its_log() {+ let hub = tempfile::tempdir().unwrap();+ let silent = TcpListener::bind("127.0.0.1:0")+ .unwrap()+ .local_addr()+ .unwrap()+ .port();+ // `sh -c 'echo boom; exit 3' sh <node flags…>` — exits straight away.+ let launcher = NodeLauncher {+ exe: PathBuf::from("sh"),+ pre_args: vec!["-c".into(), "echo boom-in-log; exit 3".into(), "sh".into()],+ };+ let err = ensure_node_with(+ hub.path(),+ Some(&format!("http://127.0.0.1:{silent}")),+ &launcher,+ Duration::from_secs(10),+ )+ .unwrap_err()+ .to_string();+ assert!(err.contains("exited"), "{err}");+ assert!(+ err.contains("boom-in-log"),+ "the node's log is shown: {err}"+ );+ assert!(!hub.path().join(NODE_PID_FILE).exists());+ assert!(!hub.path().join(NODE_PORT_FILE).exists());+ assert!(+ !hub.path().join(NODE_LOCK_DIR).exists(),+ "the lock is released"+ );+ }++ #[test]+ fn a_node_that_never_gets_healthy_is_killed_and_forgotten_at_the_timeout() {+ let hub = tempfile::tempdir().unwrap();+ let silent = TcpListener::bind("127.0.0.1:0")+ .unwrap()+ .local_addr()+ .unwrap()+ .port();+ let launcher = NodeLauncher {+ exe: PathBuf::from("sh"),+ pre_args: vec!["-c".into(), "sleep 30; true".into(), "sh".into()],+ };+ let t0 = Instant::now();+ let err = ensure_node_with(+ hub.path(),+ Some(&format!("http://127.0.0.1:{silent}")),+ &launcher,+ Duration::from_millis(800),+ )+ .unwrap_err()+ .to_string();+ assert!(err.contains("did not report a healthy cluster"), "{err}");+ assert!(t0.elapsed() < Duration::from_secs(10));+ assert!(!hub.path().join(NODE_PID_FILE).exists());+ assert!(!hub.path().join(NODE_PORT_FILE).exists());+ }++ #[test]+ fn the_node_is_started_with_the_plans_flags_plus_loopback_bind_and_config() {+ let hub = tempfile::tempdir().unwrap();+ let silent = TcpListener::bind("127.0.0.1:0")+ .unwrap()+ .local_addr()+ .unwrap()+ .port();+ // `printf '%s\n' "$@"` writes the node's argv to its log, then fails+ // fast so the test does not wait on a healthy cluster.+ let launcher = NodeLauncher {+ exe: PathBuf::from("sh"),+ pre_args: vec![+ "-c".into(),+ "printf '%s\\n' \"$@\"; exit 4".into(),+ "sh".into(),+ ],+ };+ let _ = ensure_node_with(+ hub.path(),+ Some(&format!("http://127.0.0.1:{silent}")),+ &launcher,+ Duration::from_secs(10),+ );+ let log = fs::read_to_string(hub.path().join(NODE_LOG_FILE)).unwrap();+ let argv: Vec<&str> = log.lines().collect();+ assert_eq!(argv[0], "--insecure");+ assert_eq!(&argv[1..3], ["--bind", "127.0.0.1"]);+ assert_eq!(argv[3], "--port");+ let port: u16 = argv[4].parse().unwrap();+ assert!(+ port >= FIRST_AUTO_PORT,+ "port {port} is below the floor (never 9200, never 9297-9299)"+ );+ assert_eq!(argv[5], "--data-dir");+ assert_eq!(argv[6], node_data_dir(hub.path()).to_string_lossy());+ assert_eq!(argv[7], "--config");+ assert_eq!(argv[8], hub.path().join("node.toml").to_string_lossy());+ }+ }}diff --git a/engine/crates/xerj-hub/tests/ask_flow.rs b/engine/crates/xerj-hub/tests/ask_flow.rsnew file mode 100644index 000000000..97cf0a022--- /dev/null+++ b/engine/crates/xerj-hub/tests/ask_flow.rs@@ -0,0 +1,925 @@+//! `xerj ask` end to end (plan §5 "Tests").+//!+//! Real nodes, real packs, real pulls — the pieces `src/ask.rs`'s unit tests+//! fake. One `#[tokio::test]` walks the scenarios in order because they share+//! process-global environment (`XERJ_HUB_HOME`, `XERJ_CODE_HOME`,+//! `XERJ_HUB_PUBKEY`, `HOME`):+//!+//! - **A. first question, no hosted hub:** `ask` pulls the pack (sha256 and+//! signature verified, snapshot restored into a node) and answers+//! `--def MINI_LIMIT` from it — value `128`, "answered from: local pack". A+//! second question is answered from the cache with the pack's source file+//! deleted: it cannot have downloaded anything. How and Files answers come off+//! the same pack.+//! - **B. hosted first, pull behind it:** a hosted stub answers at once, a+//! detached `xerj hub pull` is started (PID file left under `pulls/`), it+//! finishes on its own, and the next `ask` answers `LocalPack` with the hosted+//! hub forbidden.+//! - **C. the automatic node:** with nothing on the requested loopback port,+//! `ask` starts a node on port ≥ 9300 (flags, config and allowlist exactly as+//! `xerj` would), pulls into it, answers, and `hub stop` stops exactly the+//! recorded PID — a second process of the very same executable survives.+//! - **D. the disk budget:** a pull evicts a least-recently-used pack that is+//! over the 4 GiB budget (sparse files, so nothing is written).+//! - **E. offline with no pack:** exit 1, and no node is started to say so.+//!+//! The test binary doubles as the `xerj` executable the flow spawns: a+//! `NodeLauncher` points at it with `--ignored --exact child_main --nocapture --`+//! in front, and [`child_main`] turns what follows the `--` into either+//! `xerj hub …` (run in-process), a real engine node (`--insecure --port …`), or+//! a decoy that just sleeps. That keeps the spawned code paths identical to+//! production (`current_exe() <args>`) without needing the `xerj` binary.++use std::path::{Path, PathBuf};+#[cfg(unix)]+use std::process::{Command, Stdio};+use std::time::{Duration, Instant};++use serde_json::json;+use sha2::{Digest, Sha256};++use xerj_autoindex::harvest::sign::generate_keypair;+use xerj_hub::ask::{+ ask, AskArgs, AskDeps, AskOutcome, HostedHub, ModeRequest, RealBackend, Request,+ EXIT_NO_ANSWER, EXIT_OK,+};+use xerj_hub::catalogue::load_catalogue;+#[cfg(unix)]+use xerj_hub::node::StopOutcome;+use xerj_hub::node::{self, HubNode, NodeLauncher};+use xerj_hub::pack::{self, PackFileEntry};+use xerj_hub::router::HostedSearch;+use xerj_hub::{+ Answer, AnswerMode, AnsweredFrom, Catalogue, CatalogueEntry, DefRecord, Hit, LicenceClass,+ PackIndex, PackManifest, PackSource, Sketch, CATALOGUE_FORMAT_VERSION, PACK_FORMAT_VERSION,+};++const PIN_SHA: &str = "0774d05537f9762f838f7ab541b7765f1a729cb5";+const PIN8: &str = "0774d055";+const PIN_DATE: &str = "2026-10-02";+const PREFIX: &str = "xc-mini-c-b1";++// ── the child process: this binary standing in for `xerj` ────────────────────++/// Not a test. The scenarios start this binary with+/// `--ignored --exact child_main --nocapture -- <xerj arguments>`; libtest hands+/// everything after the `--` to us as plain arguments.+#[test]+#[ignore = "child-process entry point, started by the scenarios in this file"]+fn child_main() {+ let args: Vec<String> = std::env::args().skip_while(|a| a != "--").skip(1).collect();+ match args.first().map(String::as_str) {+ // `xerj hub pull …` — what the background pull runs.+ Some("hub") => std::process::exit(xerj_hub::cli::run_hub_cli(&args[1..])),+ // A second process of the same executable, to prove `hub stop` is exact.+ Some("decoy") => std::thread::sleep(Duration::from_secs(300)),+ // `--insecure --bind 127.0.0.1 --port P --data-dir D --config C` — a node.+ _ => node_main(&args),+ }+}++/// Boot an ES-compatible engine node on the port/data dir/config named by the+/// flags `xerj hub` passes, and serve until killed. The config comes from the+/// real `Config::load`, so a malformed `node.toml` or a missing snapshot+/// allowlist fails here exactly as it would in the real server.+fn node_main(args: &[String]) {+ let (mut port, mut data_dir, mut config) = (0u16, None, None);+ let mut it = args.iter();+ while let Some(a) = it.next() {+ match a.as_str() {+ "--port" => port = it.next().and_then(|p| p.parse().ok()).expect("--port N"),+ "--data-dir" => data_dir = it.next().cloned(),+ "--config" => config = it.next().cloned(),+ _ => {}+ }+ }+ let mut cfg = xerj_common::config::Config::load(config.expect("--config <file>"))+ .expect("node config loads");+ cfg.server.data_dir = data_dir.expect("--data-dir <dir>");+ cfg.auth.enabled = false;+ cfg.tls.enabled = false;+ cfg.storage.wal_sync = xerj_common::config::WalSync::Async;+ std::fs::create_dir_all(&cfg.server.data_dir).unwrap();+ let rt = tokio::runtime::Builder::new_multi_thread()+ .worker_threads(2)+ .enable_all()+ .build()+ .unwrap();+ rt.block_on(async move {+ let metrics = xerj_common::metrics::Metrics::new().unwrap();+ let engine = xerj_engine::Engine::new(cfg.clone()).unwrap();+ let state = xerj_api::state::AppState::new(cfg, engine, metrics);+ let app = xerj_api::router::build_es_compat_router(state);+ let listener = tokio::net::TcpListener::bind(("127.0.0.1", port))+ .await+ .expect("bind the node port");+ axum::serve(listener, app.into_make_service())+ .await+ .unwrap();+ });+}++/// The test binary as a `xerj` the flow can spawn.+fn test_exe_launcher() -> NodeLauncher {+ NodeLauncher {+ exe: std::env::current_exe().unwrap(),+ pre_args: ["--ignored", "--exact", "child_main", "--nocapture", "--"]+ .map(String::from)+ .to_vec(),+ }+}++/// A launcher that fails loudly if the flow ever tries to start anything.+fn forbidden_launcher() -> NodeLauncher {+ NodeLauncher {+ exe: PathBuf::from("/nonexistent/xerj-must-not-be-spawned"),+ pre_args: vec![],+ }+}++// ── hosted stand-ins ─────────────────────────────────────────────────────────++/// Answers a `MINI_LIMIT` definition lookup at once.+struct CannedHosted;++impl HostedSearch for CannedHosted {+ fn search(&self, _q: &str) -> anyhow::Result<Vec<(String, f32)>> {+ Ok(vec![])+ }+}++impl HostedHub for CannedHosted {+ fn answer(&self, _req: &Request<'_>) -> anyhow::Result<Answer> {+ Ok(Answer {+ corpus: "mini-c".to_string(),+ pin_sha: PIN_SHA.to_string(),+ pin_date: PIN_DATE.to_string(),+ age_days: 12,+ licence_spdx: "MIT".to_string(),+ licence_class: LicenceClass::AdaptWithAttribution,+ answered_from: AnsweredFrom::Hosted,+ mode: AnswerMode::Def,+ hits: vec![Hit {+ file: "limits.h".to_string(),+ line: 5,+ kind: "define".to_string(),+ text: "#define MINI_LIMIT 128".to_string(),+ value: Some("128".to_string()),+ score: 1.0,+ licence_warning: None,+ }],+ truncated: false,+ warnings: vec![],+ })+ }+}++/// A hosted hub that panics if it is contacted.+struct ForbiddenHosted;++impl HostedSearch for ForbiddenHosted {+ fn search(&self, _q: &str) -> anyhow::Result<Vec<(String, f32)>> {+ panic!("the hosted hub must not be contacted");+ }+}++impl HostedHub for ForbiddenHosted {+ fn answer(&self, _req: &Request<'_>) -> anyhow::Result<Answer> {+ panic!("the hosted hub must not be contacted");+ }+}++/// Stops what scenario C starts even when an assertion fails halfway, so a red+/// run cannot leave a node or a decoy behind. It signals only the PID recorded+/// in the hub's `node.pid` (the same rule as `hub stop`) and the decoy it holds.+#[cfg(unix)]+struct Reaper {+ hub: PathBuf,+ decoy: Option<std::process::Child>,+}++#[cfg(unix)]+impl Drop for Reaper {+ fn drop(&mut self) {+ let _ = node::stop_node(&self.hub);+ if let Some(mut d) = self.decoy.take() {+ let _ = d.kill();+ let _ = d.wait();+ }+ }+}++#[derive(Clone, Copy)]+enum HostedKind {+ None,+ Canned,+ Forbidden,+}++// ── fixtures (the same mini-c pack `pack_roundtrip.rs` builds) ───────────────++/// Boot an es-compat node on an ephemeral port with `data_dir` and the given+/// snapshot-repo allowlist, and wait until it answers.+async fn spawn_node(data_dir: &Path, allowlist: Vec<String>) -> String {+ std::fs::create_dir_all(data_dir).unwrap();+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = data_dir.to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ config.limits.snapshot_repo_allowlist = allowlist;+ let metrics = xerj_common::metrics::Metrics::new().unwrap();+ let engine = xerj_engine::Engine::new(config.clone()).unwrap();+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ let app = xerj_api::router::build_es_compat_router(state);+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let port = listener.local_addr().unwrap().port();+ tokio::spawn(async move {+ let _ = axum::serve(listener, app.into_make_service()).await;+ });+ let url = format!("http://127.0.0.1:{port}");+ let probe = url.clone();+ tokio::task::spawn_blocking(move || {+ HubNode::connect_insecure(&probe)+ .unwrap()+ .wait_until_reachable(Duration::from_secs(30))+ .unwrap();+ })+ .await+ .unwrap();+ url+}++fn sample_defs() -> Vec<DefRecord> {+ vec![+ DefRecord {+ symbol: "MINI_LIMIT".into(),+ kind: "define".into(),+ lang: "c".into(),+ file: "limits.h".into(),+ line: 5,+ text: "#define MINI_LIMIT 128".into(),+ value: Some("128".into()),+ },+ DefRecord {+ symbol: "MINI_HASH_SEED".into(),+ kind: "define".into(),+ lang: "c".into(),+ file: "hash.c".into(),+ line: 3,+ text: "#define MINI_HASH_SEED 5381".into(),+ value: Some("5381".into()),+ },+ ]+}++fn manifest() -> PackManifest {+ PackManifest {+ format_version: PACK_FORMAT_VERSION,+ slug: "mini-c".into(),+ pin_sha: PIN_SHA.into(),+ pin_date: PIN_DATE.into(),+ built_at: "2026-10-14T00:00:00Z".into(),+ builder_version: "xerj test".into(),+ index_prefix: PREFIX.into(),+ indices: vec![+ PackIndex {+ name: format!("{PREFIX}-defs"),+ docs: 2,+ bytes: 0,+ },+ PackIndex {+ name: format!("{PREFIX}-docs"),+ docs: 2,+ bytes: 0,+ },+ ],+ snapshot_name: format!("mini-c-{PIN8}"),+ licence_spdx: "MIT".into(),+ licence_class: LicenceClass::AdaptWithAttribution,+ sources: vec![PackSource {+ repo: "mini-c".into(),+ url: "file:///fixtures/mini-c".into(),+ sha: PIN_SHA.into(),+ licence: "MIT".into(),+ }],+ defs_count: 2,+ files_count: 2,+ }+}++/// Create the docs + defs indices on `url`, bulk them, and snapshot+/// `<prefix>-*` into `location`. The docs records carry `ax_path` / `line` /+/// `symbols` the way the autoindex code family does, so How and Files answers+/// have something real to rank and cite.+fn seed_and_snapshot(url: &str, location: &Path) {+ let node = HubNode::connect_insecure(url).unwrap();+ node.request_ok(+ "PUT",+ &format!("/{PREFIX}-docs"),+ Some(&json!({"mappings":{"properties":{+ "body":{"type":"text"},+ "ax_path":{"type":"keyword"},+ "line":{"type":"integer"}+ }}})),+ )+ .unwrap();+ let docs = [+ json!({+ "ax_path": "mini-c/limits.h",+ "body": "/* the clamp ceiling */\n#define MINI_LIMIT 128\n",+ "symbols": [{"name": "MINI_LIMIT", "kind": "define", "line": 2}]+ }),+ json!({+ "ax_path": "mini-c/hash.c", "line": 1,+ "body": "unsigned mini_hash(const char *s) { return MINI_HASH_SEED; }\n"+ }),+ ];+ let mut ndjson = String::new();+ for d in &docs {+ ndjson.push_str("{\"index\":{}}\n");+ ndjson.push_str(&d.to_string());+ ndjson.push('\n');+ }+ node.bulk_ndjson(&format!("/{PREFIX}-docs/_bulk"), ndjson)+ .unwrap();++ node.request_ok(+ "PUT",+ &format!("/{PREFIX}-defs"),+ Some(&json!({"mappings":{"properties":{+ "symbol":{"type":"keyword"},+ "symbol_text":{"type":"text"},+ "kind":{"type":"keyword"},+ "lang":{"type":"keyword"},+ "file":{"type":"keyword"},+ "line":{"type":"integer"},+ "text":{"type":"text"},+ "value":{"type":"keyword"}+ }}})),+ )+ .unwrap();+ let mut ndjson = String::new();+ for d in sample_defs() {+ ndjson.push_str("{\"index\":{}}\n");+ ndjson.push_str(+ &json!({+ "symbol": d.symbol, "symbol_text": d.symbol, "kind": d.kind,+ "lang": d.lang, "file": d.file, "line": d.line, "text": d.text, "value": d.value,+ })+ .to_string(),+ );+ ndjson.push('\n');+ }+ node.bulk_ndjson(&format!("/{PREFIX}-defs/_bulk"), ndjson)+ .unwrap();+ node.request_ok("POST", &format!("/{PREFIX}-defs/_refresh"), None)+ .ok();+ node.request_ok("POST", &format!("/{PREFIX}-docs/_refresh"), None)+ .ok();++ node.request_ok(+ "PUT",+ "/_snapshot/hub-build",+ Some(&json!({"type":"fs","settings":{"location": location.to_string_lossy()}})),+ )+ .unwrap();+ node.request_ok(+ "PUT",+ &format!("/_snapshot/hub-build/mini-c-{PIN8}?wait_for_completion=true"),+ Some(&json!({"indices": format!("{PREFIX}-*")})),+ )+ .unwrap();+}++fn sha256_hex(path: &Path) -> String {+ Sha256::digest(std::fs::read(path).unwrap())+ .iter()+ .map(|b| format!("{b:02x}"))+ .collect()+}++/// A scenario's own copy of the pack, "published" at a `file://` URL, and the+/// catalogue that advertises it, written into `hub_home`.+struct Published {+ archive: PathBuf,+ sig: PathBuf,+}++fn publish(src_archive: &Path, src_sig: &Path, dir: &Path, hub_home: &Path) -> Published {+ std::fs::create_dir_all(dir).unwrap();+ let archive = dir.join(format!("mini-c-{PIN8}.hubpack.tar.zst"));+ let sig = dir.join(format!("mini-c-{PIN8}.hubpack.tar.zst.sig"));+ std::fs::copy(src_archive, &archive).unwrap();+ std::fs::copy(src_sig, &sig).unwrap();+ let entry = CatalogueEntry {+ slug: "mini-c".into(),+ domain: "DEV".into(),+ topics: vec!["c".into()],+ ask_for: "a tiny C fixture corpus".into(),+ pin_sha: PIN_SHA.into(),+ pin_date: PIN_DATE.into(),+ refresh_cadence_days: 7,+ licence_spdx: "MIT".into(),+ licence_class: LicenceClass::AdaptWithAttribution,+ pack_bytes: std::fs::metadata(&archive).unwrap().len(),+ pack_url: format!("file://{}", archive.display()),+ pack_sha256: sha256_hex(&archive),+ sig_url: format!("file://{}", sig.display()),+ dependency_names: vec![],+ quality_grade: None,+ term_sketch: Sketch {+ m_bits: 64,+ k_hashes: 7,+ bits_base64: "AAAAAAAAAAA=".into(),+ },+ };+ let cat = Catalogue {+ format_version: CATALOGUE_FORMAT_VERSION,+ generated_at: "2026-10-14T00:00:00Z".into(),+ corpora: vec![entry],+ };+ std::fs::create_dir_all(hub_home).unwrap();+ std::fs::write(+ hub_home.join("catalogue.json"),+ serde_json::to_vec(&cat).unwrap(),+ )+ .unwrap();+ Published { archive, sig }+}++/// Point the process-global environment at one scenario's directories. `HOME` is+/// a scratch directory so no developer-local admin key is ever discovered.+fn set_env(hub: &Path, code: &Path, home: &Path, public: &str) {+ std::env::set_var("XERJ_HUB_HOME", hub);+ std::env::set_var("XERJ_CODE_HOME", code);+ std::env::set_var("XERJ_HUB_PUBKEY", public);+ std::env::set_var("HOME", home);+ std::env::remove_var("XERJ_URL");+ std::env::remove_var("XERJ_API_KEY");+}++struct Case {+ hub: PathBuf,+ cwd: PathBuf,+ url: Option<String>,+ launcher: NodeLauncher,+ hosted: HostedKind,+ args: AskArgs,+}++/// Run one `ask` on a blocking thread (the in-process nodes need the runtime's+/// workers free to answer the blocking HTTP calls).+async fn run(case: Case) -> AskOutcome {+ tokio::task::spawn_blocking(move || {+ let catalogue = load_catalogue("https://hub.invalid", &case.hub, true).expect("catalogue");+ let backend = RealBackend::new(case.hub.clone(), case.url.clone(), case.launcher.clone());+ let hosted: Option<Box<dyn HostedHub>> = match case.hosted {+ HostedKind::None => None,+ HostedKind::Canned => Some(Box::new(CannedHosted)),+ HostedKind::Forbidden => Some(Box::new(ForbiddenHosted)),+ };+ ask(+ &case.args,+ &AskDeps {+ hub_home: &case.hub,+ cwd: &case.cwd,+ catalogue: &catalogue,+ hosted: hosted.as_deref(),+ backend: &backend,+ },+ )+ })+ .await+ .unwrap()+}++fn def_args() -> AskArgs {+ let mut a = AskArgs::new("MINI_LIMIT");+ a.corpus = Some("mini-c".to_string());+ a.mode = ModeRequest::Def("MINI_LIMIT".to_string());+ a+}++/// `node::probe` builds a blocking HTTP client, which must not live on an async+/// worker thread.+async fn probe(url: String, timeout: Duration) -> Option<node::Health> {+ tokio::task::spawn_blocking(move || node::probe(&url, timeout))+ .await+ .unwrap()+}++async fn wait_for(what: &str, timeout: Duration, mut done: impl FnMut() -> bool) {+ let deadline = Instant::now() + timeout;+ while !done() {+ assert!(Instant::now() < deadline, "timed out waiting for {what}");+ tokio::time::sleep(Duration::from_millis(150)).await;+ }+}++// ── the scenarios ────────────────────────────────────────────────────────────++// The process environment is global: this must stay the only test here that sets it.+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]+async fn ask_end_to_end() {+ let work_tmp = tempfile::tempdir().unwrap();+ let work = std::fs::canonicalize(work_tmp.path()).unwrap();+ let allowlist = vec![work.to_string_lossy().into_owned()];+ let scratch_home = work.join("home");+ std::fs::create_dir_all(&scratch_home).unwrap();+ let cwd = work.join("cwd");+ std::fs::create_dir_all(&cwd).unwrap();+ let (seed, public) = generate_keypair().unwrap();++ // Build the pack once, on a builder node.+ let out = work.join("out");+ std::fs::create_dir_all(&out).unwrap();+ let url_a = spawn_node(&work.join("data-a"), allowlist.clone()).await;+ let location = out.join("mini-c").join("snapshot");+ {+ let (url_a, location) = (url_a.clone(), location.clone());+ tokio::task::spawn_blocking(move || seed_and_snapshot(&url_a, &location))+ .await+ .unwrap();+ }+ let archive = out.join(format!("mini-c-{PIN8}.hubpack.tar.zst"));+ {+ let files = vec![+ PackFileEntry {+ path: "limits.h".into(),+ lang: "c".into(),+ bytes: 300,+ licence: "MIT".into(),+ },+ PackFileEntry {+ path: "hash.c".into(),+ lang: "c".into(),+ bytes: 200,+ licence: "MIT".into(),+ },+ ];+ let (archive, location, seed) = (archive.clone(), location.clone(), seed.clone());+ tokio::task::spawn_blocking(move || {+ pack::write_pack(+ &archive,+ &manifest(),+ &sample_defs(),+ &files,+ &location,+ &seed,+ )+ .unwrap();+ })+ .await+ .unwrap();+ }+ let sig = out.join(format!("mini-c-{PIN8}.hubpack.tar.zst.sig"));+ assert!(+ archive.exists() && sig.exists(),+ "pack and signature written"+ );++ // ── A. no hosted hub: pull on first use, then answer from the cache ──────+ {+ let hub = work.join("hub-a");+ let code = work.join("code-a");+ let published = publish(&archive, &sig, &work.join("net-a"), &hub);+ set_env(&hub, &code, &scratch_home, &public);+ let url_b = spawn_node(&work.join("data-b"), allowlist.clone()).await;+ let case = |hosted, args| Case {+ hub: hub.clone(),+ cwd: cwd.clone(),+ url: Some(url_b.clone()),+ launcher: forbidden_launcher(),+ hosted,+ args,+ };++ let first = run(case(HostedKind::None, def_args())).await;+ assert_eq!(first.exit, EXIT_OK, "stderr: {:?}", first.stderr);+ assert!(first.stdout.contains("value: 128"), "{}", first.stdout);+ assert!(+ first.stdout.contains("answered from: local pack"),+ "{}",+ first.stdout+ );+ assert!(+ first.stdout.starts_with("mini-c pin 2026-10-02 ("),+ "{}",+ first.stdout+ );+ let pack_dir = hub.join("packs").join("mini-c").join(PIN8);+ assert!(pack_dir.join("pack.json").is_file(), "the pack is cached");+ assert!(+ pack_dir.join("last_used").is_file(),+ "every answer touches last_used"+ );+ assert!(+ code.join("state").join("mini-c.json").is_file(),+ "the ledger is written"+ );++ // The "network" goes away: a second question cannot be pulling anything,+ // and the hosted hub must not be asked either.+ std::fs::remove_file(&published.archive).unwrap();+ std::fs::remove_file(&published.sig).unwrap();+ let again = run(case(HostedKind::Forbidden, def_args())).await;+ assert_eq!(again.exit, EXIT_OK, "stderr: {:?}", again.stderr);+ assert_eq!(+ again.answer.as_ref().unwrap().answered_from,+ AnsweredFrom::LocalPack+ );+ assert!(again.stdout.contains("value: 128"));++ // A how-does-it-work question and a file listing come off the same pack.+ let mut how = AskArgs::new("MINI_LIMIT clamp ceiling");+ how.corpus = Some("mini-c".to_string());+ how.mode = ModeRequest::How;+ let how = run(case(HostedKind::Forbidden, how)).await;+ assert_eq!(+ how.exit, EXIT_OK,+ "stderr: {:?}\nstdout: {}",+ how.stderr, how.stdout+ );+ let ans = how.answer.unwrap();+ assert_eq!(ans.mode, AnswerMode::How);+ assert_eq!(ans.hits[0].file, "mini-c/limits.h");+ assert_eq!(ans.hits[0].kind, "define", "the matching definition");+ assert_eq!(ans.hits[0].line, 2, "the definition's line in the record");+ assert!(ans.hits[0].text.contains("MINI_LIMIT"));++ let mut files = AskArgs::new("unsigned return");+ files.corpus = Some("mini-c".to_string());+ files.mode = ModeRequest::Files;+ let files = run(case(HostedKind::Forbidden, files)).await;+ assert_eq!(files.exit, EXIT_OK, "stderr: {:?}", files.stderr);+ let ans = files.answer.unwrap();+ assert_eq!(ans.mode, AnswerMode::Files);+ assert!(ans+ .hits+ .iter()+ .all(|h| h.kind == "file" && h.text == "licence: MIT"));+ assert!(+ ans.hits.iter().any(|h| h.file == "mini-c/hash.c"),+ "{:?}",+ ans.hits+ );++ // `--max-tokens` is honoured on a real answer: the first hit is kept,+ // its text shortened, and the answer says it was cut.+ let mut capped = def_args();+ capped.max_tokens = Some(10);+ let capped = run(case(HostedKind::None, capped)).await;+ assert_eq!(capped.exit, EXIT_OK, "stderr: {:?}", capped.stderr);+ let ans = capped.answer.unwrap();+ assert!(ans.truncated, "{ans:?}");+ assert!(ans.hits[0].text.ends_with('…'), "{:?}", ans.hits[0].text);+ }++ // ── B. hosted answers now, a detached pull completes behind it ───────────+ {+ let hub = work.join("hub-b");+ let code = work.join("code-b");+ publish(&archive, &sig, &work.join("net-b"), &hub);+ set_env(&hub, &code, &scratch_home, &public);+ let url = spawn_node(&work.join("data-b2"), allowlist.clone()).await;+ let case = |hosted, launcher| Case {+ hub: hub.clone(),+ cwd: cwd.clone(),+ url: Some(url.clone()),+ launcher,+ hosted,+ args: def_args(),+ };++ let first = run(case(HostedKind::Canned, test_exe_launcher())).await;+ assert_eq!(first.exit, EXIT_OK, "stderr: {:?}", first.stderr);+ assert_eq!(+ first.answer.as_ref().unwrap().answered_from,+ AnsweredFrom::Hosted+ );+ assert!(+ first.stdout.contains("answered from: hosted hub"),+ "{}",+ first.stdout+ );+ assert!(+ first.stderr.iter().any(|l| l.contains("background")),+ "{:?}",+ first.stderr+ );+ assert!(+ hub.join("pulls").join("mini-c.pid").is_file(),+ "the pull's PID is recorded"+ );+ // The pull finishes on its own: the ledger is written last.+ let ledger = code.join("state").join("mini-c.json");+ wait_for("the background pull", Duration::from_secs(90), || {+ ledger.is_file()+ })+ .await;+ assert!(+ hub.join("packs")+ .join("mini-c")+ .join(PIN8)+ .join("pack.json")+ .is_file(),+ "log: {}",+ std::fs::read_to_string(hub.join("pulls").join("mini-c.log")).unwrap_or_default()+ );++ let second = run(case(HostedKind::Forbidden, forbidden_launcher())).await;+ assert_eq!(second.exit, EXIT_OK, "stderr: {:?}", second.stderr);+ assert_eq!(+ second.answer.unwrap().answered_from,+ AnsweredFrom::LocalPack+ );+ }++ // ── C. the automatic node: start, pull into it, answer, stop exactly it ──+ #[cfg(unix)]+ {+ let hub = work.join("hub-c");+ let code = work.join("code-c");+ publish(&archive, &sig, &work.join("net-c"), &hub);+ set_env(&hub, &code, &scratch_home, &public);+ // A loopback port with nothing on it: the requested node is "down".+ let silent = std::net::TcpListener::bind("127.0.0.1:0")+ .unwrap()+ .local_addr()+ .unwrap()+ .port();+ let silent_url = format!("http://127.0.0.1:{silent}");+ // A second process of the same executable that is NOT the node.+ let decoy = Command::new(std::env::current_exe().unwrap())+ .args([+ "--ignored",+ "--exact",+ "child_main",+ "--nocapture",+ "--",+ "decoy",+ ])+ .stdin(Stdio::null())+ .stdout(Stdio::null())+ .stderr(Stdio::null())+ .spawn()+ .unwrap();+ let decoy_pid = decoy.id();+ let mut reaper = Reaper {+ hub: hub.clone(),+ decoy: Some(decoy),+ };++ let out = run(Case {+ hub: hub.clone(),+ cwd: cwd.clone(),+ url: Some(silent_url.clone()),+ launcher: test_exe_launcher(),+ hosted: HostedKind::None,+ args: def_args(),+ })+ .await;+ assert_eq!(out.exit, EXIT_OK, "stderr: {:?}", out.stderr);+ assert!(out.stdout.contains("value: 128"), "{}", out.stdout);++ let pid: u32 = std::fs::read_to_string(hub.join("node.pid"))+ .expect("node.pid is recorded")+ .trim()+ .parse()+ .unwrap();+ let port: u16 = std::fs::read_to_string(hub.join("node.port"))+ .expect("node.port is recorded")+ .trim()+ .parse()+ .unwrap();+ assert!(port >= 9300, "auto node port {port} must be >= 9300");+ assert_ne!(pid, decoy_pid);+ assert!(+ probe(format!("http://127.0.0.1:{port}"), Duration::from_secs(2))+ .await+ .is_some(),+ "the recorded node answers"+ );+ // Its flags and config are what `ask` promised.+ let cfg = std::fs::read_to_string(hub.join("node.toml")).unwrap();+ assert!(cfg.contains("snapshot_repo_allowlist"), "{cfg}");+ assert!(+ cfg.contains(&hub.join("packs").to_string_lossy().into_owned()),+ "{cfg}"+ );++ // A second question finds the recorded node and starts nothing.+ let again = run(Case {+ hub: hub.clone(),+ cwd: cwd.clone(),+ url: Some(silent_url),+ launcher: forbidden_launcher(),+ hosted: HostedKind::Forbidden,+ args: def_args(),+ })+ .await;+ assert_eq!(again.exit, EXIT_OK, "stderr: {:?}", again.stderr);+ assert_eq!(+ std::fs::read_to_string(hub.join("node.pid"))+ .unwrap()+ .trim(),+ pid.to_string(),+ "the same node, not a new one"+ );++ // `hub stop` stops exactly the recorded PID.+ let hub_stop = hub.clone();+ let outcome = tokio::task::spawn_blocking(move || node::stop_node(&hub_stop))+ .await+ .unwrap()+ .unwrap();+ assert_eq!(outcome, StopOutcome::Stopped { pid });+ assert!(+ probe(format!("http://127.0.0.1:{port}"), Duration::from_secs(1))+ .await+ .is_none(),+ "the node is gone"+ );+ let decoy = reaper.decoy.as_mut().unwrap();+ assert!(+ decoy.try_wait().unwrap().is_none(),+ "the look-alike process survives"+ );+ assert!(!hub.join("node.pid").exists() && !hub.join("node.port").exists());+ }++ // ── D. a pull evicts the least recently used pack that busts the budget ──+ #[cfg(unix)]+ {+ let hub = work.join("hub-d");+ let code = work.join("code-d");+ publish(&archive, &sig, &work.join("net-d"), &hub);+ set_env(&hub, &code, &scratch_home, &public);+ // Two other packs, each a 3 GiB sparse file (no disk is written): over the+ // 4 GiB budget together. "stale" was used longest ago.+ for (slug, used_ago) in [("stale", 90_000u64), ("recent", 60)] {+ let dir = hub.join("packs").join(slug).join("eeee0001");+ std::fs::create_dir_all(&dir).unwrap();+ std::fs::write(dir.join("pack.json"), "{}").unwrap();+ let big = std::fs::File::create(dir.join("big")).unwrap();+ big.set_len(3 * 1024 * 1024 * 1024).unwrap();+ let used = dir.join("last_used");+ std::fs::write(&used, "x").unwrap();+ std::fs::File::options()+ .write(true)+ .open(&used)+ .unwrap()+ .set_modified(std::time::SystemTime::now() - Duration::from_secs(used_ago))+ .unwrap();+ }+ let url = spawn_node(&work.join("data-d"), allowlist.clone()).await;+ let out = run(Case {+ hub: hub.clone(),+ cwd: cwd.clone(),+ url: Some(url),+ launcher: forbidden_launcher(),+ hosted: HostedKind::None,+ args: def_args(),+ })+ .await;+ assert_eq!(out.exit, EXIT_OK, "stderr: {:?}", out.stderr);+ assert!(+ !hub.join("packs").join("stale").exists(),+ "the least recently used pack is evicted"+ );+ assert!(+ hub.join("packs").join("recent").join("eeee0001").exists(),+ "the more recently used pack stays: evicting one was enough"+ );+ assert!(hub+ .join("packs")+ .join("mini-c")+ .join(PIN8)+ .join("pack.json")+ .is_file());+ }++ // ── E. offline, nothing cached: exit 1, and no node is started to say so ─+ {+ let hub = work.join("hub-e");+ let code = work.join("code-e");+ publish(&archive, &sig, &work.join("net-e"), &hub);+ set_env(&hub, &code, &scratch_home, &public);+ let mut args = def_args();+ args.offline = true;+ let out = run(Case {+ hub: hub.clone(),+ cwd: cwd.clone(),+ url: Some("http://127.0.0.1:1".to_string()),+ launcher: forbidden_launcher(),+ hosted: HostedKind::Forbidden,+ args,+ })+ .await;+ assert_eq!(out.exit, EXIT_NO_ANSWER);+ let msg = out.stderr.join("\n");+ assert!(msg.contains("--offline"), "{msg}");+ assert!(msg.contains("run: xerj hub pull mini-c"), "{msg}");+ assert!(!hub.join("node.pid").exists(), "no node was started");+ assert!(!hub.join("packs").exists(), "nothing was downloaded");+ }+}diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex 4b297ef89..a2f5cb71e 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -228,6 +228,10 @@ fn help_text(feedback: bool) -> String {of peer projects (see xerj code --help)\n\xerj corpus add|index|list the corpus lifecycle behind `xerj code`\n\(see xerj corpus --help)\n\+ xerj ask \"<question>\" answer a question from a hub corpus: the\n\+ pre-indexed pack is pulled the first time\n\+ (see xerj ask --help); `xerj \"<question>\"`\n\+ is the same\n\xerj index [OPTIONS] direct NDJSON → engine ingest (see xerj\n\index --help)\n\xerj brain <folder> index + browse in one command (see xerj\n\@@ -2322,6 +2326,25 @@ async fn async_main() -> Result<()> {.unwrap_or(1);std::process::exit(code);}+ if matches!(argv1.as_deref(), Some("ask")) {+ // Corpus Hub v2 answer flow (plan §5): route the question to a hub+ // corpus, answer from its pack (pulling it the first time). Blocking+ // HTTP + process spawn — off the runtime like `hub`.+ let args: Vec<String> = std::env::args().skip(2).collect();+ let code = tokio::task::spawn_blocking(move || xerj_hub::cli::run_ask(&args))+ .await+ .unwrap_or(1);+ std::process::exit(code);+ }+ if xerj_hub::cli::is_bare_question(argv1.as_deref()) {+ // `xerj "how does lz4 frame work"`: argv[1] is not a subcommand, does+ // not start with `-`, and contains a space — the same as `xerj ask`.+ let args: Vec<String> = std::env::args().skip(1).collect();+ let code = tokio::task::spawn_blocking(move || xerj_hub::cli::run_ask(&args))+ .await+ .unwrap_or(1);+ std::process::exit(code);+ }// Start profiling before config loading, TLS, engine replay, and router// construction so delay=0 is a genuine initialization profile.