AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(hub): the xerj_hub_find / xerj_hub_def / xerj_hub_ask MCP tools (plan §7)
xerj-orga5a04b9b75ec
diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 0f92e77cf..d4f82a7c7 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -7693,12 +7693,16 @@ name = "xerj-mcp"version = "1.0.0-rc.95"dependencies = ["anyhow",+ "axum","reqwest","serde_json","tempfile","tokio","unicode-segmentation",+ "xerj-api","xerj-common",+ "xerj-engine",+ "xerj-hub","xerj-rerank",]diff --git a/engine/crates/xerj-autoindex/src/init.rs b/engine/crates/xerj-autoindex/src/init.rsindex e8de9657f..fdfe4c5a6 100644--- a/engine/crates/xerj-autoindex/src/init.rs+++ b/engine/crates/xerj-autoindex/src/init.rs@@ -160,9 +160,13 @@ sent to the hosted hub. `xerj hub config hosted off` forbids that.## MCP-`xerj mcp` in this build has no hub tool: ask the hub with the shell line above. Through MCP you-get the local-corpus tools (`xerj_search`, `xerj_code_search` and the rest). If your tool list names-a tool that starts with `xerj_hub_`, your build is newer than this file: prefer that tool.+`xerj mcp` serves three hub tools. They run the same flow as `xerj ask`:+- `xerj_hub_ask` `{question, corpus?, mode?, max_tokens?}`: the shell line above, as one call.+- `xerj_hub_def` `{slug, symbol, k?}`: `xerj ask --corpus <slug> --def <SYMBOL>`.+- `xerj_hub_find` `{question, k?}`: which corpora could answer, best first. No passages.++A hub tool answers only from packs already on this machine; it never downloads one. When a corpus+is not here, the answer is the `xerj hub pull <slug>` line: run it in a shell, then ask again.Register `xerj mcp` once, then restart the session. Success: `xerj_search` is in the tool list.Use the absolute path to the binary: an editor started from a desktop icon does not inherit your@@ -179,9 +183,9 @@ shell PATH. `command -v xerj` prints it (PowerShell: `(Get-Command xerj).Source`"##;/// The MCP tool whose presence proves `xerj mcp` is registered. It must be a-/// tool `xerj mcp` serves today: the hub tools (`xerj_hub_*`, plan step 7) are-/// not implemented yet, so naming one here would make a correct registration-/// look like a failure.+/// tool every `xerj mcp` serves. The hub tools (`xerj_hub_*`) are served too,+/// but `xerj_search` is the proof the skill and the printed text have always+/// used, and it stays the one check.const MCP_SUCCESS_TOOL: &str = "xerj_search";/// The default node URL; the printed Claude Code line carries `-e XERJ_URL=`@@ -796,18 +800,37 @@ mod tests {assert_eq!(shell_quote("/it's/xerj"), r"'/it'\''s/xerj'");}- /// `xerj mcp` serves no `xerj_hub_*` tool yet (plan step 7 is deferred;- /// docs/hub-v2-grpc-mcp-notes.md). Text init installs or prints must not- /// send an agent to one, nor make one the proof that registration worked.- /// When step 7 lands, update this test together with the skill.+ /// The hub tools `xerj mcp` serves (plan step 7; `xerj-mcp/src/hub.rs`,+ /// whose own test pins the same three names to `tool_specs()`). The skill+ /// names each, says a tool call never downloads a pack, and promises no+ /// other `xerj_hub_*` tool; the printed registration still proves itself+ /// with `xerj_search`.#[test]- fn no_unserved_hub_mcp_tool_is_promised() {- let printed = mcp_registration("/usr/local/bin/xerj", DEFAULT_URL);- for name in ["xerj_hub_ask", "xerj_hub_def", "xerj_hub_find"] {- assert!(!HUB_SKILL_MD.contains(name), "skill names {name}");- assert!(!printed.contains(name), "printed text names {name}");+ fn the_hub_skill_names_exactly_the_hub_tools_mcp_serves() {+ const SERVED: [&str; 3] = ["xerj_hub_ask", "xerj_hub_def", "xerj_hub_find"];+ for name in SERVED {+ assert!(+ HUB_SKILL_MD.contains(name),+ "the skill does not name {name}"+ );}- assert!(HUB_SKILL_MD.contains("has no hub tool"));+ let mut rest = HUB_SKILL_MD;+ while let Some(at) = rest.find("xerj_hub_") {+ let tail = &rest[at..];+ let end = tail+ .find(|c: char| !(c.is_ascii_alphanumeric() || c == '_'))+ .unwrap_or(tail.len());+ let named = &tail[..end];+ assert!(+ SERVED.contains(&named),+ "the skill names an unserved tool {named}"+ );+ rest = &tail[end..];+ }+ assert!(HUB_SKILL_MD.contains("never downloads one"));+ assert!(!HUB_SKILL_MD.contains("has no hub tool"));+ let printed = mcp_registration("/usr/local/bin/xerj", DEFAULT_URL);+ assert!(printed.contains("xerj_search"));}/// `--help` lists every file init can write, the hub pair included.diff --git a/engine/crates/xerj-mcp/Cargo.toml b/engine/crates/xerj-mcp/Cargo.tomlindex 17108a871..d6dbbf102 100644--- a/engine/crates/xerj-mcp/Cargo.toml+++ b/engine/crates/xerj-mcp/Cargo.toml@@ -4,7 +4,7 @@ version.workspace = trueedition.workspace = truelicense.workspace = trueauthors.workspace = true-description = "Model Context Protocol (MCP) stdio server exposing the 12 canonical XERJ agent operations (search, memory, second-brain graph, field map, reference-code) as thin proxies to an ES-compatible XERJ endpoint"+description = "Model Context Protocol (MCP) stdio server exposing the canonical XERJ agent operations (search, memory, second-brain graph, field map, reference-code) as thin proxies to an ES-compatible XERJ endpoint, plus the Corpus Hub ask/def/find tools"# Library first: the `xerj mcp` subcommand on the main binary calls# `xerj_mcp::run`, so a user with the installed `xerj` binary already has the@@ -45,9 +45,18 @@ xerj-rerank = { path = "../xerj-rerank" }# stack into this thin proxy; the count is identical by construction.# See src/token_budget.rs.unicode-segmentation = "1"+# Corpus Hub v2 (plan §7): the `xerj_hub_*` tools run the `xerj ask` flow+# itself (`xerj_hub::ask`), the one implementation the CLI and the gRPC+# service share — the tools are adapters, not a second copy of the flow.+xerj-hub = { path = "../xerj-hub" }[dev-dependencies]# The loopback admin-key fallback tests sandbox cwd + HOME so discovery sees# only the key files the test wrote (same pattern as xerj-autoindex's cli# tests, where the helper was born).tempfile = { workspace = true }+# The hub tools' end-to-end test answers from a real in-process node holding a+# restored fixture corpus (the keystone's mini-c), as `xerj-hub`'s own tests do.+xerj-api = { path = "../xerj-api" }+xerj-engine = { path = "../xerj-engine" }+axum.workspace = truediff --git a/engine/crates/xerj-mcp/src/hub.rs b/engine/crates/xerj-mcp/src/hub.rsnew file mode 100644index 000000000..48a65d649--- /dev/null+++ b/engine/crates/xerj-mcp/src/hub.rs@@ -0,0 +1,354 @@+//! Corpus Hub v2 — the `xerj_hub_*` MCP tools (plan §7,+//! `docs/CORPUS_HUB_V2_IMPLEMENTATION.md`; specs in+//! `docs/hub-v2-grpc-mcp-notes.md`).+//!+//! Thin adapters over the code the CLI runs — no hub logic lives here:+//!+//! | tool | runs | which is also |+//! |---|---|---|+//! | `xerj_hub_ask` | [`xerj_hub::ask::run_ask_with`] | `xerj ask "<q>" [--corpus] [--def\|--how\|--files] [--max-tokens]` |+//! | `xerj_hub_def` | [`xerj_hub::ask::run_ask_with`] with `--corpus <slug> --def <SYMBOL>` | `xerj ask --corpus <slug> --def <SYMBOL>` |+//! | `xerj_hub_find` | [`xerj_hub::ask::run_find`] | the routing half of `xerj ask` (what it would pick, and the runners-up) |+//!+//! The answer flow is the CLI's: local packs first, hosted second (this build+//! has no hosted client, so "hosted" is unavailable exactly as it is for the+//! CLI), `XERJ_HUB_URL` overrides the catalogue's base URL, and the node is the+//! one this server proxies to (`--url` / `XERJ_URL`), or the automatic hub node.+//! The tool result text is the §1 rendering with the flow's notices on top (MCP+//! has no stderr); the structured content is the [`xerj_hub::Answer`] JSON.+//!+//! One difference, on purpose: a tool call never downloads a pack. The flow+//! runs with `no_fetch`, so a corpus that is not on this machine is answered+//! with the `xerj hub pull <slug>` line instead — a tool call cannot start a+//! download, and so cannot trigger the pull's least-recently-used eviction (a+//! delete) behind an agent's back. Excluded corpora are never pulled or served+//! by the flow itself; approach-only answers carry the licence warning on every+//! hit.++use serde_json::{json, Value};+use xerj_hub::ask::{+ self, AskArgs, AskOutcome, ModeRequest, EXIT_NEEDS_DECISION, FIND_DEFAULT_K, FIND_MAX_K,+};+use xerj_hub::node::NodeLauncher;++use crate::{tool_text, Ctx};++/// The longest question or symbol a tool takes — the hosted routes' limit.+pub const MAX_INPUT_BYTES: usize = 512;+/// `k` for `xerj_hub_def` is clamped to this (the `def` maximum).+const MAX_K: u64 = 50;+/// The largest `max_tokens` accepted (the hosted `code` route's cap).+const MAX_TOKENS_CAP: u64 = 8192;++/// The three tool specs, in the order `tools/list` serves them.+pub fn specs() -> Vec<Value> {+ vec![+ json!({+ "name": "xerj_hub_find",+ "description": concat!(+ "Discover which hub corpora can answer a question. Returns candidate corpora ",+ "with their pin, licence and whether the pack is already on this machine — not ",+ "passages; ask one with xerj_hub_ask. Routed exactly as `xerj ask` routes: your ",+ "project's dependencies, the signed catalogue's term sketches, and the exact ",+ "symbol a definition question names (offline; this build has no hosted search). ",+ "Excluded-licence corpora are never listed."),+ "inputSchema": {+ "type": "object",+ "properties": {+ "question": {+ "type": "string",+ "description": "What the agent is working on; a corpus name is never required here."+ },+ "k": {+ "type": "integer",+ "minimum": 1,+ "maximum": FIND_MAX_K,+ "description": format!(+ "Max candidate corpora to return (default {FIND_DEFAULT_K}, at most {FIND_MAX_K})."+ )+ }+ },+ "required": ["question"]+ }+ }),+ json!({+ "name": "xerj_hub_def",+ "description": concat!(+ "Look up one definition in a named hub corpus: its kind, signature, file:line, and ",+ "the literal value for a const/macro/define. Returns an Answer in def mode (no ",+ "passages). Same as `xerj ask --corpus <slug> --def <SYMBOL>`: answered from the ",+ "local pack; a corpus not on this machine is never downloaded by this tool — the ",+ "answer says which `xerj hub pull` line fetches it. Approach-only corpora carry a ",+ "read-do-not-copy warning on every hit."),+ "inputSchema": {+ "type": "object",+ "properties": {+ "slug": { "type": "string", "description": "Corpus slug to resolve the symbol in." },+ "symbol": { "type": "string", "description": "Exact definition name." },+ "k": {+ "type": "integer",+ "minimum": 1,+ "maximum": MAX_K,+ "description": "Max hits to return (default 5)."+ }+ },+ "required": ["slug", "symbol"]+ }+ }),+ json!({+ "name": "xerj_hub_ask",+ "description": concat!(+ "Ask a hub corpus a question in one turn — no clone, no local indexing. Local packs ",+ "first, hosted second, exactly like `xerj ask`. The corpus is optional (routed by ",+ "the offline sketch when omitted) and is always named in the answer; when more than ",+ "one corpus fits, the candidates come back and you re-ask with `corpus`. Never ",+ "downloads a pack: a corpus not on this machine is answered with the `xerj hub ",+ "pull` line to run. Result text is the hub rendering (corpus, pin date, licence, ",+ "file:line); structured content is the Answer JSON."),+ "inputSchema": {+ "type": "object",+ "properties": {+ "question": { "type": "string", "description": "Natural-language question." },+ "corpus": { "type": "string", "description": "Optional explicit corpus slug; omit to route by sketch." },+ "mode": {+ "type": "string",+ "enum": ["def", "how", "files"],+ "description": "Answer shape; omit for auto (a lone identifier or 'what is the value of X' is def)."+ },+ "max_tokens": {+ "type": "integer",+ "minimum": 1,+ "maximum": MAX_TOKENS_CAP,+ "description": "Soft output budget for the rendered answer (about 4 characters a token)."+ }+ },+ "required": ["question"]+ }+ }),+ ]+}++/// Dispatch one `xerj_hub_*` call; `None` when `name` is not a hub tool.+pub async fn call(ctx: &Ctx, name: &str, args: &Value) -> Option<Value> {+ let result = match name {+ "xerj_hub_find" => find(args).await,+ "xerj_hub_def" => def(ctx, args).await,+ "xerj_hub_ask" => ask_tool(ctx, args).await,+ _ => return None,+ };+ Some(result.unwrap_or_else(|msg| tool_text(format!("{name}: {msg}"), true)))+}++// ── argument checks ──────────────────────────────────────────────────────────++/// A required, non-blank string of at most [`MAX_INPUT_BYTES`].+fn text_arg(args: &Value, key: &str) -> Result<String, String> {+ let raw = match args.get(key) {+ Some(Value::String(s)) => s,+ Some(_) => return Err(format!("`{key}` must be a string")),+ None => return Err(format!("`{key}` is required")),+ };+ if raw.len() > MAX_INPUT_BYTES {+ return Err(format!(+ "`{key}` is limited to {MAX_INPUT_BYTES} bytes, got {}",+ raw.len()+ ));+ }+ let t = raw.trim();+ if t.is_empty() {+ return Err(format!("`{key}` must not be blank"));+ }+ Ok(t.to_string())+}++/// A corpus slug, checked before anything is looked up or built from it.+fn slug_arg(args: &Value, key: &str, required: bool) -> Result<Option<String>, String> {+ match args.get(key) {+ None | Some(Value::Null) if !required => Ok(None),+ None | Some(Value::Null) => Err(format!("`{key}` is required")),+ Some(Value::String(s)) if xerj_hub::cli::valid_slug(s) => Ok(Some(s.clone())),+ Some(Value::String(s)) if s.is_empty() && !required => Ok(None),+ Some(_) => Err(format!(+ "`{key}` must be a corpus slug (lower-case letters, digits, '.', '_', '-')"+ )),+ }+}++/// An optional integer in `1..=max`.+fn int_arg(args: &Value, key: &str, max: u64) -> Result<Option<u64>, String> {+ match args.get(key) {+ None | Some(Value::Null) => Ok(None),+ Some(v) => match v.as_u64().filter(|n| (1..=max).contains(n)) {+ Some(n) => Ok(Some(n)),+ None => Err(format!("`{key}` must be an integer from 1 to {max}")),+ },+ }+}++// ── the tools ────────────────────────────────────────────────────────────────++async fn find(args: &Value) -> Result<Value, String> {+ let question = text_arg(args, "question")?;+ let k = int_arg(args, "k", FIND_MAX_K as u64)?.map_or(FIND_DEFAULT_K, |n| n as usize);+ let found = tokio::task::spawn_blocking(move || ask::run_find(&question, k))+ .await+ .map_err(|e| format!("join error: {e}"))??;+ Ok(json!({+ "content": [ { "type": "text", "text": ask::render_find(&found) } ],+ "structuredContent": found,+ "isError": false,+ }))+}++async fn def(ctx: &Ctx, args: &Value) -> Result<Value, String> {+ let slug = slug_arg(args, "slug", true)?.unwrap_or_default();+ let symbol = text_arg(args, "symbol")?;+ let k = int_arg(args, "k", MAX_K)?;+ let mut a = AskArgs::new(symbol.clone());+ a.corpus = Some(slug);+ a.mode = ModeRequest::Def(symbol);+ a.k = k.map(|n| n as usize);+ Ok(run(ctx, a).await)+}++async fn ask_tool(ctx: &Ctx, args: &Value) -> Result<Value, String> {+ let question = text_arg(args, "question")?;+ let corpus = slug_arg(args, "corpus", false)?;+ let mode = match args.get("mode") {+ None | Some(Value::Null) => ModeRequest::Auto,+ Some(Value::String(m)) => match m.as_str() {+ "how" => ModeRequest::How,+ "files" => ModeRequest::Files,+ // `def` names no symbol of its own: the one the question names+ // (`what is the value of X`), else the whole question — what+ // `xerj ask --def` would be given.+ "def" => ModeRequest::Def(+ ask::detect_mode(&question, &ModeRequest::Auto)+ .1+ .unwrap_or_else(|| question.clone()),+ ),+ other => return Err(format!("`mode` must be def, how or files, not '{other}'")),+ },+ Some(_) => return Err("`mode` must be a string".to_string()),+ };+ let max_tokens = int_arg(args, "max_tokens", MAX_TOKENS_CAP)?;+ let mut a = AskArgs::new(question);+ a.corpus = corpus;+ a.mode = mode;+ a.max_tokens = max_tokens.map(|n| n as usize);+ Ok(run(ctx, a).await)+}++/// Run the CLI's flow for `a` (never fetching) against this server's node and+/// turn the outcome into a tool result.+async fn run(ctx: &Ctx, mut a: AskArgs) -> Value {+ a.no_fetch = true;+ a.json = false;+ a.url = Some(ctx.base_url.clone());+ let launcher = match launcher() {+ Ok(l) => l,+ Err(e) => return tool_text(e, true),+ };+ match tokio::task::spawn_blocking(move || ask::run_ask_with(&a, None, launcher)).await {+ Ok(outcome) => outcome_result(outcome),+ Err(e) => tool_text(format!("join error: {e}"), true),+ }+}++/// An [`AskOutcome`] as a tool result. An answer — including one with no hits+/// (a miss is an answer, not an error) — carries the Answer JSON as structured+/// content. More than one candidate corpus is not an error either: the text+/// lists them for a re-ask with `corpus`. Anything else is `isError`.+fn outcome_result(o: AskOutcome) -> Value {+ let mut text = String::new();+ for line in &o.stderr {+ text.push_str(line);+ text.push('\n');+ }+ text.push_str(&o.stdout);+ match o.answer {+ Some(answer) => json!({+ "content": [ { "type": "text", "text": text } ],+ "structuredContent": answer,+ "isError": false,+ }),+ None => tool_text(text, o.exit != EXIT_NEEDS_DECISION),+ }+}++/// What starts the automatic hub node: this executable when it is `xerj`+/// (`xerj mcp`); for the standalone `xerj-mcp` binary — which takes no node+/// flags — the `xerj` beside it, else `xerj` on `PATH`.+fn launcher() -> Result<NodeLauncher, String> {+ let current = NodeLauncher::current().map_err(|e| format!("{e:#}"))?;+ let standalone = current+ .exe+ .file_stem()+ .is_some_and(|s| s.to_string_lossy() == "xerj-mcp");+ if !standalone {+ return Ok(current);+ }+ let sibling = current+ .exe+ .with_file_name(format!("xerj{}", std::env::consts::EXE_SUFFIX));+ Ok(NodeLauncher {+ exe: if sibling.is_file() {+ sibling+ } else {+ std::path::PathBuf::from("xerj")+ },+ pre_args: Vec::new(),+ })+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn arguments_are_refused_before_anything_runs() {+ let long = "x".repeat(MAX_INPUT_BYTES + 1);+ assert!(text_arg(&json!({ "question": long }), "question").is_err());+ assert!(text_arg(&json!({ "question": " " }), "question").is_err());+ assert!(text_arg(&json!({ "question": 7 }), "question").is_err());+ assert_eq!(+ text_arg(&json!({ "question": " q " }), "question").unwrap(),+ "q"+ );+ for bad in ["../etc", "a/b", "", "UPPER", "-lead"] {+ assert!(+ slug_arg(&json!({ "slug": bad }), "slug", true).is_err(),+ "{bad:?}"+ );+ }+ assert_eq!(+ slug_arg(&json!({ "slug": "mini-c" }), "slug", true).unwrap(),+ Some("mini-c".into())+ );+ assert_eq!(slug_arg(&json!({}), "corpus", false).unwrap(), None);+ assert!(int_arg(&json!({ "k": 0 }), "k", 50).is_err());+ assert!(int_arg(&json!({ "k": 51 }), "k", 50).is_err());+ assert!(int_arg(&json!({ "k": "5" }), "k", 50).is_err());+ assert_eq!(int_arg(&json!({ "k": 50 }), "k", 50).unwrap(), Some(50));+ }++ #[test]+ fn an_ambiguous_route_is_not_an_error_and_a_failure_is() {+ let ambiguous = AskOutcome {+ exit: EXIT_NEEDS_DECISION,+ stdout: "a — x\nb — y".into(),+ stderr: vec!["more than one corpus matches; re-run with --corpus <slug>".into()],+ answer: None,+ };+ let r = outcome_result(ambiguous);+ assert_eq!(r["isError"], false);+ assert!(r["content"][0]["text"].as_str().unwrap().contains("a — x"));+ let failed = AskOutcome {+ exit: 1,+ stdout: String::new(),+ stderr: vec!["no hub corpus matches; see xerj.org/llms.txt".into()],+ answer: None,+ };+ assert_eq!(outcome_result(failed)["isError"], true);+ }+}diff --git a/engine/crates/xerj-mcp/src/lib.rs b/engine/crates/xerj-mcp/src/lib.rsindex 05ac807c3..184dfd69b 100644--- a/engine/crates/xerj-mcp/src/lib.rs+++ b/engine/crates/xerj-mcp/src/lib.rs@@ -9,8 +9,9 @@//!//! It speaks the MCP **stdio transport**: newline-delimited//! JSON-RPC 2.0 messages on stdin/stdout. It exposes XERJ to any MCP-capable-//! agent host (Claude Desktop, IDE agents, custom orchestrators) as thirteen-//! tools that map 1:1 onto XERJ's real, verified REST surface. Every tool is+//! agent host (Claude Desktop, IDE agents, custom orchestrators) as sixteen+//! tools: thirteen that map 1:1 onto XERJ's real, verified REST surface, and+//! three Corpus Hub tools (`xerj_hub_*`) that run the `xerj ask` flow itself. Every tool is//! a *thin proxy*: it constructs exactly the request the running engine//! already accepts and forwards it to a configurable base URL.//!@@ -30,6 +31,7 @@//! | `xerj_brain_overview` | `GET /_graph/{brain}/overview` | whole-brain orientation: counts, hubs, detectors, timeline |//! | `xerj_plan` | `POST /_ask` | natural-language prompt in, VALIDATED query DSL out (deterministic planner, not an LLM) |//! | `xerj_map` | `POST /autoindex-catalog/_search` | per-index field map from the autoindex catalog (#1055) |+//! | `xerj_hub_find` / `xerj_hub_def` / `xerj_hub_ask` | the `xerj ask` flow (`src/hub.rs`) | which hub corpus answers; one definition; one question — from local packs, never downloading |//!//! ## Honesty notes (must match the engine, never oversell)//!@@ -87,6 +89,7 @@//! words), with overlapping file:line passages deduped before budgeting and//! over-budget hits reduced to locators.+mod hub;mod map;mod token_budget;@@ -123,6 +126,12 @@ XERJ node thatis ALREADY RUNNING. This command does not start a node; start one first with`xerj --data-dir ./data`.+It also serves three Corpus Hub tools — xerj_hub_find, xerj_hub_def,+xerj_hub_ask — which run the `xerj ask` flow: answered from packs already on+this machine (a tool call never downloads one; it names the `xerj hub pull`+line instead). Like `xerj ask`, they use the node above when it answers and+otherwise the automatic hub node.+OPTIONS:--url <URL> Base URL of the XERJ ES-compatible listener.Overrides XERJ_URL. Default: http://localhost:9200@@ -389,6 +398,23 @@ async fn handle_message(ctx: &Ctx, msg: Value) -> Option<Value> {}}+/// Handle one JSON-RPC message exactly as the stdio loop does, against the node+/// at `base_url` — for the end-to-end tests in `tests/`, which cannot reach the+/// private dispatch. Not part of the supported API.+#[doc(hidden)]+pub async fn handle_message_for_test(+ base_url: &str,+ auth: Option<String>,+ msg: Value,+) -> Option<Value> {+ let ctx = Ctx {+ client: reqwest::Client::new(),+ base_url: base_url.trim_end_matches('/').to_string(),+ auth,+ };+ handle_message(&ctx, msg).await+}+/// Build the `initialize` result, echoing the client's requested protocol/// version when present (our JSON-RPC handling is version-agnostic).fn initialize_result(msg: &Value) -> Value {@@ -603,7 +629,8 @@ fn max_tokens_arg_schema() -> Value {})}-/// The thirteen tool specifications advertised via `tools/list`. Input schemas+/// The sixteen tool specifications advertised via `tools/list` (the three+/// `xerj_hub_*` specs come from `hub::specs`, appended last). Input schemas/// are plain JSON Schema; every property maps onto a field the engine accepts.////// Public because this is the single source of truth for the published@@ -612,7 +639,7 @@ fn max_tokens_arg_schema() -> Value {/// once advertised six tools while the binary served ten — nobody noticed,/// because nothing compared them.pub fn tool_specs() -> Value {- json!([+ let mut specs = json!([{"name": "xerj_search","description":@@ -1055,7 +1082,11 @@ pub fn tool_specs() -> Value {"required": ["prompt"]}}- ])+ ]);+ if let Value::Array(tools) = &mut specs {+ tools.extend(hub::specs());+ }+ specs}/// Dispatch a `tools/call` to the matching proxy. Any bad-argument or transport@@ -1110,6 +1141,13 @@ async fn call_tool(ctx: &Ctx, msg: &Value) -> Value {// returns a tool result directly instead of a BuiltRequest."xerj_map" => return map::run(ctx, &args).await,"xerj_code_search" => return run_code_search(ctx, &args).await,+ // Corpus Hub v2 (plan §7): the `xerj ask` flow itself, not a proxy.+ hub_tool if hub_tool.starts_with("xerj_hub_") => {+ return match hub::call(ctx, hub_tool, &args).await {+ Some(result) => result,+ None => tool_text(format!("unknown tool: {hub_tool}"), true),+ }+ }other => return tool_text(format!("unknown tool: {other}"), true),};@@ -2391,6 +2429,14 @@ mod tests {.unwrap_or(false),"{name}: `max_tokens` is optional");+ } else if name == "xerj_hub_ask" {+ // The hub's own budget (docs/hub-v2-grpc-mcp-notes.md): the+ // `xerj ask --max-tokens` cap, a SOFT ~4-chars-a-token trim of+ // the rendered answer — not #1058's never-exceeded budget, and+ // its description must not claim to be.+ let d = prop["description"].as_str().expect("described");+ assert!(d.starts_with("Soft"), "{name}: {d}");+ assert!(!d.contains("NEVER exceeds"), "{name}: {d}");} else {assert!(prop.is_null(),@@ -2710,7 +2756,7 @@ mod tests {}#[test]- fn tools_list_has_all_thirteen() {+ fn tools_list_has_all_sixteen() {let specs = tool_specs();let names: Vec<&str> = specs.as_array()@@ -2718,7 +2764,7 @@ mod tests {.iter().map(|t| t["name"].as_str().unwrap()).collect();- assert_eq!(names.len(), 13);+ assert_eq!(names.len(), 16);for n in ["xerj_search","xerj_semantic_search",@@ -2735,8 +2781,12 @@ mod tests {// #977: reference-coding retrieval, served by the same pipeline// as `xerj code`."xerj_code_search",- // #1056: prompt in, validated DSL out (LAST in the list).+ // #1056: prompt in, validated DSL out."xerj_plan",+ // Corpus Hub v2 (plan §7), appended after the thirteen.+ "xerj_hub_find",+ "xerj_hub_def",+ "xerj_hub_ask",] {assert!(names.contains(&n), "missing tool {n}");}diff --git a/engine/crates/xerj-mcp/tests/hub_tools.rs b/engine/crates/xerj-mcp/tests/hub_tools.rsnew file mode 100644index 000000000..4ee7ba497--- /dev/null+++ b/engine/crates/xerj-mcp/tests/hub_tools.rs@@ -0,0 +1,393 @@+//! The `xerj_hub_*` MCP tools end to end (plan §7 "Tests"): JSON-RPC+//! `tools/list` and `tools/call`, the way an MCP host sends them, against a real+//! in-process node holding the keystone's `mini-c` fixture corpus as a restored+//! pack (ledger, pack manifest, cached catalogue — the layout `xerj hub pull`+//! leaves), so the tools run the real `xerj ask` flow and `RealBackend`.+//!+//! One test, scenarios in order: they share process-global environment+//! (`XERJ_HUB_HOME`, `XERJ_CODE_HOME`, `XERJ_HUB_URL`). It is its own test+//! binary so that environment never races the crate's unit tests.++use std::path::Path;++use serde_json::{json, Value};+use xerj_common::hub::{+ Catalogue, CatalogueEntry, LicenceClass, PackIndex, PackManifest, CATALOGUE_FORMAT_VERSION,+ PACK_FORMAT_VERSION,+};++const PIN_SHA: &str = "0774d05537f9762f838f7ab541b7765f1a729cb5";+const PIN8: &str = "0774d055";+const PIN_DATE: &str = "2026-10-02";++fn entry(slug: &str, spdx: &str, class: LicenceClass, terms: &[&str]) -> CatalogueEntry {+ CatalogueEntry {+ slug: slug.into(),+ domain: "DEV".into(),+ topics: vec![],+ ask_for: format!("fixture corpus {slug}"),+ pin_sha: PIN_SHA.into(),+ pin_date: PIN_DATE.into(),+ refresh_cadence_days: 7,+ licence_spdx: spdx.into(),+ licence_class: class,+ pack_bytes: 4096,+ pack_url: "https://example.invalid/never-fetched.tar.zst".into(),+ pack_sha256: "0".repeat(64),+ sig_url: "https://example.invalid/never-fetched.sig".into(),+ dependency_names: vec![],+ quality_grade: None,+ term_sketch: xerj_hub::sketch::build(terms.iter().map(|t| t.to_string())),+ }+}++/// What `xerj hub pull` leaves for a restored pack: the ledger, and+/// `<hub>/packs/<slug>/<pin8>/pack.json`.+fn restored(code: &Path, hub: &Path, url: &str, slug: &str, spdx: &str, class: &str) {+ let prefix = format!("xc-{slug}-b1");+ xerj_common::xccode::state::write_hubpack_state(+ code, slug, url, &prefix, PIN_SHA, PIN_DATE, spdx, class,+ )+ .unwrap();+ let dir = hub.join("packs").join(slug).join(PIN8);+ std::fs::create_dir_all(&dir).unwrap();+ let manifest = PackManifest {+ format_version: PACK_FORMAT_VERSION,+ slug: slug.into(),+ pin_sha: PIN_SHA.into(),+ pin_date: PIN_DATE.into(),+ built_at: "2026-10-02T00:00:00Z".into(),+ builder_version: "test".into(),+ index_prefix: prefix.clone(),+ indices: ["defs", "docs"]+ .iter()+ .map(|s| PackIndex {+ name: format!("{prefix}-{s}"),+ docs: 1,+ bytes: 1,+ })+ .collect(),+ snapshot_name: "snap".into(),+ licence_spdx: spdx.into(),+ licence_class: serde_json::from_value(json!(class)).unwrap(),+ sources: vec![],+ defs_count: 1,+ files_count: 1,+ };+ std::fs::write(+ dir.join("pack.json"),+ serde_json::to_vec(&manifest).unwrap(),+ )+ .unwrap();+}++async fn seed(client: &reqwest::Client, url: &str, slug: &str, symbol: &str, value: &str) {+ let defs = format!("{url}/xc-{slug}-b1-defs");+ let r = client+ .put(&defs)+ .json(&json!({ "mappings": { "properties": {+ "symbol": {"type": "keyword"}, "symbol_text": {"type": "text"},+ "kind": {"type": "keyword"}, "lang": {"type": "keyword"},+ "file": {"type": "keyword"}, "line": {"type": "integer"},+ "text": {"type": "text"}, "value": {"type": "keyword"}+ }}}))+ .send()+ .await+ .unwrap();+ assert!(r.status().is_success(), "create {defs}");+ let doc = json!({"symbol": symbol, "symbol_text": symbol, "kind": "define", "lang": "c",+ "file": "limits.h", "line": 5, "text": format!("#define {symbol} {value}"), "value": value});+ let r = client+ .post(format!("{defs}/_bulk?refresh=true"))+ .header("content-type", "application/x-ndjson")+ .body(format!("{{\"index\":{{}}}}\n{doc}\n"))+ .send()+ .await+ .unwrap();+ assert!(r.status().is_success(), "bulk {defs}");+ let docs = json!({"ax_path": format!("{slug}/hash.c"), "title": "hash.c",+ "body": "unsigned mini_hash(const char *s) { frobnicate the buffer; }"});+ let r = client+ .post(format!("{url}/xc-{slug}-b1-docs/_bulk?refresh=true"))+ .header("content-type", "application/x-ndjson")+ .body(format!("{{\"index\":{{}}}}\n{docs}\n"))+ .send()+ .await+ .unwrap();+ assert!(r.status().is_success(), "bulk docs");+}++async fn call(url: &str, name: &str, arguments: Value) -> Value {+ let msg = json!({"jsonrpc": "2.0", "id": 1, "method": "tools/call",+ "params": {"name": name, "arguments": arguments}});+ let resp = xerj_mcp::handle_message_for_test(url, None, msg)+ .await+ .expect("a request gets a response");+ resp["result"].clone()+}++fn text(result: &Value) -> String {+ result["content"][0]["text"]+ .as_str()+ .unwrap_or_default()+ .to_string()+}++#[tokio::test(flavor = "multi_thread", worker_threads = 4)]+async fn hub_tools_answer_from_a_local_pack_and_never_download() {+ let tmp = tempfile::tempdir().unwrap();+ let (hub, code) = (tmp.path().join("hub"), tmp.path().join("code"));+ std::fs::create_dir_all(&hub).unwrap();+ std::env::set_var("XERJ_HUB_HOME", &hub);+ std::env::set_var("XERJ_CODE_HOME", &code);+ // Nothing listens there: a catalogue fetch would fail loudly. The cached+ // catalogue below is fresh, so none is attempted.+ std::env::set_var("XERJ_HUB_URL", "http://127.0.0.1:9");++ // ── a real node, auth off, on an ephemeral loopback port ──+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = tmp.path().join("data").to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ config.auth.enabled = false;+ let metrics = xerj_common::metrics::Metrics::new().unwrap();+ let engine = xerj_engine::Engine::new(config.clone()).unwrap();+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ let app = xerj_api::router::build_es_compat_router(state);+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let url = format!("http://{}", listener.local_addr().unwrap());+ tokio::spawn(async move { axum::serve(listener, app.into_make_service()).await });+ let client = reqwest::Client::new();+ seed(&client, &url, "mini-c", "MINI_LIMIT", "128").await;+ seed(&client, &url, "mini-gpl", "GPL_LIMIT", "7").await;++ restored(&code, &hub, &url, "mini-c", "MIT", "adapt-with-attribution");+ restored(+ &code,+ &hub,+ &url,+ "mini-gpl",+ "GPL-3.0-only",+ "approach-only",+ );+ let cat = Catalogue {+ format_version: CATALOGUE_FORMAT_VERSION,+ generated_at: "2026-10-14T00:00:00Z".into(),+ corpora: vec![+ entry(+ "mini-c",+ "MIT",+ LicenceClass::AdaptWithAttribution,+ &["MINI_LIMIT", "mini_hash"],+ ),+ entry(+ "mini-gpl",+ "GPL-3.0-only",+ LicenceClass::ApproachOnly,+ &["GPL_LIMIT"],+ ),+ entry(+ "mini-closed",+ "LicenseRef-Closed",+ LicenceClass::Excluded,+ &["CLOSED_LIMIT"],+ ),+ // In the catalogue, pullable, not on this machine.+ entry(+ "mini-far",+ "MIT",+ LicenceClass::AdaptWithAttribution,+ &["FAR_LIMIT"],+ ),+ entry(+ "filler-a",+ "MIT",+ LicenceClass::AdaptWithAttribution,+ &["alpha"],+ ),+ entry(+ "filler-b",+ "MIT",+ LicenceClass::AdaptWithAttribution,+ &["bravo"],+ ),+ ],+ };+ std::fs::write(+ hub.join("catalogue.json"),+ serde_json::to_vec(&cat).unwrap(),+ )+ .unwrap();++ // ── tools/list carries the three, with schemas an MCP host accepts ──+ let list = xerj_mcp::handle_message_for_test(+ &url,+ None,+ json!({"jsonrpc": "2.0", "id": 1, "method": "tools/list"}),+ )+ .await+ .unwrap();+ let tools = list["result"]["tools"].as_array().unwrap();+ for name in ["xerj_hub_find", "xerj_hub_def", "xerj_hub_ask"] {+ let t = tools+ .iter()+ .find(|t| t["name"] == name)+ .unwrap_or_else(|| panic!("tools/list lacks {name}"));+ let schema = &t["inputSchema"];+ assert_eq!(schema["type"], "object", "{name}");+ let props = schema["properties"].as_object().unwrap();+ for req in schema["required"].as_array().unwrap() {+ assert!(+ props.contains_key(req.as_str().unwrap()),+ "{name}: {req} is required but undeclared"+ );+ }+ for (prop, spec) in props {+ let ty = spec["type"].as_str().unwrap_or_default();+ assert!(+ ["string", "integer"].contains(&ty),+ "{name}.{prop}: type {ty}"+ );+ assert!(+ spec["description"].as_str().is_some_and(|d| !d.is_empty()),+ "{name}.{prop}"+ );+ }+ assert!(+ t["description"].as_str().is_some_and(|d| d.len() > 40),+ "{name}"+ );+ }++ // ── xerj_hub_def against the fixture pack returns the fixture value ──+ let r = call(+ &url,+ "xerj_hub_def",+ json!({"slug": "mini-c", "symbol": "MINI_LIMIT"}),+ )+ .await;+ assert_eq!(r["isError"], false, "{r}");+ let a = &r["structuredContent"];+ assert_eq!(a["corpus"], "mini-c");+ assert_eq!(a["mode"], "def");+ assert_eq!(a["answered_from"], "local_pack");+ assert_eq!(a["pin_date"], PIN_DATE);+ assert_eq!(a["hits"][0]["value"], "128");+ assert_eq!(a["hits"][0]["file"], "limits.h");+ assert!(text(&r).contains("value: 128"), "{}", text(&r));+ assert!(+ text(&r).contains("answered from: local pack"),+ "{}",+ text(&r)+ );+ // The structured content is exactly an Answer (deny_unknown_fields).+ let _: xerj_common::hub::Answer = serde_json::from_value(a.clone()).unwrap();++ // k is carried through.+ let r = call(+ &url,+ "xerj_hub_def",+ json!({"slug": "mini-c", "symbol": "MINI_LIMIT", "k": 1}),+ )+ .await;+ assert_eq!(r["structuredContent"]["hits"].as_array().unwrap().len(), 1);++ // ── xerj_hub_ask with no corpus: routed by the sketch, named in the answer ──+ let r = call(+ &url,+ "xerj_hub_ask",+ json!({"question": "what is the value of MINI_LIMIT"}),+ )+ .await;+ assert_eq!(r["isError"], false, "{r}");+ assert_eq!(r["structuredContent"]["corpus"], "mini-c");+ assert_eq!(r["structuredContent"]["hits"][0]["value"], "128");++ // ── approach-only: every hit carries the licence warning ──+ let r = call(+ &url,+ "xerj_hub_ask",+ json!({"question": "GPL_LIMIT", "corpus": "mini-gpl", "mode": "def"}),+ )+ .await;+ assert_eq!(r["isError"], false, "{r}");+ for h in r["structuredContent"]["hits"].as_array().unwrap() {+ assert_eq!(+ h["licence_warning"],+ "approach-only licence (GPL-3.0-only): read, do not copy"+ );+ }+ assert!(text(&r).contains("approach-only licence"), "{}", text(&r));++ // ── excluded: never served, never pulled, never hinted at ──+ let r = call(+ &url,+ "xerj_hub_def",+ json!({"slug": "mini-closed", "symbol": "CLOSED_LIMIT"}),+ )+ .await;+ assert_eq!(r["isError"], true, "{r}");+ assert!(text(&r).contains("excluded"), "{}", text(&r));+ assert!(!text(&r).contains("xerj hub pull"), "{}", text(&r));++ // ── not on this machine: the pull line, and nothing downloaded or written ──+ let r = call(+ &url,+ "xerj_hub_def",+ json!({"slug": "mini-far", "symbol": "FAR_LIMIT"}),+ )+ .await;+ assert_eq!(r["isError"], true, "{r}");+ assert!(text(&r).contains("xerj hub pull mini-far"), "{}", text(&r));+ assert!(+ !hub.join("packs").join("mini-far").exists(),+ "a tool call never pulls"+ );+ assert!(!hub.join("pulls").exists(), "nor starts a background pull");++ // ── xerj_hub_find: candidates, best first, excluded never listed ──+ let r = call(+ &url,+ "xerj_hub_find",+ json!({"question": "what is the value of MINI_LIMIT"}),+ )+ .await;+ assert_eq!(r["isError"], false, "{r}");+ let found = r["structuredContent"]["corpora"].as_array().unwrap();+ assert_eq!(found[0]["slug"], "mini-c");+ assert_eq!(found[0]["cached"], true);+ let r = call(&url, "xerj_hub_find", json!({"question": "CLOSED_LIMIT"})).await;+ assert!(+ r["structuredContent"]["corpora"]+ .as_array()+ .unwrap()+ .iter()+ .all(|c| c["slug"] != "mini-closed"),+ "{r}"+ );++ // ── refused before anything runs ──+ for (name, args) in [+ ("xerj_hub_def", json!({"slug": "../etc", "symbol": "X"})),+ (+ "xerj_hub_def",+ json!({"slug": "mini-c", "symbol": "X", "k": 0}),+ ),+ ("xerj_hub_ask", json!({"question": "x".repeat(600)})),+ ("xerj_hub_ask", json!({"question": "q", "mode": "semantic"})),+ ("xerj_hub_find", json!({})),+ ] {+ let r = call(&url, name, args.clone()).await;+ assert_eq!(r["isError"], true, "{name} {args}: {r}");+ }++ // Nothing above wrote to the node: the four fixture indices, no more.+ let cat: Value = client+ .get(format!("{url}/_cat/indices/xc-*?format=json&h=index"))+ .send()+ .await+ .unwrap()+ .json()+ .await+ .unwrap();+ assert_eq!(cat.as_array().unwrap().len(), 4, "{cat}");+}diff --git a/landing/docs/agents/schemas/mcp-tools.json b/landing/docs/agents/schemas/mcp-tools.jsonindex c28a9cdd3..06429f9e1 100644--- a/landing/docs/agents/schemas/mcp-tools.json+++ b/landing/docs/agents/schemas/mcp-tools.json@@ -604,6 +604,90 @@"prompt"]}+ },+ {+ "name": "xerj_hub_find",+ "description": "Discover which hub corpora can answer a question. Returns candidate corpora with their pin, licence and whether the pack is already on this machine — not passages; ask one with xerj_hub_ask. Routed exactly as `xerj ask` routes: your project's dependencies, the signed catalogue's term sketches, and the exact symbol a definition question names (offline; this build has no hosted search). Excluded-licence corpora are never listed.",+ "inputSchema": {+ "type": "object",+ "properties": {+ "question": {+ "type": "string",+ "description": "What the agent is working on; a corpus name is never required here."+ },+ "k": {+ "type": "integer",+ "minimum": 1,+ "maximum": 10,+ "description": "Max candidate corpora to return (default 5, at most 10)."+ }+ },+ "required": [+ "question"+ ]+ }+ },+ {+ "name": "xerj_hub_def",+ "description": "Look up one definition in a named hub corpus: its kind, signature, file:line, and the literal value for a const/macro/define. Returns an Answer in def mode (no passages). Same as `xerj ask --corpus <slug> --def <SYMBOL>`: answered from the local pack; a corpus not on this machine is never downloaded by this tool — the answer says which `xerj hub pull` line fetches it. Approach-only corpora carry a read-do-not-copy warning on every hit.",+ "inputSchema": {+ "type": "object",+ "properties": {+ "slug": {+ "type": "string",+ "description": "Corpus slug to resolve the symbol in."+ },+ "symbol": {+ "type": "string",+ "description": "Exact definition name."+ },+ "k": {+ "type": "integer",+ "minimum": 1,+ "maximum": 50,+ "description": "Max hits to return (default 5)."+ }+ },+ "required": [+ "slug",+ "symbol"+ ]+ }+ },+ {+ "name": "xerj_hub_ask",+ "description": "Ask a hub corpus a question in one turn — no clone, no local indexing. Local packs first, hosted second, exactly like `xerj ask`. The corpus is optional (routed by the offline sketch when omitted) and is always named in the answer; when more than one corpus fits, the candidates come back and you re-ask with `corpus`. Never downloads a pack: a corpus not on this machine is answered with the `xerj hub pull` line to run. Result text is the hub rendering (corpus, pin date, licence, file:line); structured content is the Answer JSON.",+ "inputSchema": {+ "type": "object",+ "properties": {+ "question": {+ "type": "string",+ "description": "Natural-language question."+ },+ "corpus": {+ "type": "string",+ "description": "Optional explicit corpus slug; omit to route by sketch."+ },+ "mode": {+ "type": "string",+ "enum": [+ "def",+ "how",+ "files"+ ],+ "description": "Answer shape; omit for auto (a lone identifier or 'what is the value of X' is def)."+ },+ "max_tokens": {+ "type": "integer",+ "minimum": 1,+ "maximum": 8192,+ "description": "Soft output budget for the rendered answer (about 4 characters a token)."+ }+ },+ "required": [+ "question"+ ]+ }}]}