AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(init): install the xerj-hub skill and print the MCP registration per client (plan §9)
xerj-org0428e3f1d8fc
diff --git a/demo/playbooks/AGENT_FTX_HARNESS.md b/demo/playbooks/AGENT_FTX_HARNESS.mdindex e1b04bfa2..aab5abd61 100644--- a/demo/playbooks/AGENT_FTX_HARNESS.md+++ b/demo/playbooks/AGENT_FTX_HARNESS.md@@ -86,8 +86,11 @@ Assertions:## Phase 3 — connect AI (P1 sets up, P2 verifies)- P1: `xerj init` in a project dir → `.mcp.json` (binary path + `xerj mcp`),- `.claude/skills/xerj/SKILL.md`, `.cursor/rules/xerj.mdc` when the dirs+ `.claude/skills/xerj/SKILL.md` and `.claude/skills/xerj-hub/SKILL.md`,+ `.cursor/rules/xerj.mdc` and `.cursor/rules/xerj-hub.mdc` when the dirsexist, `AGENTS.md` append. `.bak` for anything edited. No secrets written.+ It ends by printing the MCP registration for the clients `.mcp.json` does+ not cover (`claude mcp add`, `mcpServers`, VS Code's `servers`).- P2: drive `xerj mcp` over stdio JSON-RPC — `tools/list` (13 tools),`xerj_search`, `xerj_map`, `xerj_plan` — with a scripted client:```sh@@ -101,6 +104,12 @@ Assertions:and the CLI-only boundary (autoindex, gain, init) is what `llms.txt` says.rc.80: MCP `xerj_search` returns the same top-3 as the CLI for the samequestion, and the response carries the same honest hints.+- Hub (Corpus Hub v2, plan section 9; run it on a build whose `tools/list`+ names `xerj_hub_ask`): P2 asks one public-material question through+ `xerj_hub_ask` and reaches the answer in ONE tool call — no `corpus add`,+ no `corpus index`, no shell. Assert the result is the three lines the+ llms.txt section prints (corpus, pin date and licence on the first line;+ the passage; `answered from:`). P1 does the same with `xerj "<question>"`.## Phase 4 — measure impact on the AI, in tokens and valuesdiff --git a/engine/crates/xerj-autoindex/src/init.rs b/engine/crates/xerj-autoindex/src/init.rsindex a501f4449..f6169db60 100644--- a/engine/crates/xerj-autoindex/src/init.rs+++ b/engine/crates/xerj-autoindex/src/init.rs@@ -27,7 +27,11 @@ config for each — with a .bak backup for any file it edits:.mcp.json register the `xerj mcp` server (Claude Codeproject scope; merged, other servers kept).claude/skills/xerj/SKILL.md teach `xerj search` / `xerj def` (10 lines)+ .claude/skills/xerj-hub/SKILL.md+ teach `xerj \"<question>\"`: when to ask the+ hub, `--def`, the MCP registration per client.cursor/rules/xerj.mdc same teaching file for Cursor (if .cursor/)+ .cursor/rules/xerj-hub.mdc the hub teaching file for Cursor (if .cursor/)AGENTS.md a short XERJ section (appended if the fileexists, created otherwise)@@ -38,7 +42,9 @@ OPTIONS:-h, --help this helpAfter init: restart the agent session so it picks up the new tools, then-`xerj autoindex <folder>` anything you want searchable.+`xerj autoindex <folder>` anything you want searchable. Public libraries,+standards and advisories need no indexing: `xerj \"<question>\"` asks the hub.+init also prints the MCP registration line for each client.Auth: nothing secret is written (`.mcp.json` is meant to be committed). For aloopback node the MCP server reads <data-dir>/admin.key by itself and says so@@ -68,6 +74,135 @@ fn agents_md_section() -> String {.to_string()}+/// The second teaching file (Corpus Hub v2, plan section 9): when to ask the+/// hub instead of searching local files, the one line that does it, the exact+/// form, how to stop a node safely, and the MCP registration per client.+///+/// Kept as a raw string rather than a `.md` asset: the repository ignores+/// `*.md` by default, and a `include_str!` of an ignored file compiles on a+/// developer machine and fails in CI.+///+/// Rules this text follows (docs/research/llms-txt-2026-09/README.md section+/// 6): no obligation language; every step ends with what success looks like;+/// the registration is printed per client with the root key that client reads+/// (`mcpServers` vs VS Code's `servers`); no line over 300 characters.+const HUB_SKILL_MD: &str = r##"---+name: xerj-hub+description: Answer a question about public source code, a standard, a regulation or an advisory feed from a pre-indexed hub corpus with one command, xerj "your question". Use it for exact public material you cannot quote from memory. Not for this project's own files.+---++# XERJ hub: ask, do not clone++Pick the source before you search:++- **HUB**: public material you cannot quote exactly. A library's source at a pinned commit, a+ standard, a regulation, an advisory feed.+- **LOCAL**: files only this machine has. This project, private documents, exports.+ `xerj autoindex <folder>` once, then `xerj search "<words>"` or `xerj def "<symbol>"`.+- **NEITHER**: a fact you can already state exactly. Retrieval on memorised content only adds cost.++## Ask the hub++```sh+xerj "<question>"+```++One line. No corpus name, no clone, no index. XERJ picks the corpus (your project's dependencies+first), fetches its signed, pre-indexed pack on first use, and answers. Retrieval is lexical, not+semantic: use the words and identifiers the source uses.++A good answer is three lines, then any `warning:` lines:++```text+<corpus> pin <date> (<N> days old) <licence> <file>:<line>+<the passage, or the declaration>+answered from: local pack+```++`pin` is the source revision the corpus was built from: check its age when you ask about the current+version. `answered from:` can also say `hosted hub`. Tell the user which corpus, pin date and licence+your answer rests on. `warning: approach-only licence` means read the source, do not copy it.++## Exact questions++```sh+xerj ask --def <SYMBOL>+```++Use `--def` for a value, a signature or a default. It prints the definition (`file:line`, the+declaration, and a `value:` line for a constant or macro) and no passages. Add `--corpus <slug>` when+more than one corpus defines the name.++## When it does not answer++- Exit 4: more than one corpus fits. The candidates print, one per line. Run again with+ `--corpus <slug>`.+- Exit 1, `no hub corpus matches`: the hub does not hold it. Say so. Do not answer from memory.+- Exit 3: a downloaded pack failed its checksum or signature check and was discarded. Do not use it.+ Tell the user.+- `run: xerj hub pull <slug> (<MB> MB)`: the pack is above the automatic download limit. Run that+ line, then ask again. Success: `pulled <slug> <pin8> <bytes> → <n> indices, <defs> definitions`.++## The node, and where the question goes++If a node answers on `localhost:9200`, XERJ uses it as it is. If none does, XERJ starts its own and+records its PID. Stop only the node you started: `xerj hub stop` for XERJ's own,+`kill $(cat ~/xerj-node.pid)` for one you started yourself and saved the PID of.+Never `pkill xerj` or `killall xerj`: other XERJ nodes may be running on this machine.++By default, inside a git repository the question is answered on this machine. Outside one it may be+sent to the hosted hub. `xerj hub config hosted off` forbids that.++## MCP++- `xerj_hub_ask` with `{"question": "<question>"}`: the one line. Add `"corpus": "<slug>"` to name+ a corpus.+- `xerj_hub_def` with `{"slug": "<slug>", "symbol": "<SYMBOL>"}`: the `--def` form.+- `xerj_hub_find` with `{"question": "<question>"}`: which corpora hold this.++Register `xerj mcp` once, then restart the session. Success: `xerj_hub_ask` is in the tool list.+Use the absolute path to the binary (`command -v xerj`): an editor started from a desktop icon does+not inherit your shell PATH.++- Claude Code: `claude mcp add --scope user xerj -- "$(command -v xerj)" mcp`+- `.mcp.json` (Claude Code, project scope; `xerj init` writes it), Cursor, Windsurf, Claude Desktop,+ Cline: root key `mcpServers`.+ `{"mcpServers": {"xerj": {"command": "/abs/path/to/xerj", "args": ["mcp"]}}}`+- VS Code (`.vscode/mcp.json`): root key `servers`, not `mcpServers`. A block under the wrong key is+ silently ignored.+ `{"servers": {"xerj": {"type": "stdio", "command": "/abs/path/to/xerj", "args": ["mcp"]}}}`+"##;++/// What `xerj init` prints for the clients `.mcp.json` does not cover. The same+/// three registrations the hub skill carries; a unit test pins the two together+/// so they cannot drift apart.+const MCP_REGISTRATION: &str = r##"MCP registration for other clients (restart the session after; success is `xerj_hub_ask` in the tool list):+ Claude Code, every project: claude mcp add --scope user xerj -- "$(command -v xerj)" mcp+ Cursor, Windsurf, Claude Desktop, Cline: copy the "xerj" entry from .mcp.json; the root key is "mcpServers"+ VS Code (.vscode/mcp.json): root key "servers", not "mcpServers"; the entry needs "type": "stdio""##;++/// What init says last: where to go next, the hub's one line, and the+/// per-client MCP registration.+fn closing_message() -> String {+ format!(+ "\ndone. Restart the agent session to pick up the tools, then\n\+ `xerj autoindex <folder>` anything you want searchable.\n\n\+ Public libraries, standards and advisories need no indexing: ask the hub with\n\+ `xerj \"<question>\"` (the xerj-hub skill says when).\n\n\+ {MCP_REGISTRATION}"+ )+}++/// Write `content` to `path` unless a file is already there, so a skill the+/// user has edited is never overwritten. Returns the report line for it.+fn write_if_absent(path: &Path, content: &str, dry: bool) -> Result<String, String> {+ if path.exists() {+ return Ok(report("kept", path));+ }+ write_file(path, content, dry).map_err(|e| e.to_string())?;+ Ok(report("wrote", path))+}+/// Back up `path` to `path.bak` before the first edit. Returns true if a/// backup was made (false when the file did not exist).fn backup(path: &Path, dry: bool) -> bool {@@ -167,10 +302,7 @@ pub fn run_init_cli() -> i32 {for l in &lines {println!("{l}");}- println!(- "\ndone. Restart the agent session to pick up the tools, then\n\- `xerj autoindex <folder>` anything you want searchable."- );+ println!("{}", closing_message());0}Err(e) => {@@ -201,31 +333,39 @@ pub(crate) fn run_init_in(cwd: &Path, url: &str, dry: bool) -> Result<Vec<Stringout.push(report(if had { "merged" } else { "wrote" }, &mcp_path));}- // 2) Claude Code skill — written whenever the project (or user) uses- // Claude Code; the directory existing is the signal.+ // 2) Claude Code skills — written whenever the project (or user) uses+ // Claude Code; the directory existing is the signal. Two files: the+ // local-code skill, and the hub skill (when to ask the hub instead).let claude_dir_present = cwd.join(".claude").is_dir()|| dirs_home().map(|h| h.join(".claude").is_dir()).unwrap_or(false);if claude_dir_present {- let skill = cwd.join(".claude/skills/xerj/SKILL.md");- if skill.exists() {- out.push(report("kept", &skill));- } else {- write_file(&skill, &skill_md(), dry).map_err(|e| e.to_string())?;- out.push(report("wrote", &skill));- }+ out.push(write_if_absent(+ &cwd.join(".claude/skills/xerj/SKILL.md"),+ &skill_md(),+ dry,+ )?);+ out.push(write_if_absent(+ &cwd.join(".claude/skills/xerj-hub/SKILL.md"),+ HUB_SKILL_MD,+ dry,+ )?);}- // 3) Cursor rule — only when the project already has a .cursor directory.+ // 3) Cursor rules — only when the project already has a .cursor directory.+ // The same two teaching files, one rule each.if cwd.join(".cursor").is_dir() {- let rule = cwd.join(".cursor/rules/xerj.mdc");- if rule.exists() {- out.push(report("kept", &rule));- } else {- write_file(&rule, &skill_md(), dry).map_err(|e| e.to_string())?;- out.push(report("wrote", &rule));- }+ out.push(write_if_absent(+ &cwd.join(".cursor/rules/xerj.mdc"),+ &skill_md(),+ dry,+ )?);+ out.push(write_if_absent(+ &cwd.join(".cursor/rules/xerj-hub.mdc"),+ HUB_SKILL_MD,+ dry,+ )?);}// 4) AGENTS.md — the cross-agent convention. Append a short section if the@@ -385,6 +525,204 @@ mod tests {assert!(d.path().join(".cursor/rules/xerj.mdc").exists());}+ // ── the hub skill (Corpus Hub v2, plan section 9) ───────────────────++ /// True when `lines` holds a report line for `action` whose path ends in+ /// `tail`. Paths are compared with `/` so the test reads the same on+ /// Windows, where `Path::display` prints backslashes.+ fn has_report(lines: &[String], action: &str, tail: &str) -> bool {+ lines.iter().any(|l| {+ let l = l.replace('\\', "/");+ l.contains(action) && l.ends_with(tail)+ })+ }++ /// `.claude/` is the signal for both skills: the hub skill lands beside the+ /// local-code skill and is a different file with different content.+ #[test]+ fn claude_projects_get_the_hub_skill_beside_the_xerj_skill() {+ let d = tmp();+ fs::create_dir_all(d.path().join(".claude")).unwrap();+ let lines = run_init_in(d.path(), "http://x:1", false).unwrap();+ let local = fs::read_to_string(d.path().join(".claude/skills/xerj/SKILL.md")).unwrap();+ let hub = fs::read_to_string(d.path().join(".claude/skills/xerj-hub/SKILL.md")).unwrap();+ assert_eq!(local, skill_md());+ assert_eq!(hub, HUB_SKILL_MD);+ assert_ne!(local, hub);+ assert!(+ has_report(&lines, "wrote", "skills/xerj-hub/SKILL.md"),+ "{lines:?}"+ );+ }++ /// Cursor gets the same hub text as a rule, and only when `.cursor/` exists.+ #[test]+ fn cursor_projects_get_the_hub_rule_with_the_same_text() {+ let d = tmp();+ run_init_in(d.path(), "http://x:1", false).unwrap();+ assert!(!d.path().join(".cursor/rules/xerj-hub.mdc").exists());+ fs::create_dir_all(d.path().join(".cursor")).unwrap();+ let lines = run_init_in(d.path(), "http://x:1", false).unwrap();+ assert_eq!(+ fs::read_to_string(d.path().join(".cursor/rules/xerj-hub.mdc")).unwrap(),+ HUB_SKILL_MD+ );+ assert!(+ has_report(&lines, "wrote", "rules/xerj-hub.mdc"),+ "{lines:?}"+ );+ }++ /// A hub skill the user edited is never overwritten.+ #[test]+ fn an_edited_hub_skill_is_kept_not_overwritten() {+ let d = tmp();+ let hub = d.path().join(".claude/skills/xerj-hub/SKILL.md");+ fs::create_dir_all(hub.parent().unwrap()).unwrap();+ fs::write(&hub, "my own rules\n").unwrap();+ let lines = run_init_in(d.path(), "http://x:1", false).unwrap();+ assert_eq!(fs::read_to_string(&hub).unwrap(), "my own rules\n");+ assert!(+ has_report(&lines, "kept", "skills/xerj-hub/SKILL.md"),+ "{lines:?}"+ );+ }++ /// `--dry-run` reports both hub files and writes neither.+ #[test]+ fn dry_run_reports_the_hub_files_and_writes_none() {+ let d = tmp();+ fs::create_dir_all(d.path().join(".claude")).unwrap();+ fs::create_dir_all(d.path().join(".cursor")).unwrap();+ let lines = run_init_in(d.path(), "http://x:1", true).unwrap();+ assert!(!d.path().join(".claude/skills").exists());+ assert!(!d.path().join(".cursor/rules").exists());+ assert!(has_report(&lines, "wrote", "skills/xerj-hub/SKILL.md"));+ assert!(has_report(&lines, "wrote", "rules/xerj-hub.mdc"));+ }++ /// The second run rewrites nothing: all four teaching files are `kept`.+ #[test]+ fn second_run_keeps_all_four_teaching_files() {+ // `ENV_LOCK`: see `second_run_is_idempotent` — the auth note reads the+ // process cwd and HOME, which the sandboxed tests swap.+ let _env = ENV_LOCK.lock().unwrap_or_else(|poison| poison.into_inner());+ let d = tmp();+ fs::create_dir_all(d.path().join(".claude")).unwrap();+ fs::create_dir_all(d.path().join(".cursor")).unwrap();+ run_init_in(d.path(), "http://x:1", false).unwrap();+ let lines = run_init_in(d.path(), "http://x:1", false).unwrap();+ for tail in [+ "skills/xerj/SKILL.md",+ "skills/xerj-hub/SKILL.md",+ "rules/xerj.mdc",+ "rules/xerj-hub.mdc",+ ] {+ assert!(has_report(&lines, "kept", tail), "{tail}: {lines:?}");+ }+ }++ /// The reason the file exists: the decision rule, the one line, `--def`,+ /// the safe-stop rule, and the registration per client. Deleting any of+ /// them has to fail here.+ #[test]+ fn hub_skill_carries_the_rule_the_line_and_the_registration() {+ for needed in [+ "**HUB**",+ "**LOCAL**",+ "**NEITHER**",+ r#"xerj "<question>""#,+ "xerj ask --def <SYMBOL>",+ "Stop only the node you started",+ "Never `pkill xerj` or `killall xerj`",+ "xerj hub pull",+ "xerj_hub_ask",+ "claude mcp add",+ "root key `mcpServers`",+ "root key `servers`, not `mcpServers`",+ ] {+ assert!(+ HUB_SKILL_MD.contains(needed),+ "the hub skill lost {needed:?}"+ );+ }+ }++ /// The frontmatter a skill loader reads: a `name` that matches the+ /// directory, and a `description` that is a valid plain YAML scalar with no+ /// angle brackets (the Agent Skills spec forbids XML-like tags there).+ #[test]+ fn hub_skill_frontmatter_is_loadable() {+ let rest = HUB_SKILL_MD+ .strip_prefix("---\n")+ .expect("the skill opens with frontmatter");+ let (front, _body) = rest.split_once("\n---\n").expect("the frontmatter closes");+ fn field<'a>(front: &'a str, key: &str) -> &'a str {+ front+ .lines()+ .find_map(|l| l.strip_prefix(key))+ .unwrap_or_else(|| panic!("frontmatter has no {key}"))+ }+ assert_eq!(field(front, "name: "), "xerj-hub");+ let description = field(front, "description: ");+ assert!(!description.is_empty() && description.len() <= 1024);+ assert!(+ !description.contains('<') && !description.contains('>'),+ "no angle brackets in the description: {description}"+ );+ assert!(+ !description.contains(": ") && !description.contains(" #"),+ "a plain YAML scalar may not contain `: ` or ` #`: {description}"+ );+ }++ /// llms.txt study rule 7 (`docs/research/llms-txt-2026-09/README.md`): no+ /// line over about 300 characters in text an agent reads.+ #[test]+ fn hub_texts_have_no_line_over_300_characters() {+ for (name, text) in [+ ("hub skill", HUB_SKILL_MD),+ ("mcp registration", MCP_REGISTRATION),+ ] {+ for (i, line) in text.lines().enumerate() {+ let n = line.chars().count();+ assert!(n <= 300, "{name} line {} is {n} characters", i + 1);+ }+ }+ }++ /// What init prints and what the skill teaches are one registration, not+ /// two that drift: the same Claude Code command and the same root keys.+ #[test]+ fn printed_registration_and_hub_skill_agree() {+ let claude = r#"claude mcp add --scope user xerj -- "$(command -v xerj)" mcp"#;+ assert!(HUB_SKILL_MD.contains(claude));+ assert!(MCP_REGISTRATION.contains(claude));+ for key in ["mcpServers", "servers", "stdio"] {+ assert!(HUB_SKILL_MD.contains(key), "skill: {key}");+ assert!(MCP_REGISTRATION.contains(key), "printed: {key}");+ }+ let closing = closing_message();+ assert!(closing.contains(MCP_REGISTRATION));+ assert!(closing.contains(r#"xerj "<question>""#));+ assert!(closing.contains("xerj autoindex <folder>"));+ }++ /// `--help` lists every file init can write, the hub pair included.+ #[test]+ fn help_lists_every_file_init_writes() {+ for file in [+ ".mcp.json",+ ".claude/skills/xerj/SKILL.md",+ ".claude/skills/xerj-hub/SKILL.md",+ ".cursor/rules/xerj.mdc",+ ".cursor/rules/xerj-hub.mdc",+ "AGENTS.md",+ ] {+ assert!(USAGE.contains(file), "--help does not list {file}");+ }+ }+// ── auth note (#961) ────────────────────────────────────────────────/// The auth note reads the working directory and `HOME` (that is what