Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(init): install the xerj-hub skill and print the MCP registration per client (plan §9)

xerj-org0428e3f1d8fc
diff --git a/demo/playbooks/AGENT_FTX_HARNESS.md b/demo/playbooks/AGENT_FTX_HARNESS.md
index e1b04bfa2..aab5abd61 100644
--- a/demo/playbooks/AGENT_FTX_HARNESS.md
+++ b/demo/playbooks/AGENT_FTX_HARNESS.md
@@ -86,8 +86,11 @@ Assertions:
## Phase 3 — connect AI (P1 sets up, P2 verifies)
- P1: `xerj init` in a project dir → `.mcp.json` (binary path + `xerj mcp`),
- `.claude/skills/xerj/SKILL.md`, `.cursor/rules/xerj.mdc` when the dirs
+ `.claude/skills/xerj/SKILL.md` and `.claude/skills/xerj-hub/SKILL.md`,
+ `.cursor/rules/xerj.mdc` and `.cursor/rules/xerj-hub.mdc` when the dirs
exist, `AGENTS.md` append. `.bak` for anything edited. No secrets written.
+ It ends by printing the MCP registration for the clients `.mcp.json` does
+ not cover (`claude mcp add`, `mcpServers`, VS Code's `servers`).
- P2: drive `xerj mcp` over stdio JSON-RPC — `tools/list` (13 tools),
`xerj_search`, `xerj_map`, `xerj_plan` — with a scripted client:
```sh
@@ -101,6 +104,12 @@ Assertions:
and the CLI-only boundary (autoindex, gain, init) is what `llms.txt` says.
rc.80: MCP `xerj_search` returns the same top-3 as the CLI for the same
question, and the response carries the same honest hints.
+- Hub (Corpus Hub v2, plan section 9; run it on a build whose `tools/list`
+ names `xerj_hub_ask`): P2 asks one public-material question through
+ `xerj_hub_ask` and reaches the answer in ONE tool call — no `corpus add`,
+ no `corpus index`, no shell. Assert the result is the three lines the
+ llms.txt section prints (corpus, pin date and licence on the first line;
+ the passage; `answered from:`). P1 does the same with `xerj "<question>"`.
## Phase 4 — measure impact on the AI, in tokens and values
diff --git a/engine/crates/xerj-autoindex/src/init.rs b/engine/crates/xerj-autoindex/src/init.rs
index a501f4449..f6169db60 100644
--- a/engine/crates/xerj-autoindex/src/init.rs
+++ b/engine/crates/xerj-autoindex/src/init.rs
@@ -27,7 +27,11 @@ config for each — with a .bak backup for any file it edits:
.mcp.json register the `xerj mcp` server (Claude Code
project scope; merged, other servers kept)
.claude/skills/xerj/SKILL.md teach `xerj search` / `xerj def` (10 lines)
+ .claude/skills/xerj-hub/SKILL.md
+ teach `xerj \"<question>\"`: when to ask the
+ hub, `--def`, the MCP registration per client
.cursor/rules/xerj.mdc same teaching file for Cursor (if .cursor/)
+ .cursor/rules/xerj-hub.mdc the hub teaching file for Cursor (if .cursor/)
AGENTS.md a short XERJ section (appended if the file
exists, created otherwise)
@@ -38,7 +42,9 @@ OPTIONS:
-h, --help this help
After init: restart the agent session so it picks up the new tools, then
-`xerj autoindex <folder>` anything you want searchable.
+`xerj autoindex <folder>` anything you want searchable. Public libraries,
+standards and advisories need no indexing: `xerj \"<question>\"` asks the hub.
+init also prints the MCP registration line for each client.
Auth: nothing secret is written (`.mcp.json` is meant to be committed). For a
loopback node the MCP server reads <data-dir>/admin.key by itself and says so
@@ -68,6 +74,135 @@ fn agents_md_section() -> String {
.to_string()
}
+/// The second teaching file (Corpus Hub v2, plan section 9): when to ask the
+/// hub instead of searching local files, the one line that does it, the exact
+/// form, how to stop a node safely, and the MCP registration per client.
+///
+/// Kept as a raw string rather than a `.md` asset: the repository ignores
+/// `*.md` by default, and a `include_str!` of an ignored file compiles on a
+/// developer machine and fails in CI.
+///
+/// Rules this text follows (docs/research/llms-txt-2026-09/README.md section
+/// 6): no obligation language; every step ends with what success looks like;
+/// the registration is printed per client with the root key that client reads
+/// (`mcpServers` vs VS Code's `servers`); no line over 300 characters.
+const HUB_SKILL_MD: &str = r##"---
+name: xerj-hub
+description: Answer a question about public source code, a standard, a regulation or an advisory feed from a pre-indexed hub corpus with one command, xerj "your question". Use it for exact public material you cannot quote from memory. Not for this project's own files.
+---
+
+# XERJ hub: ask, do not clone
+
+Pick the source before you search:
+
+- **HUB**: public material you cannot quote exactly. A library's source at a pinned commit, a
+ standard, a regulation, an advisory feed.
+- **LOCAL**: files only this machine has. This project, private documents, exports.
+ `xerj autoindex <folder>` once, then `xerj search "<words>"` or `xerj def "<symbol>"`.
+- **NEITHER**: a fact you can already state exactly. Retrieval on memorised content only adds cost.
+
+## Ask the hub
+
+```sh
+xerj "<question>"
+```
+
+One line. No corpus name, no clone, no index. XERJ picks the corpus (your project's dependencies
+first), fetches its signed, pre-indexed pack on first use, and answers. Retrieval is lexical, not
+semantic: use the words and identifiers the source uses.
+
+A good answer is three lines, then any `warning:` lines:
+
+```text
+<corpus> pin <date> (<N> days old) <licence> <file>:<line>
+<the passage, or the declaration>
+answered from: local pack
+```
+
+`pin` is the source revision the corpus was built from: check its age when you ask about the current
+version. `answered from:` can also say `hosted hub`. Tell the user which corpus, pin date and licence
+your answer rests on. `warning: approach-only licence` means read the source, do not copy it.
+
+## Exact questions
+
+```sh
+xerj ask --def <SYMBOL>
+```
+
+Use `--def` for a value, a signature or a default. It prints the definition (`file:line`, the
+declaration, and a `value:` line for a constant or macro) and no passages. Add `--corpus <slug>` when
+more than one corpus defines the name.
+
+## When it does not answer
+
+- Exit 4: more than one corpus fits. The candidates print, one per line. Run again with
+ `--corpus <slug>`.
+- Exit 1, `no hub corpus matches`: the hub does not hold it. Say so. Do not answer from memory.
+- Exit 3: a downloaded pack failed its checksum or signature check and was discarded. Do not use it.
+ Tell the user.
+- `run: xerj hub pull <slug> (<MB> MB)`: the pack is above the automatic download limit. Run that
+ line, then ask again. Success: `pulled <slug> <pin8> <bytes> → <n> indices, <defs> definitions`.
+
+## The node, and where the question goes
+
+If a node answers on `localhost:9200`, XERJ uses it as it is. If none does, XERJ starts its own and
+records its PID. Stop only the node you started: `xerj hub stop` for XERJ's own,
+`kill $(cat ~/xerj-node.pid)` for one you started yourself and saved the PID of.
+Never `pkill xerj` or `killall xerj`: other XERJ nodes may be running on this machine.
+
+By default, inside a git repository the question is answered on this machine. Outside one it may be
+sent to the hosted hub. `xerj hub config hosted off` forbids that.
+
+## MCP
+
+- `xerj_hub_ask` with `{"question": "<question>"}`: the one line. Add `"corpus": "<slug>"` to name
+ a corpus.
+- `xerj_hub_def` with `{"slug": "<slug>", "symbol": "<SYMBOL>"}`: the `--def` form.
+- `xerj_hub_find` with `{"question": "<question>"}`: which corpora hold this.
+
+Register `xerj mcp` once, then restart the session. Success: `xerj_hub_ask` is in the tool list.
+Use the absolute path to the binary (`command -v xerj`): an editor started from a desktop icon does
+not inherit your shell PATH.
+
+- Claude Code: `claude mcp add --scope user xerj -- "$(command -v xerj)" mcp`
+- `.mcp.json` (Claude Code, project scope; `xerj init` writes it), Cursor, Windsurf, Claude Desktop,
+ Cline: root key `mcpServers`.
+ `{"mcpServers": {"xerj": {"command": "/abs/path/to/xerj", "args": ["mcp"]}}}`
+- VS Code (`.vscode/mcp.json`): root key `servers`, not `mcpServers`. A block under the wrong key is
+ silently ignored.
+ `{"servers": {"xerj": {"type": "stdio", "command": "/abs/path/to/xerj", "args": ["mcp"]}}}`
+"##;
+
+/// What `xerj init` prints for the clients `.mcp.json` does not cover. The same
+/// three registrations the hub skill carries; a unit test pins the two together
+/// so they cannot drift apart.
+const MCP_REGISTRATION: &str = r##"MCP registration for other clients (restart the session after; success is `xerj_hub_ask` in the tool list):
+ Claude Code, every project: claude mcp add --scope user xerj -- "$(command -v xerj)" mcp
+ Cursor, Windsurf, Claude Desktop, Cline: copy the "xerj" entry from .mcp.json; the root key is "mcpServers"
+ VS Code (.vscode/mcp.json): root key "servers", not "mcpServers"; the entry needs "type": "stdio""##;
+
+/// What init says last: where to go next, the hub's one line, and the
+/// per-client MCP registration.
+fn closing_message() -> String {
+ format!(
+ "\ndone. Restart the agent session to pick up the tools, then\n\
+ `xerj autoindex <folder>` anything you want searchable.\n\n\
+ Public libraries, standards and advisories need no indexing: ask the hub with\n\
+ `xerj \"<question>\"` (the xerj-hub skill says when).\n\n\
+ {MCP_REGISTRATION}"
+ )
+}
+
+/// Write `content` to `path` unless a file is already there, so a skill the
+/// user has edited is never overwritten. Returns the report line for it.
+fn write_if_absent(path: &Path, content: &str, dry: bool) -> Result<String, String> {
+ if path.exists() {
+ return Ok(report("kept", path));
+ }
+ write_file(path, content, dry).map_err(|e| e.to_string())?;
+ Ok(report("wrote", path))
+}
+
/// Back up `path` to `path.bak` before the first edit. Returns true if a
/// backup was made (false when the file did not exist).
fn backup(path: &Path, dry: bool) -> bool {
@@ -167,10 +302,7 @@ pub fn run_init_cli() -> i32 {
for l in &lines {
println!("{l}");
}
- println!(
- "\ndone. Restart the agent session to pick up the tools, then\n\
- `xerj autoindex <folder>` anything you want searchable."
- );
+ println!("{}", closing_message());
0
}
Err(e) => {
@@ -201,31 +333,39 @@ pub(crate) fn run_init_in(cwd: &Path, url: &str, dry: bool) -> Result<Vec<String
out.push(report(if had { "merged" } else { "wrote" }, &mcp_path));
}
- // 2) Claude Code skill — written whenever the project (or user) uses
- // Claude Code; the directory existing is the signal.
+ // 2) Claude Code skills — written whenever the project (or user) uses
+ // Claude Code; the directory existing is the signal. Two files: the
+ // local-code skill, and the hub skill (when to ask the hub instead).
let claude_dir_present = cwd.join(".claude").is_dir()
|| dirs_home()
.map(|h| h.join(".claude").is_dir())
.unwrap_or(false);
if claude_dir_present {
- let skill = cwd.join(".claude/skills/xerj/SKILL.md");
- if skill.exists() {
- out.push(report("kept", &skill));
- } else {
- write_file(&skill, &skill_md(), dry).map_err(|e| e.to_string())?;
- out.push(report("wrote", &skill));
- }
+ out.push(write_if_absent(
+ &cwd.join(".claude/skills/xerj/SKILL.md"),
+ &skill_md(),
+ dry,
+ )?);
+ out.push(write_if_absent(
+ &cwd.join(".claude/skills/xerj-hub/SKILL.md"),
+ HUB_SKILL_MD,
+ dry,
+ )?);
}
- // 3) Cursor rule — only when the project already has a .cursor directory.
+ // 3) Cursor rules — only when the project already has a .cursor directory.
+ // The same two teaching files, one rule each.
if cwd.join(".cursor").is_dir() {
- let rule = cwd.join(".cursor/rules/xerj.mdc");
- if rule.exists() {
- out.push(report("kept", &rule));
- } else {
- write_file(&rule, &skill_md(), dry).map_err(|e| e.to_string())?;
- out.push(report("wrote", &rule));
- }
+ out.push(write_if_absent(
+ &cwd.join(".cursor/rules/xerj.mdc"),
+ &skill_md(),
+ dry,
+ )?);
+ out.push(write_if_absent(
+ &cwd.join(".cursor/rules/xerj-hub.mdc"),
+ HUB_SKILL_MD,
+ dry,
+ )?);
}
// 4) AGENTS.md — the cross-agent convention. Append a short section if the
@@ -385,6 +525,204 @@ mod tests {
assert!(d.path().join(".cursor/rules/xerj.mdc").exists());
}
+ // ── the hub skill (Corpus Hub v2, plan section 9) ───────────────────
+
+ /// True when `lines` holds a report line for `action` whose path ends in
+ /// `tail`. Paths are compared with `/` so the test reads the same on
+ /// Windows, where `Path::display` prints backslashes.
+ fn has_report(lines: &[String], action: &str, tail: &str) -> bool {
+ lines.iter().any(|l| {
+ let l = l.replace('\\', "/");
+ l.contains(action) && l.ends_with(tail)
+ })
+ }
+
+ /// `.claude/` is the signal for both skills: the hub skill lands beside the
+ /// local-code skill and is a different file with different content.
+ #[test]
+ fn claude_projects_get_the_hub_skill_beside_the_xerj_skill() {
+ let d = tmp();
+ fs::create_dir_all(d.path().join(".claude")).unwrap();
+ let lines = run_init_in(d.path(), "http://x:1", false).unwrap();
+ let local = fs::read_to_string(d.path().join(".claude/skills/xerj/SKILL.md")).unwrap();
+ let hub = fs::read_to_string(d.path().join(".claude/skills/xerj-hub/SKILL.md")).unwrap();
+ assert_eq!(local, skill_md());
+ assert_eq!(hub, HUB_SKILL_MD);
+ assert_ne!(local, hub);
+ assert!(
+ has_report(&lines, "wrote", "skills/xerj-hub/SKILL.md"),
+ "{lines:?}"
+ );
+ }
+
+ /// Cursor gets the same hub text as a rule, and only when `.cursor/` exists.
+ #[test]
+ fn cursor_projects_get_the_hub_rule_with_the_same_text() {
+ let d = tmp();
+ run_init_in(d.path(), "http://x:1", false).unwrap();
+ assert!(!d.path().join(".cursor/rules/xerj-hub.mdc").exists());
+ fs::create_dir_all(d.path().join(".cursor")).unwrap();
+ let lines = run_init_in(d.path(), "http://x:1", false).unwrap();
+ assert_eq!(
+ fs::read_to_string(d.path().join(".cursor/rules/xerj-hub.mdc")).unwrap(),
+ HUB_SKILL_MD
+ );
+ assert!(
+ has_report(&lines, "wrote", "rules/xerj-hub.mdc"),
+ "{lines:?}"
+ );
+ }
+
+ /// A hub skill the user edited is never overwritten.
+ #[test]
+ fn an_edited_hub_skill_is_kept_not_overwritten() {
+ let d = tmp();
+ let hub = d.path().join(".claude/skills/xerj-hub/SKILL.md");
+ fs::create_dir_all(hub.parent().unwrap()).unwrap();
+ fs::write(&hub, "my own rules\n").unwrap();
+ let lines = run_init_in(d.path(), "http://x:1", false).unwrap();
+ assert_eq!(fs::read_to_string(&hub).unwrap(), "my own rules\n");
+ assert!(
+ has_report(&lines, "kept", "skills/xerj-hub/SKILL.md"),
+ "{lines:?}"
+ );
+ }
+
+ /// `--dry-run` reports both hub files and writes neither.
+ #[test]
+ fn dry_run_reports_the_hub_files_and_writes_none() {
+ let d = tmp();
+ fs::create_dir_all(d.path().join(".claude")).unwrap();
+ fs::create_dir_all(d.path().join(".cursor")).unwrap();
+ let lines = run_init_in(d.path(), "http://x:1", true).unwrap();
+ assert!(!d.path().join(".claude/skills").exists());
+ assert!(!d.path().join(".cursor/rules").exists());
+ assert!(has_report(&lines, "wrote", "skills/xerj-hub/SKILL.md"));
+ assert!(has_report(&lines, "wrote", "rules/xerj-hub.mdc"));
+ }
+
+ /// The second run rewrites nothing: all four teaching files are `kept`.
+ #[test]
+ fn second_run_keeps_all_four_teaching_files() {
+ // `ENV_LOCK`: see `second_run_is_idempotent` — the auth note reads the
+ // process cwd and HOME, which the sandboxed tests swap.
+ let _env = ENV_LOCK.lock().unwrap_or_else(|poison| poison.into_inner());
+ let d = tmp();
+ fs::create_dir_all(d.path().join(".claude")).unwrap();
+ fs::create_dir_all(d.path().join(".cursor")).unwrap();
+ run_init_in(d.path(), "http://x:1", false).unwrap();
+ let lines = run_init_in(d.path(), "http://x:1", false).unwrap();
+ for tail in [
+ "skills/xerj/SKILL.md",
+ "skills/xerj-hub/SKILL.md",
+ "rules/xerj.mdc",
+ "rules/xerj-hub.mdc",
+ ] {
+ assert!(has_report(&lines, "kept", tail), "{tail}: {lines:?}");
+ }
+ }
+
+ /// The reason the file exists: the decision rule, the one line, `--def`,
+ /// the safe-stop rule, and the registration per client. Deleting any of
+ /// them has to fail here.
+ #[test]
+ fn hub_skill_carries_the_rule_the_line_and_the_registration() {
+ for needed in [
+ "**HUB**",
+ "**LOCAL**",
+ "**NEITHER**",
+ r#"xerj "<question>""#,
+ "xerj ask --def <SYMBOL>",
+ "Stop only the node you started",
+ "Never `pkill xerj` or `killall xerj`",
+ "xerj hub pull",
+ "xerj_hub_ask",
+ "claude mcp add",
+ "root key `mcpServers`",
+ "root key `servers`, not `mcpServers`",
+ ] {
+ assert!(
+ HUB_SKILL_MD.contains(needed),
+ "the hub skill lost {needed:?}"
+ );
+ }
+ }
+
+ /// The frontmatter a skill loader reads: a `name` that matches the
+ /// directory, and a `description` that is a valid plain YAML scalar with no
+ /// angle brackets (the Agent Skills spec forbids XML-like tags there).
+ #[test]
+ fn hub_skill_frontmatter_is_loadable() {
+ let rest = HUB_SKILL_MD
+ .strip_prefix("---\n")
+ .expect("the skill opens with frontmatter");
+ let (front, _body) = rest.split_once("\n---\n").expect("the frontmatter closes");
+ fn field<'a>(front: &'a str, key: &str) -> &'a str {
+ front
+ .lines()
+ .find_map(|l| l.strip_prefix(key))
+ .unwrap_or_else(|| panic!("frontmatter has no {key}"))
+ }
+ assert_eq!(field(front, "name: "), "xerj-hub");
+ let description = field(front, "description: ");
+ assert!(!description.is_empty() && description.len() <= 1024);
+ assert!(
+ !description.contains('<') && !description.contains('>'),
+ "no angle brackets in the description: {description}"
+ );
+ assert!(
+ !description.contains(": ") && !description.contains(" #"),
+ "a plain YAML scalar may not contain `: ` or ` #`: {description}"
+ );
+ }
+
+ /// llms.txt study rule 7 (`docs/research/llms-txt-2026-09/README.md`): no
+ /// line over about 300 characters in text an agent reads.
+ #[test]
+ fn hub_texts_have_no_line_over_300_characters() {
+ for (name, text) in [
+ ("hub skill", HUB_SKILL_MD),
+ ("mcp registration", MCP_REGISTRATION),
+ ] {
+ for (i, line) in text.lines().enumerate() {
+ let n = line.chars().count();
+ assert!(n <= 300, "{name} line {} is {n} characters", i + 1);
+ }
+ }
+ }
+
+ /// What init prints and what the skill teaches are one registration, not
+ /// two that drift: the same Claude Code command and the same root keys.
+ #[test]
+ fn printed_registration_and_hub_skill_agree() {
+ let claude = r#"claude mcp add --scope user xerj -- "$(command -v xerj)" mcp"#;
+ assert!(HUB_SKILL_MD.contains(claude));
+ assert!(MCP_REGISTRATION.contains(claude));
+ for key in ["mcpServers", "servers", "stdio"] {
+ assert!(HUB_SKILL_MD.contains(key), "skill: {key}");
+ assert!(MCP_REGISTRATION.contains(key), "printed: {key}");
+ }
+ let closing = closing_message();
+ assert!(closing.contains(MCP_REGISTRATION));
+ assert!(closing.contains(r#"xerj "<question>""#));
+ assert!(closing.contains("xerj autoindex <folder>"));
+ }
+
+ /// `--help` lists every file init can write, the hub pair included.
+ #[test]
+ fn help_lists_every_file_init_writes() {
+ for file in [
+ ".mcp.json",
+ ".claude/skills/xerj/SKILL.md",
+ ".claude/skills/xerj-hub/SKILL.md",
+ ".cursor/rules/xerj.mdc",
+ ".cursor/rules/xerj-hub.mdc",
+ "AGENTS.md",
+ ] {
+ assert!(USAGE.contains(file), "--help does not list {file}");
+ }
+ }
+
// ── auth note (#961) ────────────────────────────────────────────────
/// The auth note reads the working directory and `HOME` (that is what