AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge pull request #1411 from xerj-org/fix/init-backup-fail-closed
xerj-org2e130ae7eaad
diff --cc engine/crates/xerj-autoindex/src/init.rsindex 67da4aa47,d2784fae3..c9fb411fd--- a/engine/crates/xerj-autoindex/src/init.rs+++ b/engine/crates/xerj-autoindex/src/init.rs@@@ -74,193 -68,13 +74,195 @@@ fn agents_md_section() -> String.to_string()}+/// The second teaching file (Corpus Hub v2, plan section 9): when to ask the+/// hub instead of searching local files, the one line that does it, the exact+/// form, how to stop a node safely, and the MCP registration per client.+///+/// Kept as a raw string rather than a `.md` asset: the repository ignores+/// `*.md` by default, and a `include_str!` of an ignored file compiles on a+/// developer machine and fails in CI.+///+/// Rules this text follows (docs/research/llms-txt-2026-09/README.md section+/// 6): no obligation language; every step ends with what success looks like;+/// the registration is printed per client with the root key that client reads+/// (`mcpServers` vs VS Code's `servers`); no line over 300 characters.+const HUB_SKILL_MD: &str = r##"---+name: xerj-hub+description: Answer a question about public source code, a standard, a regulation or an advisory feed from a pre-indexed hub corpus with one command, xerj "your question". Use it for exact public material you cannot quote from memory. Not for this project's own files.+---++# XERJ hub: ask, do not clone++Pick the source before you search:++- **HUB**: public material you cannot quote exactly. A library's source at a pinned commit, a+ standard, a regulation, an advisory feed.+- **LOCAL**: files only this machine has. This project, private documents, exports.+ `xerj autoindex <folder>` once, then `xerj search "<words>"` or `xerj def "<symbol>"`.+- **NEITHER**: a fact you can already state exactly. Retrieval on memorised content only adds cost.++## Ask the hub++```sh+xerj "<question>"+```++One line. No corpus name, no clone, no index. XERJ picks the corpus (your project's dependencies+first), fetches its signed, pre-indexed pack on first use, and answers. Retrieval is lexical, not+semantic: use the words and identifiers the source uses. The bare form needs a space in the+question; for a single word write `xerj ask "<word>"`.++A good answer is three lines, then any `warning:` lines:++```text+<corpus> pin <date> (<N> days old) <licence> <file>:<line>+<the passage, or the declaration>+answered from: local pack+```++`pin` is the source revision the corpus was built from: check its age when you ask about the current+version. `answered from:` can also say `hosted hub`. Tell the user which corpus, pin date and licence+your answer rests on. `warning: approach-only licence` means read the source, do not copy it.++## Exact questions++```sh+xerj ask --def <SYMBOL>+```++Use `--def` for a value, a signature or a default. It prints the definition (`file:line`, the+declaration, and a `value:` line for a constant or macro) and no passages. Add `--corpus <slug>` when+more than one corpus defines the name.++## When it does not answer++- Exit 4: more than one corpus fits. The candidates print, one per line. Run again with+ `--corpus <slug>`.+- Exit 1, `no hub corpus matches`: the hub does not hold it. Say so. Do not answer from memory.+- Exit 3: a downloaded pack failed its checksum or signature check and was discarded. Do not use it.+ Tell the user.+- Exit 5: this build has no hub public key, so no pack can be verified and nothing was downloaded.+ Tell the user; a retry will not help.+- `run: xerj hub pull <slug> (<MB> MB)`: the pack is above the automatic download limit. Run that+ line, then ask again. Success: `pulled <slug> <pin8> <bytes> → <n> indices, <defs> definitions`.++## The node, and where the question goes++If a node answers on `--url`, `$XERJ_URL` or `localhost:9200` (first one set), XERJ uses it as it+is. If none does, XERJ starts its own on a free loopback port and records its PID.+Stop only the node you started: `xerj hub stop` for XERJ's own (success: `stopped the hub node+(pid <N>)` or `no hub node was started by xerj; nothing to stop`); for a node you started by hand,+kill only the PID you recorded when you started it. Never `pkill xerj` or `killall xerj`: other XERJ nodes may be running+on this machine.++By default, inside a git repository the question is answered on this machine. Outside one it may be+sent to the hosted hub. `xerj hub config hosted off` forbids that.++## MCP++`xerj mcp` serves three hub tools. They run the same flow as `xerj ask`:+- `xerj_hub_ask` `{question, corpus?, mode?, max_tokens?}`: the shell line above, as one call.+- `xerj_hub_def` `{slug, symbol, k?}`: `xerj ask --corpus <slug> --def <SYMBOL>`.+- `xerj_hub_find` `{question, k?}`: which corpora could answer, best first. No passages.++A hub tool answers only from packs already on this machine; it never downloads one. When a corpus+is not here, the answer is the `xerj hub pull <slug>` line: run it in a shell, then ask again.+Nor does a hub tool fetch the catalogue: it reads the cached copy. If it says no catalogue is+cached, run `xerj hub sync` in a shell once, then call the tool again.++Register `xerj mcp` once, then restart the session. Success: `xerj_search` is in the tool list.+Use the absolute path to the binary: an editor started from a desktop icon does not inherit your+shell PATH. `command -v xerj` prints it (PowerShell: `(Get-Command xerj).Source`).++- Claude Code: `claude mcp add --scope user xerj -- "$(command -v xerj)" mcp`+ (PowerShell or cmd: put the full path of `xerj.exe` in place of `"$(command -v xerj)"`)+- `.mcp.json` (Claude Code, project scope; `xerj init` writes it), Cursor, Windsurf, Claude Desktop,+ Cline: root key `mcpServers`.+ `{"mcpServers": {"xerj": {"command": "/abs/path/to/xerj", "args": ["mcp"]}}}`+- VS Code (`.vscode/mcp.json`): root key `servers`, not `mcpServers`. A block under the wrong key is+ silently ignored.+ `{"servers": {"xerj": {"type": "stdio", "command": "/abs/path/to/xerj", "args": ["mcp"]}}}`+"##;++/// The MCP tool whose presence proves `xerj mcp` is registered. It must be a+/// tool every `xerj mcp` serves. The hub tools (`xerj_hub_*`) are served too,+/// but `xerj_search` is the proof the skill and the printed text have always+/// used, and it stays the one check.+const MCP_SUCCESS_TOOL: &str = "xerj_search";++/// The default node URL; the printed Claude Code line carries `-e XERJ_URL=`+/// only when init was pointed somewhere else.+const DEFAULT_URL: &str = "http://localhost:9200";++/// Quote `s` as one argument for the shell the user is most likely to paste+/// into: POSIX single quotes on Unix, double quotes on Windows (PowerShell and+/// cmd both accept them; a Windows path cannot contain `"`).+fn shell_quote(s: &str) -> String {+ if cfg!(windows) {+ format!("\"{s}\"")+ } else {+ format!("'{}'", s.replace('\'', r"'\''"))+ }+}++/// What `xerj init` prints for the clients `.mcp.json` does not cover: the+/// Claude Code user-scope line with THIS binary's absolute path and the node+/// URL `.mcp.json` got (no `$(command -v …)`, which only a POSIX shell+/// expands), then the root key each other client reads. A unit test pins it to+/// the hub skill so the two cannot drift apart.+fn mcp_registration(bin: &str, url: &str) -> String {+ let env = if url == DEFAULT_URL {+ String::new()+ } else {+ format!(" -e XERJ_URL={}", shell_quote(url))+ };+ format!(+ "MCP registration for other clients (restart the session after; success: `{MCP_SUCCESS_TOOL}` is in the tool list):\n\+ \x20 Claude Code, every project: claude mcp add --scope user xerj{env} -- {} mcp\n\+ \x20 Cursor, Windsurf, Claude Desktop, Cline: copy the \"xerj\" entry from .mcp.json; the root key is \"mcpServers\"\n\+ \x20 VS Code (.vscode/mcp.json): root key \"servers\", not \"mcpServers\"; the entry needs \"type\": \"stdio\"",+ shell_quote(bin)+ )+}++/// What init says last: where to go next, the hub's one line, and the+/// per-client MCP registration.+fn closing_message(bin: &str, url: &str) -> String {+ format!(+ "\ndone. Restart the agent session to pick up the tools, then\n\+ `xerj autoindex <folder>` anything you want searchable.\n\n\+ Public libraries, standards and advisories need no indexing: ask the hub with\n\+ `xerj \"<question>\"` (the xerj-hub skill says when).\n\n\+ {}",+ mcp_registration(bin, url)+ )+}++/// This binary's absolute path, the same string `.mcp.json` gets.+fn self_bin() -> String {+ std::env::current_exe()+ .ok()+ .and_then(|p| p.to_str().map(str::to_string))+ .unwrap_or_else(|| "xerj".to_string())+}++/// Write `content` to `path` unless a file is already there, so a skill the+/// user has edited is never overwritten. Returns the report line for it.+fn write_if_absent(path: &Path, content: &str, dry: bool) -> Result<String, String> {+ if path.exists() {+ return Ok(report("kept", path));+ }+ write_file(path, content, dry).map_err(|e| e.to_string())?;+ Ok(report("wrote", path))+}+/// Back up `path` to `path.bak` before the first edit. Returns true if a- /// backup was made (false when the file did not exist).- fn backup(path: &Path, dry: bool) -> bool {+ /// backup was made (false when the file did not exist). A backup that cannot+ /// be written is an error, so the caller never edits a file it could not save+ /// a copy of first.+ fn backup(path: &Path, dry: bool) -> Result<bool, String> {if !path.exists() {- return false;+ return Ok(false);}if !dry {let bak = path.with_extension(match path.extension() {