Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge pull request #1411 from xerj-org/fix/init-backup-fail-closed

xerj-org2e130ae7eaad
diff --cc engine/crates/xerj-autoindex/src/init.rs
index 67da4aa47,d2784fae3..c9fb411fd
--- a/engine/crates/xerj-autoindex/src/init.rs
+++ b/engine/crates/xerj-autoindex/src/init.rs
@@@ -74,193 -68,13 +74,195 @@@ fn agents_md_section() -> String
.to_string()
}
+/// The second teaching file (Corpus Hub v2, plan section 9): when to ask the
+/// hub instead of searching local files, the one line that does it, the exact
+/// form, how to stop a node safely, and the MCP registration per client.
+///
+/// Kept as a raw string rather than a `.md` asset: the repository ignores
+/// `*.md` by default, and a `include_str!` of an ignored file compiles on a
+/// developer machine and fails in CI.
+///
+/// Rules this text follows (docs/research/llms-txt-2026-09/README.md section
+/// 6): no obligation language; every step ends with what success looks like;
+/// the registration is printed per client with the root key that client reads
+/// (`mcpServers` vs VS Code's `servers`); no line over 300 characters.
+const HUB_SKILL_MD: &str = r##"---
+name: xerj-hub
+description: Answer a question about public source code, a standard, a regulation or an advisory feed from a pre-indexed hub corpus with one command, xerj "your question". Use it for exact public material you cannot quote from memory. Not for this project's own files.
+---
+
+# XERJ hub: ask, do not clone
+
+Pick the source before you search:
+
+- **HUB**: public material you cannot quote exactly. A library's source at a pinned commit, a
+ standard, a regulation, an advisory feed.
+- **LOCAL**: files only this machine has. This project, private documents, exports.
+ `xerj autoindex <folder>` once, then `xerj search "<words>"` or `xerj def "<symbol>"`.
+- **NEITHER**: a fact you can already state exactly. Retrieval on memorised content only adds cost.
+
+## Ask the hub
+
+```sh
+xerj "<question>"
+```
+
+One line. No corpus name, no clone, no index. XERJ picks the corpus (your project's dependencies
+first), fetches its signed, pre-indexed pack on first use, and answers. Retrieval is lexical, not
+semantic: use the words and identifiers the source uses. The bare form needs a space in the
+question; for a single word write `xerj ask "<word>"`.
+
+A good answer is three lines, then any `warning:` lines:
+
+```text
+<corpus> pin <date> (<N> days old) <licence> <file>:<line>
+<the passage, or the declaration>
+answered from: local pack
+```
+
+`pin` is the source revision the corpus was built from: check its age when you ask about the current
+version. `answered from:` can also say `hosted hub`. Tell the user which corpus, pin date and licence
+your answer rests on. `warning: approach-only licence` means read the source, do not copy it.
+
+## Exact questions
+
+```sh
+xerj ask --def <SYMBOL>
+```
+
+Use `--def` for a value, a signature or a default. It prints the definition (`file:line`, the
+declaration, and a `value:` line for a constant or macro) and no passages. Add `--corpus <slug>` when
+more than one corpus defines the name.
+
+## When it does not answer
+
+- Exit 4: more than one corpus fits. The candidates print, one per line. Run again with
+ `--corpus <slug>`.
+- Exit 1, `no hub corpus matches`: the hub does not hold it. Say so. Do not answer from memory.
+- Exit 3: a downloaded pack failed its checksum or signature check and was discarded. Do not use it.
+ Tell the user.
+- Exit 5: this build has no hub public key, so no pack can be verified and nothing was downloaded.
+ Tell the user; a retry will not help.
+- `run: xerj hub pull <slug> (<MB> MB)`: the pack is above the automatic download limit. Run that
+ line, then ask again. Success: `pulled <slug> <pin8> <bytes> → <n> indices, <defs> definitions`.
+
+## The node, and where the question goes
+
+If a node answers on `--url`, `$XERJ_URL` or `localhost:9200` (first one set), XERJ uses it as it
+is. If none does, XERJ starts its own on a free loopback port and records its PID.
+Stop only the node you started: `xerj hub stop` for XERJ's own (success: `stopped the hub node
+(pid <N>)` or `no hub node was started by xerj; nothing to stop`); for a node you started by hand,
+kill only the PID you recorded when you started it. Never `pkill xerj` or `killall xerj`: other XERJ nodes may be running
+on this machine.
+
+By default, inside a git repository the question is answered on this machine. Outside one it may be
+sent to the hosted hub. `xerj hub config hosted off` forbids that.
+
+## MCP
+
+`xerj mcp` serves three hub tools. They run the same flow as `xerj ask`:
+- `xerj_hub_ask` `{question, corpus?, mode?, max_tokens?}`: the shell line above, as one call.
+- `xerj_hub_def` `{slug, symbol, k?}`: `xerj ask --corpus <slug> --def <SYMBOL>`.
+- `xerj_hub_find` `{question, k?}`: which corpora could answer, best first. No passages.
+
+A hub tool answers only from packs already on this machine; it never downloads one. When a corpus
+is not here, the answer is the `xerj hub pull <slug>` line: run it in a shell, then ask again.
+Nor does a hub tool fetch the catalogue: it reads the cached copy. If it says no catalogue is
+cached, run `xerj hub sync` in a shell once, then call the tool again.
+
+Register `xerj mcp` once, then restart the session. Success: `xerj_search` is in the tool list.
+Use the absolute path to the binary: an editor started from a desktop icon does not inherit your
+shell PATH. `command -v xerj` prints it (PowerShell: `(Get-Command xerj).Source`).
+
+- Claude Code: `claude mcp add --scope user xerj -- "$(command -v xerj)" mcp`
+ (PowerShell or cmd: put the full path of `xerj.exe` in place of `"$(command -v xerj)"`)
+- `.mcp.json` (Claude Code, project scope; `xerj init` writes it), Cursor, Windsurf, Claude Desktop,
+ Cline: root key `mcpServers`.
+ `{"mcpServers": {"xerj": {"command": "/abs/path/to/xerj", "args": ["mcp"]}}}`
+- VS Code (`.vscode/mcp.json`): root key `servers`, not `mcpServers`. A block under the wrong key is
+ silently ignored.
+ `{"servers": {"xerj": {"type": "stdio", "command": "/abs/path/to/xerj", "args": ["mcp"]}}}`
+"##;
+
+/// The MCP tool whose presence proves `xerj mcp` is registered. It must be a
+/// tool every `xerj mcp` serves. The hub tools (`xerj_hub_*`) are served too,
+/// but `xerj_search` is the proof the skill and the printed text have always
+/// used, and it stays the one check.
+const MCP_SUCCESS_TOOL: &str = "xerj_search";
+
+/// The default node URL; the printed Claude Code line carries `-e XERJ_URL=`
+/// only when init was pointed somewhere else.
+const DEFAULT_URL: &str = "http://localhost:9200";
+
+/// Quote `s` as one argument for the shell the user is most likely to paste
+/// into: POSIX single quotes on Unix, double quotes on Windows (PowerShell and
+/// cmd both accept them; a Windows path cannot contain `"`).
+fn shell_quote(s: &str) -> String {
+ if cfg!(windows) {
+ format!("\"{s}\"")
+ } else {
+ format!("'{}'", s.replace('\'', r"'\''"))
+ }
+}
+
+/// What `xerj init` prints for the clients `.mcp.json` does not cover: the
+/// Claude Code user-scope line with THIS binary's absolute path and the node
+/// URL `.mcp.json` got (no `$(command -v …)`, which only a POSIX shell
+/// expands), then the root key each other client reads. A unit test pins it to
+/// the hub skill so the two cannot drift apart.
+fn mcp_registration(bin: &str, url: &str) -> String {
+ let env = if url == DEFAULT_URL {
+ String::new()
+ } else {
+ format!(" -e XERJ_URL={}", shell_quote(url))
+ };
+ format!(
+ "MCP registration for other clients (restart the session after; success: `{MCP_SUCCESS_TOOL}` is in the tool list):\n\
+ \x20 Claude Code, every project: claude mcp add --scope user xerj{env} -- {} mcp\n\
+ \x20 Cursor, Windsurf, Claude Desktop, Cline: copy the \"xerj\" entry from .mcp.json; the root key is \"mcpServers\"\n\
+ \x20 VS Code (.vscode/mcp.json): root key \"servers\", not \"mcpServers\"; the entry needs \"type\": \"stdio\"",
+ shell_quote(bin)
+ )
+}
+
+/// What init says last: where to go next, the hub's one line, and the
+/// per-client MCP registration.
+fn closing_message(bin: &str, url: &str) -> String {
+ format!(
+ "\ndone. Restart the agent session to pick up the tools, then\n\
+ `xerj autoindex <folder>` anything you want searchable.\n\n\
+ Public libraries, standards and advisories need no indexing: ask the hub with\n\
+ `xerj \"<question>\"` (the xerj-hub skill says when).\n\n\
+ {}",
+ mcp_registration(bin, url)
+ )
+}
+
+/// This binary's absolute path, the same string `.mcp.json` gets.
+fn self_bin() -> String {
+ std::env::current_exe()
+ .ok()
+ .and_then(|p| p.to_str().map(str::to_string))
+ .unwrap_or_else(|| "xerj".to_string())
+}
+
+/// Write `content` to `path` unless a file is already there, so a skill the
+/// user has edited is never overwritten. Returns the report line for it.
+fn write_if_absent(path: &Path, content: &str, dry: bool) -> Result<String, String> {
+ if path.exists() {
+ return Ok(report("kept", path));
+ }
+ write_file(path, content, dry).map_err(|e| e.to_string())?;
+ Ok(report("wrote", path))
+}
+
/// Back up `path` to `path.bak` before the first edit. Returns true if a
- /// backup was made (false when the file did not exist).
- fn backup(path: &Path, dry: bool) -> bool {
+ /// backup was made (false when the file did not exist). A backup that cannot
+ /// be written is an error, so the caller never edits a file it could not save
+ /// a copy of first.
+ fn backup(path: &Path, dry: bool) -> Result<bool, String> {
if !path.exists() {
- return false;
+ return Ok(false);
}
if !dry {
let bak = path.with_extension(match path.extension() {