Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Review node-status diagnostics against partial failures and non-catalog data

xerj-org601f1bbe21d2
diff --cc CHANGELOG.md
index 031595f2d,de99fcf42..6280b1dd9
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,32 -7,8 +7,37 @@@ and this project adheres to [Semantic V
## [Unreleased]
+### Added
+
++- MCP `xerj_status` reports node reachability, cluster health and visible
++ indices before a search. It distinguishes a missing autoindex catalog from
++ other indexed data and preserves failed or malformed probes as errors
++ rather than reporting a successful empty node (#2515).
++
+- Empty autoindex maps now explain that managed corpus slugs and autoindex
+ dataset/index names belong to separate discovery surfaces. CLI diagnostics,
+ MCP empty responses and the `xerj_map` description point to corpus discovery
+ without claiming a corpus exists. Missing catalogs remain errors; auth
+ refusals and successful dataset maps retain their behavior (#2412).
+
+- Public Agent Skills Discovery v0.2.0 index and a documentation-routing skill
+ at `/.well-known/agent-skills/`. The index is generated from the published
+ skill's metadata and exact-byte SHA-256, with a CI drift check.
+
### Fixed
+- SQL source extraction no longer slices statement heads inside a UTF-8
+ character, so unrecognized CJK statement/comment heads do not panic (#2547).
+- Short Markdown files with YAML frontmatter retain their prose body even
+ when bullets or `Key: value` lines also parse as YAML (#2447). Existing
+ frontmatter-title behavior and structured CSV/JSON body detection remain.
+- Empty files keep separate inventory identities instead of being reported
+ as content duplicates of unrelated paths (#2449). Non-empty byte-identical
+ files still deduplicate.
+- A search with both `query` and a top-level `knn` returns `size` hits, not
+ `k` (#2561). The page was widened to `k`, so `size: 5, k: 20` returned
+ 20 hits. `k` still sets how many nearest neighbours join the candidate
+ set, and `from`/`size` page through the union, as in Elasticsearch.
- Index deletion no longer waits for unrelated index statistics while
resolving its selectors. Index listings and health checks also release
their index-map locks before awaiting statistics, so a stalled schema
diff --cc engine/crates/xerj-mcp/src/lib.rs
index 0f5474078,dd096450b..7b08269b7
--- a/engine/crates/xerj-mcp/src/lib.rs
+++ b/engine/crates/xerj-mcp/src/lib.rs
@@@ -1406,6 -1407,25 +1407,24 @@@ pub fn tool_specs_with(include_hub: boo
"required": ["prefix"]
}
},
+ {
+ "name": "xerj_status",
+ "description": concat!(
+ "Is this node up, and what is on it? Answers the \"is it running\" ",
+ "question in one call, without a search and without reading the ",
- "autoindex catalog: three GETs, a few lines of text. Reports the ",
++ "autoindex catalog: up to three GETs with bounded response bodies. Reports the ",
+ "Elasticsearch-compatible version the node claims, cluster health, ",
- "the index count and names, and whether `autoindex-catalog` exists. ",
++ "visible index count and up to 20 names, and whether `autoindex-catalog` ",
++ "appears in that list. A missing catalog does not mean other indices are empty. ",
+ "Call this before concluding that a 404 from xerj_map or ",
- "xerj_search means the node is down: on a running but unindexed node ",
- "that 404 is the empty case, and this tool says so in words. Also ",
- "distinguishes a refused key (the node is up, the credentials are not ",
- "accepted) from nothing listening at all."
++ "xerj_search means the node is down. Failed, refused, malformed or oversized ",
++ "probes remain errors with unknown fields, rather than successful empty results."
+ ),
+ "inputSchema": {
+ "type": "object",
+ "properties": {}
+ }
+ },
{
"name": "xerj_code_search",
"description": concat!(
diff --cc engine/crates/xerj-mcp/src/status.rs
index 000000000,47de20f8b..8a0dec936
mode 000000,100644..100644
--- a/engine/crates/xerj-mcp/src/status.rs
+++ b/engine/crates/xerj-mcp/src/status.rs
@@@ -1,0 -1,402 +1,569 @@@
-//! `xerj_status`: is this node up, and what is on it? (#2515)
++//! `xerj_status`: reachability, cluster health and visible indices (#2515).
+ //!
-//! The field report that opened #2515 probed a node by calling `xerj_map`,
-//! which answered 404 "no autoindex catalog". Read from an agent's side a 404
-//! is hard to tell apart from a connection that never landed, so the reporter
-//! fell back to `netstat` and a hand-written `GET /` to answer "is it running".
-//!
-//! This tool answers that question directly, and separates the states that
-//! were previously one error: not reachable, reachable but refusing the
-//! credentials, reachable and empty, reachable with data.
-//!
-//! It is deliberately cheap — three `GET`s, no search, no catalog read — so it
-//! can be the first call of a session without spending the budget a real query
-//! needs. The answer is a few lines of text.
++//! Uses up to three read-only GETs. A missing autoindex catalog does not mean
++//! that the node has no data: ordinary indices and managed corpora have their
++//! own discovery paths. Failed or malformed probes remain unknown/errors.
+
+ use serde_json::Value;
+
+ use crate::map::CATALOG_INDEX;
+ use crate::{tool_text, transport_failure_text, Ctx};
+
-/// Index rows printed before the rest are summarized as a count, so a node
-/// with hundreds of indices still answers in a readable number of lines.
+ const MAX_LISTED_INDICES: usize = 20;
++const MAX_RESPONSE_BYTES: usize = 256 * 1024;
+
-/// One `GET`, keeping the status even when it is an error: for a liveness
-/// probe a 404 or a 401 is an answer, not a failure.
-async fn get(ctx: &Ctx, path: &str) -> Result<(u16, String), (String, reqwest::Error)> {
++/// Bound each response before parsing, including chunked bodies. A readable
++/// HTTP error is evidence of reachability, not a successful status probe.
++async fn get(ctx: &Ctx, path: &str) -> Result<(u16, String), String> {
+ let url = format!("{}{}", ctx.base_url, path);
+ let mut req = ctx.client.get(&url);
+ if let Some(auth) = &ctx.auth {
+ req = req.header("Authorization", auth);
+ }
- match req.send().await {
- Ok(resp) => {
- let status = resp.status().as_u16();
- Ok((status, resp.text().await.unwrap_or_default()))
++ let mut response = req
++ .send()
++ .await
++ .map_err(|error| transport_failure_text(&ctx.base_url, &url, &error))?;
++ let status = response.status().as_u16();
++ let limit_error = || {
++ format!(
++ "GET {path} answered HTTP {status}, but its response exceeds the {MAX_RESPONSE_BYTES}-byte status-probe limit"
++ )
++ };
++ if response
++ .content_length()
++ .is_some_and(|n| n > MAX_RESPONSE_BYTES as u64)
++ {
++ return Err(limit_error());
++ }
++ let mut body = Vec::new();
++ while let Some(chunk) = response.chunk().await.map_err(|error| {
++ format!(
++ "GET {path} answered HTTP {status}, but its response body could not be read: {}",
++ transport_failure_text(&ctx.base_url, &url, &error)
++ )
++ })? {
++ if chunk.len() > MAX_RESPONSE_BYTES.saturating_sub(body.len()) {
++ return Err(limit_error());
+ }
- Err(e) => Err((url, e)),
++ body.extend_from_slice(&chunk);
+ }
++ let text = String::from_utf8(body)
++ .map_err(|_| format!("GET {path} answered HTTP {status}, but its body is not UTF-8"))?;
++ Ok((status, text))
+ }
+
+ fn parse(text: &str) -> Option<Value> {
+ serde_json::from_str(text).ok()
+ }
+
++/// Keep remote names to a single bounded display line.
++fn display(value: &str) -> String {
++ let mut chars = value.chars();
++ let mut out: String = chars
++ .by_ref()
++ .take(200)
++ .map(|c| if c.is_control() { ' ' } else { c })
++ .collect();
++ if chars.next().is_some() {
++ out.push('…');
++ }
++ out
++}
++
++fn index_rows(text: &str) -> Result<Vec<(String, Option<u64>)>, String> {
++ let value = parse(text).ok_or("index list is not valid JSON")?;
++ let rows = value.as_array().ok_or("index list is not a JSON array")?;
++ rows.iter()
++ .map(|row| {
++ let name = row
++ .get("index")
++ .and_then(Value::as_str)
++ .filter(|name| !name.is_empty())
++ .ok_or("index list contains a row without an index name")?;
++ let docs = row.get("docs.count").and_then(|count| {
++ count
++ .as_u64()
++ .or_else(|| count.as_str()?.parse::<u64>().ok())
++ });
++ Ok((name.to_string(), docs))
++ })
++ .collect()
++}
++
+ pub(crate) async fn run(ctx: &Ctx, _args: &Value) -> Value {
+ let (status, banner_text) = match get(ctx, "/").await {
- Ok(ok) => ok,
- Err((url, e)) => {
++ Ok(response) => response,
++ Err(error) => {
+ return tool_text(
+ format!(
- "xerj_status: no node is answering at {}.\n{}",
- ctx.base_url,
- transport_failure_text(&ctx.base_url, &url, &e)
++ "xerj_status: could not read node status at {}.\n{error}",
++ ctx.base_url
+ ),
+ true,
- );
++ )
+ }
+ };
-
- // A 401/403 answers the liveness question ("up") but nothing else will
- // work, so say that plainly instead of reporting an empty node.
+ if status == 401 || status == 403 {
+ return tool_text(
+ format!(
- "xerj_status: the node at {} is UP but refused these credentials \
- (root endpoint answered HTTP {status}).\n\
- This is an auth problem, not a connection problem: queries will fail \
- the same way. Point XERJ_AUTH (or `xerj mcp --auth`) at a key this node \
- accepts, or ask its owner for the admin key.",
++ "xerj_status: the HTTP service at {} is UP but refused the root request \
++ (HTTP {status}).\nThis is an auth problem, not a connection problem. \
++ Check XERJ_AUTH (or `xerj mcp --auth`) and the permissions for this endpoint. \
++ Other endpoints have not been probed.",
++ ctx.base_url
++ ),
++ true,
++ );
++ }
++ if !(200..300).contains(&status) {
++ return tool_text(
++ format!(
++ "xerj_status: an HTTP service answered at {}, but the root probe failed \
++ with HTTP {status}. Node status is unknown.",
+ ctx.base_url
+ ),
+ true,
+ );
+ }
-
+ let banner = parse(&banner_text);
- let es_version = banner
++ let version = banner
+ .as_ref()
+ .and_then(|b| b.get("version"))
+ .and_then(|v| v.get("number"))
+ .and_then(Value::as_str);
- let cluster_name = banner
- .as_ref()
- .and_then(|b| b.get("cluster_name"))
- .and_then(Value::as_str);
-
++ let Some(version) = version else {
++ return tool_text(
++ format!(
++ "xerj_status: an HTTP service answered at {}, but the root endpoint did not \
++ return a JSON node banner with version.number. Node status is unknown.",
++ ctx.base_url
++ ),
++ true,
++ );
++ };
+ let mut lines = vec![format!(
- "xerj_status: the node at {} is UP (root endpoint answered HTTP {status}).",
- ctx.base_url
++ "xerj_status: the node at {} is UP (root endpoint answered HTTP {status}).", ctx.base_url
++ ), format!(
++ " version: {} (the Elasticsearch-compatible number this node reports, not XERJ's own version)", display(version)
+ )];
- match es_version {
- Some(v) => lines.push(format!(
- " version: {v} (the Elasticsearch-compatible number this node reports, \
- not XERJ's own version)"
- )),
- None => lines.push(
- " version: not reported — the root endpoint did not return a JSON banner".into(),
- ),
- }
- if let Some(name) = cluster_name {
- lines.push(format!(" cluster name: {name}"));
- }
-
- match get(ctx, "/_cluster/health").await {
- Ok((code, text)) => match parse(&text) {
- Some(h) if (200..300).contains(&code) => {
- let state = h.get("status").and_then(Value::as_str).unwrap_or("unknown");
- let nodes = h.get("number_of_nodes").and_then(Value::as_u64);
- let shards = h.get("active_shards").and_then(Value::as_u64);
- let mut facts = Vec::new();
- if let Some(n) = nodes {
- facts.push(format!("{n} node{}", if n == 1 { "" } else { "s" }));
- }
- if let Some(s) = shards {
- facts.push(format!("{s} active shards"));
- }
- lines.push(if facts.is_empty() {
- format!(" cluster health: {state}")
- } else {
- format!(" cluster health: {state} ({})", facts.join(", "))
- });
- }
- _ => lines.push(format!(
- " cluster health: unavailable (GET /_cluster/health answered HTTP {code})"
- )),
- },
- Err((url, e)) => lines.push(format!(
- " cluster health: unavailable ({})",
- transport_failure_text(&ctx.base_url, &url, &e)
- )),
++ if let Some(name) = banner
++ .as_ref()
++ .and_then(|b| b.get("cluster_name"))
++ .and_then(Value::as_str)
++ {
++ lines.push(format!(" cluster name: {}", display(name)));
+ }
-
- let indices = match get(ctx, "/_cat/indices?format=json&h=index,docs.count").await {
++ let mut incomplete = false;
++ let health = match get(ctx, "/_cluster/health").await {
+ Ok((code, text)) if (200..300).contains(&code) => parse(&text)
- .and_then(|v| v.as_array().cloned())
- .map(|rows| {
- rows.iter()
- .filter_map(|r| {
- let name = r.get("index").and_then(Value::as_str)?;
- let docs = r.get("docs.count").and_then(|d| {
- d.as_str()
- .and_then(|s| s.parse::<u64>().ok())
- .or_else(|| d.as_u64())
- });
- Some((name.to_string(), docs))
- })
- .collect::<Vec<_>>()
- }),
- _ => None,
++ .filter(|h| h.get("status").and_then(Value::as_str).is_some())
++ .ok_or_else(|| "GET /_cluster/health did not return a health status".to_string()),
++ Ok((code, _)) => Err(format!("GET /_cluster/health answered HTTP {code}")),
++ Err(error) => Err(error),
+ };
-
- match &indices {
- Some(rows) if rows.is_empty() => {
- lines.push(" indices: none".into());
- }
- Some(rows) => {
++ match health {
++ Ok(h) => {
++ let state = h["status"].as_str().expect("validated health status");
++ let mut facts = Vec::new();
++ if let Some(n) = h.get("number_of_nodes").and_then(Value::as_u64) {
++ facts.push(format!("{n} node{}", if n == 1 { "" } else { "s" }));
++ }
++ if let Some(n) = h.get("active_shards").and_then(Value::as_u64) {
++ facts.push(format!("{n} active shards"));
++ }
+ lines.push(format!(
- " indices: {}{}",
- rows.len(),
- if rows.len() > MAX_LISTED_INDICES {
- format!(" (first {MAX_LISTED_INDICES} listed)")
- } else {
++ " cluster health: {}{}",
++ display(state),
++ if facts.is_empty() {
+ String::new()
++ } else {
++ format!(" ({})", facts.join(", "))
+ }
+ ));
- let width = rows
- .iter()
- .take(MAX_LISTED_INDICES)
- .map(|(n, _)| n.len())
- .max()
- .unwrap_or(0);
- for (name, docs) in rows.iter().take(MAX_LISTED_INDICES) {
- match docs {
- Some(d) => lines.push(format!(" {name:<width$} {d} docs")),
- None => lines.push(format!(" {name}")),
- }
- }
+ }
- None => lines
- .push(" indices: unavailable (GET /_cat/indices did not return a JSON list)".into()),
++ Err(error) => {
++ incomplete = true;
++ lines.push(format!(" cluster health: unavailable ({error})"));
++ }
+ }
-
- // The point of the whole tool: an empty node must not read as a broken
- // one. `xerj_map` answers 404 "no autoindex catalog" here, and that is
- // the node saying it has nothing indexed — not a connectivity failure.
- match &indices {
- Some(rows) => {
- let catalog = rows.iter().find(|(name, _)| name == CATALOG_INDEX);
- match catalog {
- Some((_, docs)) => lines.push(format!(
- " autoindex catalog: PRESENT (`{CATALOG_INDEX}`{}, so xerj_map and \
- xerj_search have data to read).",
- match docs {
- Some(d) => format!(" holds {d} docs"),
- None => String::new(),
++ let indices = match get(ctx, "/_cat/indices?format=json&h=index,docs.count").await {
++ Ok((code, text)) if (200..300).contains(&code) => index_rows(&text),
++ Ok((code, _)) => Err(format!("GET /_cat/indices answered HTTP {code}")),
++ Err(error) => Err(error),
++ };
++ match indices {
++ Ok(rows) => {
++ if rows.is_empty() {
++ lines.push(" indices: none returned for these credentials".into());
++ } else {
++ lines.push(format!(
++ " indices: {} visible{}",
++ rows.len(),
++ if rows.len() > MAX_LISTED_INDICES {
++ format!(" (first {MAX_LISTED_INDICES} listed)")
++ } else {
++ String::new()
+ }
- )),
- None => {
- lines.push(
- " autoindex catalog: ABSENT — this node is running and has nothing \
- indexed yet."
- .into(),
- );
- lines.push(
- " xerj_map and xerj_search answer \"no autoindex catalog\" on this \
- node. That is an empty node, not a connection failure: index a folder \
- with `xerj autoindex <folder>`, then call this tool again."
- .into(),
- );
++ ));
++ for (name, docs) in rows.iter().take(MAX_LISTED_INDICES) {
++ lines.push(match docs {
++ Some(n) => format!(" {} {n} docs", display(name)),
++ None => format!(" {} document count unknown", display(name)),
++ });
+ }
+ }
++ if let Some((_, docs)) = rows.iter().find(|(name, _)| name == CATALOG_INDEX) {
++ lines.push(format!(
++ " autoindex catalog: PRESENT in the index list (`{CATALOG_INDEX}`{}).",
++ docs.map(|n| format!(" holds {n} docs")).unwrap_or_default()
++ ));
++ lines.push(" Catalog presence alone does not establish dataset availability; use xerj_map to inspect it.".into());
++ } else {
++ lines.push(" autoindex catalog: ABSENT from the returned index list.".into());
++ lines.push(" This is not a connection failure. Ordinary indices may still be searchable, and managed corpora have a separate namespace.".into());
++ lines.push(" Use `xerj autoindex <folder>` to create an autoindex catalog; use xerj_map to inspect catalog datasets.".into());
++ }
++ }
++ Err(error) => {
++ incomplete = true;
++ lines.push(format!(" indices: unavailable ({error})"));
++ lines.push(format!(" autoindex catalog: unknown — whether `{CATALOG_INDEX}` exists was not established."));
+ }
- None => lines.push(format!(
- " autoindex catalog: unknown — the index list could not be read, so whether \
- `{CATALOG_INDEX}` exists is not answered here."
- )),
+ }
-
- tool_text(lines.join("\n"), false)
++ tool_text(lines.join("\n"), incomplete)
+ }
+
+ #[cfg(test)]
+ mod tests {
+ use super::*;
+ use serde_json::json;
+ use std::collections::HashMap;
+
+ /// Serve a fixed route table on a loopback port. Keys are paths without
+ /// their query string, because the real calls carry `?format=json&...`.
+ async fn serve(routes: HashMap<&'static str, (u16, String)>) -> String {
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let addr = listener.local_addr().unwrap();
+ tokio::spawn(async move {
+ loop {
+ let Ok((mut sock, _)) = listener.accept().await else {
+ return;
+ };
+ let mut buf = vec![0u8; 8192];
+ let n = tokio::io::AsyncReadExt::read(&mut sock, &mut buf)
+ .await
+ .unwrap_or(0);
+ let request = String::from_utf8_lossy(&buf[..n]).to_string();
+ let target = request.split_whitespace().nth(1).unwrap_or("/");
+ let path = target.split('?').next().unwrap_or("/").to_string();
+ let (status, body) = routes
+ .get(path.as_str())
+ .cloned()
+ .unwrap_or((404, "{}".to_string()));
+ let reply = format!(
+ "HTTP/1.1 {status} X\r\nContent-Length: {}\r\nConnection: close\r\n\
+ Content-Type: application/json\r\n\r\n{body}",
+ body.len()
+ );
+ let _ = tokio::io::AsyncWriteExt::write_all(&mut sock, reply.as_bytes()).await;
+ }
+ });
+ format!("http://{addr}")
+ }
+
+ fn ctx(base_url: String) -> Ctx {
+ Ctx {
- client: reqwest::Client::new(),
++ client: crate::node_http_builder(&base_url)
++ .timeout(std::time::Duration::from_secs(5))
++ .build()
++ .unwrap(),
+ base_url,
+ auth: None,
+ }
+ }
+
+ fn banner() -> String {
+ json!({
+ "name": "node-1",
+ "cluster_name": "xerj",
+ "version": { "number": "8.13.0", "distribution": "xerj" },
+ "tagline": "You Know, for Search"
+ })
+ .to_string()
+ }
+
+ fn health(state: &str) -> String {
+ json!({
+ "cluster_name": "xerj",
+ "status": state,
+ "number_of_nodes": 1,
+ "active_shards": 15
+ })
+ .to_string()
+ }
+
+ fn cat(rows: Value) -> String {
+ rows.to_string()
+ }
+
+ fn text(v: &Value) -> String {
+ v["content"][0]["text"]
+ .as_str()
+ .unwrap_or_default()
+ .to_string()
+ }
+
+ /// The bug in #2515: `xerj_map` 404s on an unindexed node, and from the
+ /// agent's side that reads like the node is not there. The status tool has
+ /// to say UP and EMPTY in one answer, and name the 404 as the empty case.
+ #[tokio::test]
+ async fn an_unindexed_node_reports_up_and_empty_not_unreachable() {
+ let mut routes = HashMap::new();
+ routes.insert("/", (200, banner()));
+ routes.insert("/_cluster/health", (200, health("green")));
+ routes.insert("/_cat/indices", (200, cat(json!([]))));
+ let base = serve(routes).await;
+
+ let out = run(&ctx(base), &json!({})).await;
+ assert_eq!(out["isError"], false, "{}", text(&out));
+ let body = text(&out);
+ assert!(body.contains("is UP"), "{body}");
+ assert!(body.contains("indices: none"), "{body}");
+ assert!(body.contains("autoindex catalog: ABSENT"), "{body}");
+ assert!(
+ body.contains("not a connection failure"),
+ "the empty node must not read as a broken one: {body}"
+ );
+ assert!(body.contains("xerj autoindex <folder>"), "{body}");
+ assert!(body.contains("8.13.0"), "{body}");
+ assert!(body.contains("green"), "{body}");
+ }
+
+ #[tokio::test]
+ async fn an_indexed_node_names_the_catalog_and_its_doc_count() {
+ let mut routes = HashMap::new();
+ routes.insert("/", (200, banner()));
+ routes.insert("/_cluster/health", (200, health("yellow")));
+ routes.insert(
+ "/_cat/indices",
+ (
+ 200,
+ cat(json!([
+ { "index": "autoindex-catalog", "docs.count": "12" },
+ { "index": "ax", "docs.count": "812" }
+ ])),
+ ),
+ );
+ let base = serve(routes).await;
+
+ let out = run(&ctx(base), &json!({})).await;
+ assert_eq!(out["isError"], false, "{}", text(&out));
+ let body = text(&out);
+ assert!(body.contains("indices: 2"), "{body}");
+ assert!(body.contains("autoindex-catalog"), "{body}");
+ assert!(body.contains("12 docs"), "{body}");
+ assert!(body.contains("autoindex catalog: PRESENT"), "{body}");
+ assert!(!body.contains("ABSENT"), "{body}");
+ assert!(body.contains("yellow"), "{body}");
+ }
+
+ /// Nothing listening: the one case that *is* a connectivity failure, and it
+ /// has to say so rather than printing an empty successful answer.
+ #[tokio::test]
+ async fn nothing_listening_is_the_only_unreachable_answer() {
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let addr = listener.local_addr().unwrap();
+ drop(listener);
+
+ let out = run(&ctx(format!("http://{addr}")), &json!({})).await;
+ assert_eq!(out["isError"], true, "{}", text(&out));
+ let body = text(&out);
- assert!(body.contains("no node is answering"), "{body}");
++ assert!(body.contains("could not read node status"), "{body}");
+ }
+
+ /// A 401 means the node is running. Reporting that as "unreachable" would
+ /// send an agent looking at ports when the problem is the key.
+ #[tokio::test]
+ async fn a_refused_key_is_up_with_an_auth_problem_not_down() {
+ let mut routes = HashMap::new();
+ routes.insert("/", (401, "{\"error\":\"unauthorized\"}".to_string()));
+ let base = serve(routes).await;
+
+ let out = run(&ctx(base), &json!({})).await;
+ let body = text(&out);
+ assert!(body.contains("is UP but refused"), "{body}");
+ assert!(
+ body.contains("auth problem, not a connection problem"),
+ "{body}"
+ );
+ assert!(body.contains("XERJ_AUTH"), "{body}");
+ }
+
+ /// A node that answers the banner but not `_cat/indices` is up. The catalog
+ /// question is then unknown, which must not be reported as absent.
+ #[tokio::test]
+ async fn an_unreadable_index_list_is_unknown_not_absent() {
+ let mut routes = HashMap::new();
+ routes.insert("/", (200, banner()));
+ routes.insert("/_cluster/health", (200, health("green")));
+ routes.insert("/_cat/indices", (500, "boom".to_string()));
+ let base = serve(routes).await;
+
+ let out = run(&ctx(base), &json!({})).await;
- assert_eq!(out["isError"], false, "{}", text(&out));
++ assert_eq!(out["isError"], true, "{}", text(&out));
+ let body = text(&out);
+ assert!(body.contains("autoindex catalog: unknown"), "{body}");
+ assert!(!body.contains("ABSENT"), "{body}");
+ }
++ #[tokio::test]
++ async fn ordinary_indices_without_a_catalog_are_not_an_empty_node() {
++ let routes = HashMap::from([
++ ("/", (200, banner())),
++ ("/_cluster/health", (200, health("green"))),
++ (
++ "/_cat/indices",
++ (200, cat(json!([{"index":"logs", "docs.count":"42"}]))),
++ ),
++ ]);
++ let out = run(&ctx(serve(routes).await), &json!({})).await;
++ assert_eq!(out["isError"], false);
++ let body = text(&out);
++ assert!(body.contains("logs"), "{body}");
++ assert!(body.contains("42 docs"), "{body}");
++ assert!(!body.contains("has nothing indexed"), "{body}");
++ assert!(!body.contains("That is an empty node"), "{body}");
++ assert!(!body.contains("xerj_search answer"), "{body}");
++ }
++
++ #[tokio::test]
++ async fn malformed_index_rows_remain_unknown_instead_of_becoming_empty() {
++ let routes = HashMap::from([
++ ("/", (200, banner())),
++ ("/_cluster/health", (200, health("green"))),
++ ("/_cat/indices", (200, cat(json!([{"docs.count":"42"}])))),
++ ]);
++ let out = run(&ctx(serve(routes).await), &json!({})).await;
++ assert_eq!(out["isError"], true, "{}", text(&out));
++ let body = text(&out);
++ assert!(body.contains("autoindex catalog: unknown"), "{body}");
++ assert!(!body.contains("indices: none"), "{body}");
++ assert!(!body.contains("ABSENT"), "{body}");
++ }
++
++ #[tokio::test]
++ async fn root_failures_and_non_node_banners_are_errors() {
++ for (code, reply) in [
++ (503, "{}"),
++ (404, "{}"),
++ (200, "<html>other service</html>"),
++ (200, "[]"),
++ ] {
++ let routes = HashMap::from([("/", (code, reply.to_string()))]);
++ let out = run(&ctx(serve(routes).await), &json!({})).await;
++ assert_eq!(out["isError"], true, "{}", text(&out));
++ assert!(!text(&out).contains("node at"), "{}", text(&out));
++ }
++ }
++
++ #[tokio::test]
++ async fn endpoint_auth_refusals_are_errors_not_absence() {
++ for endpoint in ["/_cluster/health", "/_cat/indices"] {
++ for code in [401, 403] {
++ let mut routes = HashMap::from([
++ ("/", (200, banner())),
++ ("/_cluster/health", (200, health("green"))),
++ ("/_cat/indices", (200, cat(json!([])))),
++ ]);
++ routes.insert(endpoint, (code, "{}".to_string()));
++ let out = run(&ctx(serve(routes).await), &json!({})).await;
++ assert_eq!(out["isError"], true, "{}", text(&out));
++ assert!(
++ text(&out).contains(&format!("HTTP {code}")),
++ "{}",
++ text(&out)
++ );
++ if endpoint == "/_cat/indices" {
++ assert!(!text(&out).contains("ABSENT"), "{}", text(&out));
++ }
++ }
++ }
++ }
++
++ #[tokio::test]
++ async fn an_empty_catalog_does_not_promise_searchable_datasets() {
++ let routes = HashMap::from([
++ ("/", (200, banner())),
++ ("/_cluster/health", (200, health("green"))),
++ (
++ "/_cat/indices",
++ (200, cat(json!([{"index":CATALOG_INDEX,"docs.count":"0"}]))),
++ ),
++ ]);
++ let out = run(&ctx(serve(routes).await), &json!({})).await;
++ assert_eq!(out["isError"], false);
++ assert!(text(&out).contains("PRESENT"));
++ assert!(text(&out).contains("0 docs"));
++ assert!(!text(&out).contains("have data to read"));
++ }
++
++ #[tokio::test]
++ async fn an_oversized_index_response_is_unknown_not_empty() {
++ let rows = cat(json!([{"index":"x".repeat(256 * 1024),"docs.count":"1"}]));
++ let routes = HashMap::from([
++ ("/", (200, banner())),
++ ("/_cluster/health", (200, health("green"))),
++ ("/_cat/indices", (200, rows)),
++ ]);
++ let out = run(&ctx(serve(routes).await), &json!({})).await;
++ assert_eq!(out["isError"], true, "{}", text(&out));
++ assert!(text(&out).contains("status-probe limit"));
++ assert!(text(&out).contains("catalog: unknown"));
++ }
++
++ async fn raw_root(reply: Vec<u8>) -> String {
++ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
++ let address = listener.local_addr().unwrap();
++ tokio::spawn(async move {
++ let (mut socket, _) = listener.accept().await.unwrap();
++ let mut request = Vec::new();
++ while request.len() < 8192 && !request.windows(4).any(|w| w == b"\r\n\r\n") {
++ let mut chunk = [0; 1024];
++ let n = tokio::io::AsyncReadExt::read(&mut socket, &mut chunk)
++ .await
++ .unwrap();
++ if n == 0 {
++ return;
++ }
++ request.extend_from_slice(&chunk[..n]);
++ }
++ let _ = tokio::io::AsyncWriteExt::write_all(&mut socket, &reply).await;
++ });
++ format!("http://{address}")
++ }
++
++ #[tokio::test]
++ async fn truncated_http_bodies_are_errors_instead_of_empty_text() {
++ let base = raw_root(
++ b"HTTP/1.1 200 OK\r\nContent-Length: 100\r\nConnection: close\r\n\r\n{}".to_vec(),
++ )
++ .await;
++ let result = get(&ctx(base), "/").await;
++ assert!(
++ result.is_err(),
++ "an incomplete response must not become an empty successful body"
++ );
++ }
++
++ #[tokio::test]
++ async fn chunked_responses_are_also_bounded_before_parsing() {
++ let payload = "x".repeat(256 * 1024 + 1);
++ let reply = format!("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\nConnection: close\r\n\r\n{:x}\r\n{payload}\r\n0\r\n\r\n", payload.len());
++ let base = raw_root(reply.into_bytes()).await;
++ let result = get(&ctx(base), "/").await;
++ assert!(
++ result.is_err(),
++ "a missing Content-Length must not bypass the body limit"
++ );
++ }
+ }
diff --cc landing/docs/cli.html
index e91fe7473,69a334f0b..2f707df18
--- a/landing/docs/cli.html
+++ b/landing/docs/cli.html
@@@ -530,7 -530,7 +530,7 @@@ Grok CLI | ~/.grok/conf
<p>Aider is absent because it has no MCP client support, T3 Code hosts an MCP endpoint rather than consuming one, and Grok Bot has no documented local stdio registration.</p>
- <p>Twenty-two tools are served (nineteen in a build with no hub key, which omits the three hub tools). Eleven are thin proxies over an endpoint the engine already exposes, and engine errors come back verbatim: <code class="inline">xerj_search</code>, <code class="inline">xerj_semantic_search</code>, <code class="inline">xerj_vector_search</code>, <code class="inline">xerj_hybrid_search</code>, <code class="inline">xerj_memory_store</code>, <code class="inline">xerj_memory_recall</code>, <code class="inline">xerj_brain_overview</code>, <code class="inline">xerj_brain_ego</code>, <code class="inline">xerj_brain_link</code>, <code class="inline">xerj_brain_unlink</code>, <code class="inline">xerj_plan</code> (<code class="inline">POST /_ask</code>). Two are not plain proxies: <code class="inline">xerj_map</code> returns the autoindex catalog's per-index field map (trimmed to a byte cap), and <code class="inline">xerj_code_search</code> runs the <code class="inline">xerj code</code> pipeline and returns its rendered text. <code class="inline">xerj_field_report</code> files an agent field report (a GitHub PR first, the <code class="inline">/api/field-report</code> endpoint when GitHub is unavailable; it sends by default). <code class="inline">xerj_index_budget</code> counts a folder and estimates its index against the node's disk budget before <code class="inline">xerj autoindex</code> hashes anything (it does not index), <code class="inline">xerj_autoindex_remove</code> drops one autoindexed folder after listing it and deletes nothing unless <code class="inline">confirm</code> is true, and <code class="inline">xerj_corpus_feedback</code> rates a hub corpus. <code class="inline">xerj_def</code> (below) finds local definitions, and <code class="inline">xerj_batch_search</code> runs 1 to 50 IDs or questions in one call, in input order, through the same adapter as <code class="inline">xerj batch-search</code>. The three Corpus Hub tools, <code class="inline">xerj_hub_ask</code>, <code class="inline">xerj_hub_def</code> and <code class="inline">xerj_hub_find</code>, run the <a href="#ask"><code class="inline">xerj ask</code></a> flow, but never download a pack and never fetch the catalogue: run <code class="inline">xerj hub sync</code> and <code class="inline">xerj hub pull &lt;slug&gt;</code> in a shell first (details on the <a href="/docs/corpus-hub#mcp">Corpus Hub page</a>). Two caveats the tool descriptions carry themselves: <code class="inline">xerj_semantic_search</code> embeds server-side with the <em>lexical</em> feature-hashing embedder unless the node runs <code class="inline">--embed-mode neural</code>, and the <code class="inline">xerj_brain_*</code> tools work a deterministic link index, not a graph database.</p>
- <p>Twenty-three tools are served (twenty in a build with no hub key, which omits the three hub tools). Eleven are thin proxies over an endpoint the engine already exposes, and engine errors come back verbatim: <code class="inline">xerj_search</code>, <code class="inline">xerj_semantic_search</code>, <code class="inline">xerj_vector_search</code>, <code class="inline">xerj_hybrid_search</code>, <code class="inline">xerj_memory_store</code>, <code class="inline">xerj_memory_recall</code>, <code class="inline">xerj_brain_overview</code>, <code class="inline">xerj_brain_ego</code>, <code class="inline">xerj_brain_link</code>, <code class="inline">xerj_brain_unlink</code>, <code class="inline">xerj_plan</code> (<code class="inline">POST /_ask</code>). Two are not plain proxies: <code class="inline">xerj_map</code> returns the autoindex catalog's per-index field map (trimmed to a byte cap), and <code class="inline">xerj_code_search</code> runs the <code class="inline">xerj code</code> pipeline and returns its rendered text. <code class="inline">xerj_field_report</code> files an agent field report (a GitHub PR first, the <code class="inline">/api/field-report</code> endpoint when GitHub is unavailable; it sends by default). <code class="inline">xerj_index_budget</code> counts a folder and estimates its index against the node's disk budget before <code class="inline">xerj autoindex</code> hashes anything (it does not index), <code class="inline">xerj_autoindex_remove</code> drops one autoindexed folder after listing it and deletes nothing unless <code class="inline">confirm</code> is true, <code class="inline">xerj_status</code> answers "is this node up, and is anything indexed" with three GETs so an empty node's 404 is not mistaken for a node that is not there, and <code class="inline">xerj_corpus_feedback</code> rates a hub corpus. <code class="inline">xerj_def</code> (below) finds local definitions, and <code class="inline">xerj_batch_search</code> runs 1 to 50 IDs or questions in one call, in input order, through the same adapter as <code class="inline">xerj batch-search</code>. The three Corpus Hub tools, <code class="inline">xerj_hub_ask</code>, <code class="inline">xerj_hub_def</code> and <code class="inline">xerj_hub_find</code>, run the <a href="#ask"><code class="inline">xerj ask</code></a> flow, but never download a pack and never fetch the catalogue: run <code class="inline">xerj hub sync</code> and <code class="inline">xerj hub pull &lt;slug&gt;</code> in a shell first (details on the <a href="/docs/corpus-hub#mcp">Corpus Hub page</a>). Two caveats the tool descriptions carry themselves: <code class="inline">xerj_semantic_search</code> embeds server-side with the <em>lexical</em> feature-hashing embedder unless the node runs <code class="inline">--embed-mode neural</code>, and the <code class="inline">xerj_brain_*</code> tools work a deterministic link index, not a graph database.</p>
++ <p>Twenty-three tools are served (twenty in a build with no hub key, which omits the three hub tools). Eleven are thin proxies over an endpoint the engine already exposes, and engine errors come back verbatim: <code class="inline">xerj_search</code>, <code class="inline">xerj_semantic_search</code>, <code class="inline">xerj_vector_search</code>, <code class="inline">xerj_hybrid_search</code>, <code class="inline">xerj_memory_store</code>, <code class="inline">xerj_memory_recall</code>, <code class="inline">xerj_brain_overview</code>, <code class="inline">xerj_brain_ego</code>, <code class="inline">xerj_brain_link</code>, <code class="inline">xerj_brain_unlink</code>, <code class="inline">xerj_plan</code> (<code class="inline">POST /_ask</code>). Two are not plain proxies: <code class="inline">xerj_map</code> returns the autoindex catalog's per-index field map (trimmed to a byte cap), and <code class="inline">xerj_code_search</code> runs the <code class="inline">xerj code</code> pipeline and returns its rendered text. <code class="inline">xerj_field_report</code> files an agent field report (a GitHub PR first, the <code class="inline">/api/field-report</code> endpoint when GitHub is unavailable; it sends by default). <code class="inline">xerj_index_budget</code> counts a folder and estimates its index against the node's disk budget before <code class="inline">xerj autoindex</code> hashes anything (it does not index), <code class="inline">xerj_autoindex_remove</code> drops one autoindexed folder after listing it and deletes nothing unless <code class="inline">confirm</code> is true, <code class="inline">xerj_status</code> probes node reachability, cluster health and visible indices with up to three GETs; a missing autoindex catalog does not imply that other indices are empty, and failed probes remain errors with unknown fields, and <code class="inline">xerj_corpus_feedback</code> rates a hub corpus. <code class="inline">xerj_def</code> (below) finds local definitions, and <code class="inline">xerj_batch_search</code> runs 1 to 50 IDs or questions in one call, in input order, through the same adapter as <code class="inline">xerj batch-search</code>. The three Corpus Hub tools, <code class="inline">xerj_hub_ask</code>, <code class="inline">xerj_hub_def</code> and <code class="inline">xerj_hub_find</code>, run the <a href="#ask"><code class="inline">xerj ask</code></a> flow, but never download a pack and never fetch the catalogue: run <code class="inline">xerj hub sync</code> and <code class="inline">xerj hub pull &lt;slug&gt;</code> in a shell first (details on the <a href="/docs/corpus-hub#mcp">Corpus Hub page</a>). Two caveats the tool descriptions carry themselves: <code class="inline">xerj_semantic_search</code> embeds server-side with the <em>lexical</em> feature-hashing embedder unless the node runs <code class="inline">--embed-mode neural</code>, and the <code class="inline">xerj_brain_*</code> tools work a deterministic link index, not a graph database.</p>
<p><code class="inline">xerj_def</code> finds local definitions with the query, lexical ranking and location deduplication of <code class="inline">xerj def --json</code>. For example, <code class="inline">{"symbol":"Cache","prefix":"project","kind":"struct","lang":"rust","k":5}</code> searches <code class="inline">project-*</code>. Use <code class="inline">index</code> instead of <code class="inline">prefix</code> for an explicit index; non-empty scope arguments cannot be combined, and with neither it searches <code class="inline">ax-*</code>. <code class="inline">symbol</code> also accepts capability phrases. <code class="inline">k</code> is 1–50 (default 5); optional <code class="inline">max_tokens</code> caps the response. Declarations are in <code class="inline">hits.hits[]._source.code</code>, locations in <code class="inline">ax_path</code>/<code class="inline">path</code> and <code class="inline">line</code>/<code class="inline">start_line</code>; older file hits may use <code class="inline">fields._passage</code>. <code class="inline">hits.total</code> counts engine matches before deduplication; no match is a successful empty hit list. As in the CLI, kind/language filters are ignored when no target index maps the field or any maps it as numeric. This queries the configured node; <code class="inline">xerj_hub_def</code> instead searches a named Hub pack.</p>