AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Review node-status diagnostics against partial failures and non-catalog data
xerj-org601f1bbe21d2
diff --cc CHANGELOG.mdindex 031595f2d,de99fcf42..6280b1dd9--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,32 -7,8 +7,37 @@@ and this project adheres to [Semantic V## [Unreleased]+### Added+++- MCP `xerj_status` reports node reachability, cluster health and visible++ indices before a search. It distinguishes a missing autoindex catalog from++ other indexed data and preserves failed or malformed probes as errors++ rather than reporting a successful empty node (#2515).+++- Empty autoindex maps now explain that managed corpus slugs and autoindex+ dataset/index names belong to separate discovery surfaces. CLI diagnostics,+ MCP empty responses and the `xerj_map` description point to corpus discovery+ without claiming a corpus exists. Missing catalogs remain errors; auth+ refusals and successful dataset maps retain their behavior (#2412).++- Public Agent Skills Discovery v0.2.0 index and a documentation-routing skill+ at `/.well-known/agent-skills/`. The index is generated from the published+ skill's metadata and exact-byte SHA-256, with a CI drift check.+### Fixed+- SQL source extraction no longer slices statement heads inside a UTF-8+ character, so unrecognized CJK statement/comment heads do not panic (#2547).+- Short Markdown files with YAML frontmatter retain their prose body even+ when bullets or `Key: value` lines also parse as YAML (#2447). Existing+ frontmatter-title behavior and structured CSV/JSON body detection remain.+- Empty files keep separate inventory identities instead of being reported+ as content duplicates of unrelated paths (#2449). Non-empty byte-identical+ files still deduplicate.+- A search with both `query` and a top-level `knn` returns `size` hits, not+ `k` (#2561). The page was widened to `k`, so `size: 5, k: 20` returned+ 20 hits. `k` still sets how many nearest neighbours join the candidate+ set, and `from`/`size` page through the union, as in Elasticsearch.- Index deletion no longer waits for unrelated index statistics whileresolving its selectors. Index listings and health checks also releasetheir index-map locks before awaiting statistics, so a stalled schemadiff --cc engine/crates/xerj-mcp/src/lib.rsindex 0f5474078,dd096450b..7b08269b7--- a/engine/crates/xerj-mcp/src/lib.rs+++ b/engine/crates/xerj-mcp/src/lib.rs@@@ -1406,6 -1407,25 +1407,24 @@@ pub fn tool_specs_with(include_hub: boo"required": ["prefix"]}},+ {+ "name": "xerj_status",+ "description": concat!(+ "Is this node up, and what is on it? Answers the \"is it running\" ",+ "question in one call, without a search and without reading the ",- "autoindex catalog: three GETs, a few lines of text. Reports the ",++ "autoindex catalog: up to three GETs with bounded response bodies. Reports the ",+ "Elasticsearch-compatible version the node claims, cluster health, ",- "the index count and names, and whether `autoindex-catalog` exists. ",++ "visible index count and up to 20 names, and whether `autoindex-catalog` ",++ "appears in that list. A missing catalog does not mean other indices are empty. ",+ "Call this before concluding that a 404 from xerj_map or ",- "xerj_search means the node is down: on a running but unindexed node ",- "that 404 is the empty case, and this tool says so in words. Also ",- "distinguishes a refused key (the node is up, the credentials are not ",- "accepted) from nothing listening at all."++ "xerj_search means the node is down. Failed, refused, malformed or oversized ",++ "probes remain errors with unknown fields, rather than successful empty results."+ ),+ "inputSchema": {+ "type": "object",+ "properties": {}+ }+ },{"name": "xerj_code_search","description": concat!(diff --cc engine/crates/xerj-mcp/src/status.rsindex 000000000,47de20f8b..8a0dec936mode 000000,100644..100644--- a/engine/crates/xerj-mcp/src/status.rs+++ b/engine/crates/xerj-mcp/src/status.rs@@@ -1,0 -1,402 +1,569 @@@-//! `xerj_status`: is this node up, and what is on it? (#2515)++//! `xerj_status`: reachability, cluster health and visible indices (#2515).+ //!-//! The field report that opened #2515 probed a node by calling `xerj_map`,-//! which answered 404 "no autoindex catalog". Read from an agent's side a 404-//! is hard to tell apart from a connection that never landed, so the reporter-//! fell back to `netstat` and a hand-written `GET /` to answer "is it running".-//!-//! This tool answers that question directly, and separates the states that-//! were previously one error: not reachable, reachable but refusing the-//! credentials, reachable and empty, reachable with data.-//!-//! It is deliberately cheap — three `GET`s, no search, no catalog read — so it-//! can be the first call of a session without spending the budget a real query-//! needs. The answer is a few lines of text.++//! Uses up to three read-only GETs. A missing autoindex catalog does not mean++//! that the node has no data: ordinary indices and managed corpora have their++//! own discovery paths. Failed or malformed probes remain unknown/errors.++ use serde_json::Value;++ use crate::map::CATALOG_INDEX;+ use crate::{tool_text, transport_failure_text, Ctx};+-/// Index rows printed before the rest are summarized as a count, so a node-/// with hundreds of indices still answers in a readable number of lines.+ const MAX_LISTED_INDICES: usize = 20;++const MAX_RESPONSE_BYTES: usize = 256 * 1024;+-/// One `GET`, keeping the status even when it is an error: for a liveness-/// probe a 404 or a 401 is an answer, not a failure.-async fn get(ctx: &Ctx, path: &str) -> Result<(u16, String), (String, reqwest::Error)> {++/// Bound each response before parsing, including chunked bodies. A readable++/// HTTP error is evidence of reachability, not a successful status probe.++async fn get(ctx: &Ctx, path: &str) -> Result<(u16, String), String> {+ let url = format!("{}{}", ctx.base_url, path);+ let mut req = ctx.client.get(&url);+ if let Some(auth) = &ctx.auth {+ req = req.header("Authorization", auth);+ }- match req.send().await {- Ok(resp) => {- let status = resp.status().as_u16();- Ok((status, resp.text().await.unwrap_or_default()))++ let mut response = req++ .send()++ .await++ .map_err(|error| transport_failure_text(&ctx.base_url, &url, &error))?;++ let status = response.status().as_u16();++ let limit_error = || {++ format!(++ "GET {path} answered HTTP {status}, but its response exceeds the {MAX_RESPONSE_BYTES}-byte status-probe limit"++ )++ };++ if response++ .content_length()++ .is_some_and(|n| n > MAX_RESPONSE_BYTES as u64)++ {++ return Err(limit_error());++ }++ let mut body = Vec::new();++ while let Some(chunk) = response.chunk().await.map_err(|error| {++ format!(++ "GET {path} answered HTTP {status}, but its response body could not be read: {}",++ transport_failure_text(&ctx.base_url, &url, &error)++ )++ })? {++ if chunk.len() > MAX_RESPONSE_BYTES.saturating_sub(body.len()) {++ return Err(limit_error());+ }- Err(e) => Err((url, e)),++ body.extend_from_slice(&chunk);+ }++ let text = String::from_utf8(body)++ .map_err(|_| format!("GET {path} answered HTTP {status}, but its body is not UTF-8"))?;++ Ok((status, text))+ }++ fn parse(text: &str) -> Option<Value> {+ serde_json::from_str(text).ok()+ }+++/// Keep remote names to a single bounded display line.++fn display(value: &str) -> String {++ let mut chars = value.chars();++ let mut out: String = chars++ .by_ref()++ .take(200)++ .map(|c| if c.is_control() { ' ' } else { c })++ .collect();++ if chars.next().is_some() {++ out.push('…');++ }++ out++}++++fn index_rows(text: &str) -> Result<Vec<(String, Option<u64>)>, String> {++ let value = parse(text).ok_or("index list is not valid JSON")?;++ let rows = value.as_array().ok_or("index list is not a JSON array")?;++ rows.iter()++ .map(|row| {++ let name = row++ .get("index")++ .and_then(Value::as_str)++ .filter(|name| !name.is_empty())++ .ok_or("index list contains a row without an index name")?;++ let docs = row.get("docs.count").and_then(|count| {++ count++ .as_u64()++ .or_else(|| count.as_str()?.parse::<u64>().ok())++ });++ Ok((name.to_string(), docs))++ })++ .collect()++}+++ pub(crate) async fn run(ctx: &Ctx, _args: &Value) -> Value {+ let (status, banner_text) = match get(ctx, "/").await {- Ok(ok) => ok,- Err((url, e)) => {++ Ok(response) => response,++ Err(error) => {+ return tool_text(+ format!(- "xerj_status: no node is answering at {}.\n{}",- ctx.base_url,- transport_failure_text(&ctx.base_url, &url, &e)++ "xerj_status: could not read node status at {}.\n{error}",++ ctx.base_url+ ),+ true,- );++ )+ }+ };-- // A 401/403 answers the liveness question ("up") but nothing else will- // work, so say that plainly instead of reporting an empty node.+ if status == 401 || status == 403 {+ return tool_text(+ format!(- "xerj_status: the node at {} is UP but refused these credentials \- (root endpoint answered HTTP {status}).\n\- This is an auth problem, not a connection problem: queries will fail \- the same way. Point XERJ_AUTH (or `xerj mcp --auth`) at a key this node \- accepts, or ask its owner for the admin key.",++ "xerj_status: the HTTP service at {} is UP but refused the root request \++ (HTTP {status}).\nThis is an auth problem, not a connection problem. \++ Check XERJ_AUTH (or `xerj mcp --auth`) and the permissions for this endpoint. \++ Other endpoints have not been probed.",++ ctx.base_url++ ),++ true,++ );++ }++ if !(200..300).contains(&status) {++ return tool_text(++ format!(++ "xerj_status: an HTTP service answered at {}, but the root probe failed \++ with HTTP {status}. Node status is unknown.",+ ctx.base_url+ ),+ true,+ );+ }-+ let banner = parse(&banner_text);- let es_version = banner++ let version = banner+ .as_ref()+ .and_then(|b| b.get("version"))+ .and_then(|v| v.get("number"))+ .and_then(Value::as_str);- let cluster_name = banner- .as_ref()- .and_then(|b| b.get("cluster_name"))- .and_then(Value::as_str);-++ let Some(version) = version else {++ return tool_text(++ format!(++ "xerj_status: an HTTP service answered at {}, but the root endpoint did not \++ return a JSON node banner with version.number. Node status is unknown.",++ ctx.base_url++ ),++ true,++ );++ };+ let mut lines = vec![format!(- "xerj_status: the node at {} is UP (root endpoint answered HTTP {status}).",- ctx.base_url++ "xerj_status: the node at {} is UP (root endpoint answered HTTP {status}).", ctx.base_url++ ), format!(++ " version: {} (the Elasticsearch-compatible number this node reports, not XERJ's own version)", display(version)+ )];- match es_version {- Some(v) => lines.push(format!(- " version: {v} (the Elasticsearch-compatible number this node reports, \- not XERJ's own version)"- )),- None => lines.push(- " version: not reported — the root endpoint did not return a JSON banner".into(),- ),- }- if let Some(name) = cluster_name {- lines.push(format!(" cluster name: {name}"));- }-- match get(ctx, "/_cluster/health").await {- Ok((code, text)) => match parse(&text) {- Some(h) if (200..300).contains(&code) => {- let state = h.get("status").and_then(Value::as_str).unwrap_or("unknown");- let nodes = h.get("number_of_nodes").and_then(Value::as_u64);- let shards = h.get("active_shards").and_then(Value::as_u64);- let mut facts = Vec::new();- if let Some(n) = nodes {- facts.push(format!("{n} node{}", if n == 1 { "" } else { "s" }));- }- if let Some(s) = shards {- facts.push(format!("{s} active shards"));- }- lines.push(if facts.is_empty() {- format!(" cluster health: {state}")- } else {- format!(" cluster health: {state} ({})", facts.join(", "))- });- }- _ => lines.push(format!(- " cluster health: unavailable (GET /_cluster/health answered HTTP {code})"- )),- },- Err((url, e)) => lines.push(format!(- " cluster health: unavailable ({})",- transport_failure_text(&ctx.base_url, &url, &e)- )),++ if let Some(name) = banner++ .as_ref()++ .and_then(|b| b.get("cluster_name"))++ .and_then(Value::as_str)++ {++ lines.push(format!(" cluster name: {}", display(name)));+ }-- let indices = match get(ctx, "/_cat/indices?format=json&h=index,docs.count").await {++ let mut incomplete = false;++ let health = match get(ctx, "/_cluster/health").await {+ Ok((code, text)) if (200..300).contains(&code) => parse(&text)- .and_then(|v| v.as_array().cloned())- .map(|rows| {- rows.iter()- .filter_map(|r| {- let name = r.get("index").and_then(Value::as_str)?;- let docs = r.get("docs.count").and_then(|d| {- d.as_str()- .and_then(|s| s.parse::<u64>().ok())- .or_else(|| d.as_u64())- });- Some((name.to_string(), docs))- })- .collect::<Vec<_>>()- }),- _ => None,++ .filter(|h| h.get("status").and_then(Value::as_str).is_some())++ .ok_or_else(|| "GET /_cluster/health did not return a health status".to_string()),++ Ok((code, _)) => Err(format!("GET /_cluster/health answered HTTP {code}")),++ Err(error) => Err(error),+ };-- match &indices {- Some(rows) if rows.is_empty() => {- lines.push(" indices: none".into());- }- Some(rows) => {++ match health {++ Ok(h) => {++ let state = h["status"].as_str().expect("validated health status");++ let mut facts = Vec::new();++ if let Some(n) = h.get("number_of_nodes").and_then(Value::as_u64) {++ facts.push(format!("{n} node{}", if n == 1 { "" } else { "s" }));++ }++ if let Some(n) = h.get("active_shards").and_then(Value::as_u64) {++ facts.push(format!("{n} active shards"));++ }+ lines.push(format!(- " indices: {}{}",- rows.len(),- if rows.len() > MAX_LISTED_INDICES {- format!(" (first {MAX_LISTED_INDICES} listed)")- } else {++ " cluster health: {}{}",++ display(state),++ if facts.is_empty() {+ String::new()++ } else {++ format!(" ({})", facts.join(", "))+ }+ ));- let width = rows- .iter()- .take(MAX_LISTED_INDICES)- .map(|(n, _)| n.len())- .max()- .unwrap_or(0);- for (name, docs) in rows.iter().take(MAX_LISTED_INDICES) {- match docs {- Some(d) => lines.push(format!(" {name:<width$} {d} docs")),- None => lines.push(format!(" {name}")),- }- }+ }- None => lines- .push(" indices: unavailable (GET /_cat/indices did not return a JSON list)".into()),++ Err(error) => {++ incomplete = true;++ lines.push(format!(" cluster health: unavailable ({error})"));++ }+ }-- // The point of the whole tool: an empty node must not read as a broken- // one. `xerj_map` answers 404 "no autoindex catalog" here, and that is- // the node saying it has nothing indexed — not a connectivity failure.- match &indices {- Some(rows) => {- let catalog = rows.iter().find(|(name, _)| name == CATALOG_INDEX);- match catalog {- Some((_, docs)) => lines.push(format!(- " autoindex catalog: PRESENT (`{CATALOG_INDEX}`{}, so xerj_map and \- xerj_search have data to read).",- match docs {- Some(d) => format!(" holds {d} docs"),- None => String::new(),++ let indices = match get(ctx, "/_cat/indices?format=json&h=index,docs.count").await {++ Ok((code, text)) if (200..300).contains(&code) => index_rows(&text),++ Ok((code, _)) => Err(format!("GET /_cat/indices answered HTTP {code}")),++ Err(error) => Err(error),++ };++ match indices {++ Ok(rows) => {++ if rows.is_empty() {++ lines.push(" indices: none returned for these credentials".into());++ } else {++ lines.push(format!(++ " indices: {} visible{}",++ rows.len(),++ if rows.len() > MAX_LISTED_INDICES {++ format!(" (first {MAX_LISTED_INDICES} listed)")++ } else {++ String::new()+ }- )),- None => {- lines.push(- " autoindex catalog: ABSENT — this node is running and has nothing \- indexed yet."- .into(),- );- lines.push(- " xerj_map and xerj_search answer \"no autoindex catalog\" on this \- node. That is an empty node, not a connection failure: index a folder \- with `xerj autoindex <folder>`, then call this tool again."- .into(),- );++ ));++ for (name, docs) in rows.iter().take(MAX_LISTED_INDICES) {++ lines.push(match docs {++ Some(n) => format!(" {} {n} docs", display(name)),++ None => format!(" {} document count unknown", display(name)),++ });+ }+ }++ if let Some((_, docs)) = rows.iter().find(|(name, _)| name == CATALOG_INDEX) {++ lines.push(format!(++ " autoindex catalog: PRESENT in the index list (`{CATALOG_INDEX}`{}).",++ docs.map(|n| format!(" holds {n} docs")).unwrap_or_default()++ ));++ lines.push(" Catalog presence alone does not establish dataset availability; use xerj_map to inspect it.".into());++ } else {++ lines.push(" autoindex catalog: ABSENT from the returned index list.".into());++ lines.push(" This is not a connection failure. Ordinary indices may still be searchable, and managed corpora have a separate namespace.".into());++ lines.push(" Use `xerj autoindex <folder>` to create an autoindex catalog; use xerj_map to inspect catalog datasets.".into());++ }++ }++ Err(error) => {++ incomplete = true;++ lines.push(format!(" indices: unavailable ({error})"));++ lines.push(format!(" autoindex catalog: unknown — whether `{CATALOG_INDEX}` exists was not established."));+ }- None => lines.push(format!(- " autoindex catalog: unknown — the index list could not be read, so whether \- `{CATALOG_INDEX}` exists is not answered here."- )),+ }-- tool_text(lines.join("\n"), false)++ tool_text(lines.join("\n"), incomplete)+ }++ #[cfg(test)]+ mod tests {+ use super::*;+ use serde_json::json;+ use std::collections::HashMap;++ /// Serve a fixed route table on a loopback port. Keys are paths without+ /// their query string, because the real calls carry `?format=json&...`.+ async fn serve(routes: HashMap<&'static str, (u16, String)>) -> String {+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let addr = listener.local_addr().unwrap();+ tokio::spawn(async move {+ loop {+ let Ok((mut sock, _)) = listener.accept().await else {+ return;+ };+ let mut buf = vec![0u8; 8192];+ let n = tokio::io::AsyncReadExt::read(&mut sock, &mut buf)+ .await+ .unwrap_or(0);+ let request = String::from_utf8_lossy(&buf[..n]).to_string();+ let target = request.split_whitespace().nth(1).unwrap_or("/");+ let path = target.split('?').next().unwrap_or("/").to_string();+ let (status, body) = routes+ .get(path.as_str())+ .cloned()+ .unwrap_or((404, "{}".to_string()));+ let reply = format!(+ "HTTP/1.1 {status} X\r\nContent-Length: {}\r\nConnection: close\r\n\+ Content-Type: application/json\r\n\r\n{body}",+ body.len()+ );+ let _ = tokio::io::AsyncWriteExt::write_all(&mut sock, reply.as_bytes()).await;+ }+ });+ format!("http://{addr}")+ }++ fn ctx(base_url: String) -> Ctx {+ Ctx {- client: reqwest::Client::new(),++ client: crate::node_http_builder(&base_url)++ .timeout(std::time::Duration::from_secs(5))++ .build()++ .unwrap(),+ base_url,+ auth: None,+ }+ }++ fn banner() -> String {+ json!({+ "name": "node-1",+ "cluster_name": "xerj",+ "version": { "number": "8.13.0", "distribution": "xerj" },+ "tagline": "You Know, for Search"+ })+ .to_string()+ }++ fn health(state: &str) -> String {+ json!({+ "cluster_name": "xerj",+ "status": state,+ "number_of_nodes": 1,+ "active_shards": 15+ })+ .to_string()+ }++ fn cat(rows: Value) -> String {+ rows.to_string()+ }++ fn text(v: &Value) -> String {+ v["content"][0]["text"]+ .as_str()+ .unwrap_or_default()+ .to_string()+ }++ /// The bug in #2515: `xerj_map` 404s on an unindexed node, and from the+ /// agent's side that reads like the node is not there. The status tool has+ /// to say UP and EMPTY in one answer, and name the 404 as the empty case.+ #[tokio::test]+ async fn an_unindexed_node_reports_up_and_empty_not_unreachable() {+ let mut routes = HashMap::new();+ routes.insert("/", (200, banner()));+ routes.insert("/_cluster/health", (200, health("green")));+ routes.insert("/_cat/indices", (200, cat(json!([]))));+ let base = serve(routes).await;++ let out = run(&ctx(base), &json!({})).await;+ assert_eq!(out["isError"], false, "{}", text(&out));+ let body = text(&out);+ assert!(body.contains("is UP"), "{body}");+ assert!(body.contains("indices: none"), "{body}");+ assert!(body.contains("autoindex catalog: ABSENT"), "{body}");+ assert!(+ body.contains("not a connection failure"),+ "the empty node must not read as a broken one: {body}"+ );+ assert!(body.contains("xerj autoindex <folder>"), "{body}");+ assert!(body.contains("8.13.0"), "{body}");+ assert!(body.contains("green"), "{body}");+ }++ #[tokio::test]+ async fn an_indexed_node_names_the_catalog_and_its_doc_count() {+ let mut routes = HashMap::new();+ routes.insert("/", (200, banner()));+ routes.insert("/_cluster/health", (200, health("yellow")));+ routes.insert(+ "/_cat/indices",+ (+ 200,+ cat(json!([+ { "index": "autoindex-catalog", "docs.count": "12" },+ { "index": "ax", "docs.count": "812" }+ ])),+ ),+ );+ let base = serve(routes).await;++ let out = run(&ctx(base), &json!({})).await;+ assert_eq!(out["isError"], false, "{}", text(&out));+ let body = text(&out);+ assert!(body.contains("indices: 2"), "{body}");+ assert!(body.contains("autoindex-catalog"), "{body}");+ assert!(body.contains("12 docs"), "{body}");+ assert!(body.contains("autoindex catalog: PRESENT"), "{body}");+ assert!(!body.contains("ABSENT"), "{body}");+ assert!(body.contains("yellow"), "{body}");+ }++ /// Nothing listening: the one case that *is* a connectivity failure, and it+ /// has to say so rather than printing an empty successful answer.+ #[tokio::test]+ async fn nothing_listening_is_the_only_unreachable_answer() {+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let addr = listener.local_addr().unwrap();+ drop(listener);++ let out = run(&ctx(format!("http://{addr}")), &json!({})).await;+ assert_eq!(out["isError"], true, "{}", text(&out));+ let body = text(&out);- assert!(body.contains("no node is answering"), "{body}");++ assert!(body.contains("could not read node status"), "{body}");+ }++ /// A 401 means the node is running. Reporting that as "unreachable" would+ /// send an agent looking at ports when the problem is the key.+ #[tokio::test]+ async fn a_refused_key_is_up_with_an_auth_problem_not_down() {+ let mut routes = HashMap::new();+ routes.insert("/", (401, "{\"error\":\"unauthorized\"}".to_string()));+ let base = serve(routes).await;++ let out = run(&ctx(base), &json!({})).await;+ let body = text(&out);+ assert!(body.contains("is UP but refused"), "{body}");+ assert!(+ body.contains("auth problem, not a connection problem"),+ "{body}"+ );+ assert!(body.contains("XERJ_AUTH"), "{body}");+ }++ /// A node that answers the banner but not `_cat/indices` is up. The catalog+ /// question is then unknown, which must not be reported as absent.+ #[tokio::test]+ async fn an_unreadable_index_list_is_unknown_not_absent() {+ let mut routes = HashMap::new();+ routes.insert("/", (200, banner()));+ routes.insert("/_cluster/health", (200, health("green")));+ routes.insert("/_cat/indices", (500, "boom".to_string()));+ let base = serve(routes).await;++ let out = run(&ctx(base), &json!({})).await;- assert_eq!(out["isError"], false, "{}", text(&out));++ assert_eq!(out["isError"], true, "{}", text(&out));+ let body = text(&out);+ assert!(body.contains("autoindex catalog: unknown"), "{body}");+ assert!(!body.contains("ABSENT"), "{body}");+ }++ #[tokio::test]++ async fn ordinary_indices_without_a_catalog_are_not_an_empty_node() {++ let routes = HashMap::from([++ ("/", (200, banner())),++ ("/_cluster/health", (200, health("green"))),++ (++ "/_cat/indices",++ (200, cat(json!([{"index":"logs", "docs.count":"42"}]))),++ ),++ ]);++ let out = run(&ctx(serve(routes).await), &json!({})).await;++ assert_eq!(out["isError"], false);++ let body = text(&out);++ assert!(body.contains("logs"), "{body}");++ assert!(body.contains("42 docs"), "{body}");++ assert!(!body.contains("has nothing indexed"), "{body}");++ assert!(!body.contains("That is an empty node"), "{body}");++ assert!(!body.contains("xerj_search answer"), "{body}");++ }++++ #[tokio::test]++ async fn malformed_index_rows_remain_unknown_instead_of_becoming_empty() {++ let routes = HashMap::from([++ ("/", (200, banner())),++ ("/_cluster/health", (200, health("green"))),++ ("/_cat/indices", (200, cat(json!([{"docs.count":"42"}])))),++ ]);++ let out = run(&ctx(serve(routes).await), &json!({})).await;++ assert_eq!(out["isError"], true, "{}", text(&out));++ let body = text(&out);++ assert!(body.contains("autoindex catalog: unknown"), "{body}");++ assert!(!body.contains("indices: none"), "{body}");++ assert!(!body.contains("ABSENT"), "{body}");++ }++++ #[tokio::test]++ async fn root_failures_and_non_node_banners_are_errors() {++ for (code, reply) in [++ (503, "{}"),++ (404, "{}"),++ (200, "<html>other service</html>"),++ (200, "[]"),++ ] {++ let routes = HashMap::from([("/", (code, reply.to_string()))]);++ let out = run(&ctx(serve(routes).await), &json!({})).await;++ assert_eq!(out["isError"], true, "{}", text(&out));++ assert!(!text(&out).contains("node at"), "{}", text(&out));++ }++ }++++ #[tokio::test]++ async fn endpoint_auth_refusals_are_errors_not_absence() {++ for endpoint in ["/_cluster/health", "/_cat/indices"] {++ for code in [401, 403] {++ let mut routes = HashMap::from([++ ("/", (200, banner())),++ ("/_cluster/health", (200, health("green"))),++ ("/_cat/indices", (200, cat(json!([])))),++ ]);++ routes.insert(endpoint, (code, "{}".to_string()));++ let out = run(&ctx(serve(routes).await), &json!({})).await;++ assert_eq!(out["isError"], true, "{}", text(&out));++ assert!(++ text(&out).contains(&format!("HTTP {code}")),++ "{}",++ text(&out)++ );++ if endpoint == "/_cat/indices" {++ assert!(!text(&out).contains("ABSENT"), "{}", text(&out));++ }++ }++ }++ }++++ #[tokio::test]++ async fn an_empty_catalog_does_not_promise_searchable_datasets() {++ let routes = HashMap::from([++ ("/", (200, banner())),++ ("/_cluster/health", (200, health("green"))),++ (++ "/_cat/indices",++ (200, cat(json!([{"index":CATALOG_INDEX,"docs.count":"0"}]))),++ ),++ ]);++ let out = run(&ctx(serve(routes).await), &json!({})).await;++ assert_eq!(out["isError"], false);++ assert!(text(&out).contains("PRESENT"));++ assert!(text(&out).contains("0 docs"));++ assert!(!text(&out).contains("have data to read"));++ }++++ #[tokio::test]++ async fn an_oversized_index_response_is_unknown_not_empty() {++ let rows = cat(json!([{"index":"x".repeat(256 * 1024),"docs.count":"1"}]));++ let routes = HashMap::from([++ ("/", (200, banner())),++ ("/_cluster/health", (200, health("green"))),++ ("/_cat/indices", (200, rows)),++ ]);++ let out = run(&ctx(serve(routes).await), &json!({})).await;++ assert_eq!(out["isError"], true, "{}", text(&out));++ assert!(text(&out).contains("status-probe limit"));++ assert!(text(&out).contains("catalog: unknown"));++ }++++ async fn raw_root(reply: Vec<u8>) -> String {++ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();++ let address = listener.local_addr().unwrap();++ tokio::spawn(async move {++ let (mut socket, _) = listener.accept().await.unwrap();++ let mut request = Vec::new();++ while request.len() < 8192 && !request.windows(4).any(|w| w == b"\r\n\r\n") {++ let mut chunk = [0; 1024];++ let n = tokio::io::AsyncReadExt::read(&mut socket, &mut chunk)++ .await++ .unwrap();++ if n == 0 {++ return;++ }++ request.extend_from_slice(&chunk[..n]);++ }++ let _ = tokio::io::AsyncWriteExt::write_all(&mut socket, &reply).await;++ });++ format!("http://{address}")++ }++++ #[tokio::test]++ async fn truncated_http_bodies_are_errors_instead_of_empty_text() {++ let base = raw_root(++ b"HTTP/1.1 200 OK\r\nContent-Length: 100\r\nConnection: close\r\n\r\n{}".to_vec(),++ )++ .await;++ let result = get(&ctx(base), "/").await;++ assert!(++ result.is_err(),++ "an incomplete response must not become an empty successful body"++ );++ }++++ #[tokio::test]++ async fn chunked_responses_are_also_bounded_before_parsing() {++ let payload = "x".repeat(256 * 1024 + 1);++ let reply = format!("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\nConnection: close\r\n\r\n{:x}\r\n{payload}\r\n0\r\n\r\n", payload.len());++ let base = raw_root(reply.into_bytes()).await;++ let result = get(&ctx(base), "/").await;++ assert!(++ result.is_err(),++ "a missing Content-Length must not bypass the body limit"++ );++ }+ }diff --cc landing/docs/cli.htmlindex e91fe7473,69a334f0b..2f707df18--- a/landing/docs/cli.html+++ b/landing/docs/cli.html@@@ -530,7 -530,7 +530,7 @@@ Grok CLI | ~/.grok/conf<p>Aider is absent because it has no MCP client support, T3 Code hosts an MCP endpoint rather than consuming one, and Grok Bot has no documented local stdio registration.</p>- <p>Twenty-two tools are served (nineteen in a build with no hub key, which omits the three hub tools). Eleven are thin proxies over an endpoint the engine already exposes, and engine errors come back verbatim: <code class="inline">xerj_search</code>, <code class="inline">xerj_semantic_search</code>, <code class="inline">xerj_vector_search</code>, <code class="inline">xerj_hybrid_search</code>, <code class="inline">xerj_memory_store</code>, <code class="inline">xerj_memory_recall</code>, <code class="inline">xerj_brain_overview</code>, <code class="inline">xerj_brain_ego</code>, <code class="inline">xerj_brain_link</code>, <code class="inline">xerj_brain_unlink</code>, <code class="inline">xerj_plan</code> (<code class="inline">POST /_ask</code>). Two are not plain proxies: <code class="inline">xerj_map</code> returns the autoindex catalog's per-index field map (trimmed to a byte cap), and <code class="inline">xerj_code_search</code> runs the <code class="inline">xerj code</code> pipeline and returns its rendered text. <code class="inline">xerj_field_report</code> files an agent field report (a GitHub PR first, the <code class="inline">/api/field-report</code> endpoint when GitHub is unavailable; it sends by default). <code class="inline">xerj_index_budget</code> counts a folder and estimates its index against the node's disk budget before <code class="inline">xerj autoindex</code> hashes anything (it does not index), <code class="inline">xerj_autoindex_remove</code> drops one autoindexed folder after listing it and deletes nothing unless <code class="inline">confirm</code> is true, and <code class="inline">xerj_corpus_feedback</code> rates a hub corpus. <code class="inline">xerj_def</code> (below) finds local definitions, and <code class="inline">xerj_batch_search</code> runs 1 to 50 IDs or questions in one call, in input order, through the same adapter as <code class="inline">xerj batch-search</code>. The three Corpus Hub tools, <code class="inline">xerj_hub_ask</code>, <code class="inline">xerj_hub_def</code> and <code class="inline">xerj_hub_find</code>, run the <a href="#ask"><code class="inline">xerj ask</code></a> flow, but never download a pack and never fetch the catalogue: run <code class="inline">xerj hub sync</code> and <code class="inline">xerj hub pull <slug></code> in a shell first (details on the <a href="/docs/corpus-hub#mcp">Corpus Hub page</a>). Two caveats the tool descriptions carry themselves: <code class="inline">xerj_semantic_search</code> embeds server-side with the <em>lexical</em> feature-hashing embedder unless the node runs <code class="inline">--embed-mode neural</code>, and the <code class="inline">xerj_brain_*</code> tools work a deterministic link index, not a graph database.</p>- <p>Twenty-three tools are served (twenty in a build with no hub key, which omits the three hub tools). Eleven are thin proxies over an endpoint the engine already exposes, and engine errors come back verbatim: <code class="inline">xerj_search</code>, <code class="inline">xerj_semantic_search</code>, <code class="inline">xerj_vector_search</code>, <code class="inline">xerj_hybrid_search</code>, <code class="inline">xerj_memory_store</code>, <code class="inline">xerj_memory_recall</code>, <code class="inline">xerj_brain_overview</code>, <code class="inline">xerj_brain_ego</code>, <code class="inline">xerj_brain_link</code>, <code class="inline">xerj_brain_unlink</code>, <code class="inline">xerj_plan</code> (<code class="inline">POST /_ask</code>). Two are not plain proxies: <code class="inline">xerj_map</code> returns the autoindex catalog's per-index field map (trimmed to a byte cap), and <code class="inline">xerj_code_search</code> runs the <code class="inline">xerj code</code> pipeline and returns its rendered text. <code class="inline">xerj_field_report</code> files an agent field report (a GitHub PR first, the <code class="inline">/api/field-report</code> endpoint when GitHub is unavailable; it sends by default). <code class="inline">xerj_index_budget</code> counts a folder and estimates its index against the node's disk budget before <code class="inline">xerj autoindex</code> hashes anything (it does not index), <code class="inline">xerj_autoindex_remove</code> drops one autoindexed folder after listing it and deletes nothing unless <code class="inline">confirm</code> is true, <code class="inline">xerj_status</code> answers "is this node up, and is anything indexed" with three GETs so an empty node's 404 is not mistaken for a node that is not there, and <code class="inline">xerj_corpus_feedback</code> rates a hub corpus. <code class="inline">xerj_def</code> (below) finds local definitions, and <code class="inline">xerj_batch_search</code> runs 1 to 50 IDs or questions in one call, in input order, through the same adapter as <code class="inline">xerj batch-search</code>. The three Corpus Hub tools, <code class="inline">xerj_hub_ask</code>, <code class="inline">xerj_hub_def</code> and <code class="inline">xerj_hub_find</code>, run the <a href="#ask"><code class="inline">xerj ask</code></a> flow, but never download a pack and never fetch the catalogue: run <code class="inline">xerj hub sync</code> and <code class="inline">xerj hub pull <slug></code> in a shell first (details on the <a href="/docs/corpus-hub#mcp">Corpus Hub page</a>). Two caveats the tool descriptions carry themselves: <code class="inline">xerj_semantic_search</code> embeds server-side with the <em>lexical</em> feature-hashing embedder unless the node runs <code class="inline">--embed-mode neural</code>, and the <code class="inline">xerj_brain_*</code> tools work a deterministic link index, not a graph database.</p>++ <p>Twenty-three tools are served (twenty in a build with no hub key, which omits the three hub tools). Eleven are thin proxies over an endpoint the engine already exposes, and engine errors come back verbatim: <code class="inline">xerj_search</code>, <code class="inline">xerj_semantic_search</code>, <code class="inline">xerj_vector_search</code>, <code class="inline">xerj_hybrid_search</code>, <code class="inline">xerj_memory_store</code>, <code class="inline">xerj_memory_recall</code>, <code class="inline">xerj_brain_overview</code>, <code class="inline">xerj_brain_ego</code>, <code class="inline">xerj_brain_link</code>, <code class="inline">xerj_brain_unlink</code>, <code class="inline">xerj_plan</code> (<code class="inline">POST /_ask</code>). Two are not plain proxies: <code class="inline">xerj_map</code> returns the autoindex catalog's per-index field map (trimmed to a byte cap), and <code class="inline">xerj_code_search</code> runs the <code class="inline">xerj code</code> pipeline and returns its rendered text. <code class="inline">xerj_field_report</code> files an agent field report (a GitHub PR first, the <code class="inline">/api/field-report</code> endpoint when GitHub is unavailable; it sends by default). <code class="inline">xerj_index_budget</code> counts a folder and estimates its index against the node's disk budget before <code class="inline">xerj autoindex</code> hashes anything (it does not index), <code class="inline">xerj_autoindex_remove</code> drops one autoindexed folder after listing it and deletes nothing unless <code class="inline">confirm</code> is true, <code class="inline">xerj_status</code> probes node reachability, cluster health and visible indices with up to three GETs; a missing autoindex catalog does not imply that other indices are empty, and failed probes remain errors with unknown fields, and <code class="inline">xerj_corpus_feedback</code> rates a hub corpus. <code class="inline">xerj_def</code> (below) finds local definitions, and <code class="inline">xerj_batch_search</code> runs 1 to 50 IDs or questions in one call, in input order, through the same adapter as <code class="inline">xerj batch-search</code>. The three Corpus Hub tools, <code class="inline">xerj_hub_ask</code>, <code class="inline">xerj_hub_def</code> and <code class="inline">xerj_hub_find</code>, run the <a href="#ask"><code class="inline">xerj ask</code></a> flow, but never download a pack and never fetch the catalogue: run <code class="inline">xerj hub sync</code> and <code class="inline">xerj hub pull <slug></code> in a shell first (details on the <a href="/docs/corpus-hub#mcp">Corpus Hub page</a>). Two caveats the tool descriptions carry themselves: <code class="inline">xerj_semantic_search</code> embeds server-side with the <em>lexical</em> feature-hashing embedder unless the node runs <code class="inline">--embed-mode neural</code>, and the <code class="inline">xerj_brain_*</code> tools work a deterministic link index, not a graph database.</p><p><code class="inline">xerj_def</code> finds local definitions with the query, lexical ranking and location deduplication of <code class="inline">xerj def --json</code>. For example, <code class="inline">{"symbol":"Cache","prefix":"project","kind":"struct","lang":"rust","k":5}</code> searches <code class="inline">project-*</code>. Use <code class="inline">index</code> instead of <code class="inline">prefix</code> for an explicit index; non-empty scope arguments cannot be combined, and with neither it searches <code class="inline">ax-*</code>. <code class="inline">symbol</code> also accepts capability phrases. <code class="inline">k</code> is 1–50 (default 5); optional <code class="inline">max_tokens</code> caps the response. Declarations are in <code class="inline">hits.hits[]._source.code</code>, locations in <code class="inline">ax_path</code>/<code class="inline">path</code> and <code class="inline">line</code>/<code class="inline">start_line</code>; older file hits may use <code class="inline">fields._passage</code>. <code class="inline">hits.total</code> counts engine matches before deduplication; no match is a successful empty hit list. As in the CLI, kind/language filters are ignored when no target index maps the field or any maps it as numeric. This queries the configured node; <code class="inline">xerj_hub_def</code> instead searches a named Hub pack.</p>