Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Validate MCP contracts and generate the 23-tool schema from the binary

xerj-orga89b55bb0bd3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c2cf82b34..dcc256682 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -26,6 +26,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- MCP `xerj_search` advertises plain text in `query` and structured clauses
+ in `query_dsl`. Legacy object-valued `query` and omitted-query browsing
+ remain supported; conflicting fields fail before HTTP requests (#2549).
+- Search payloads and batch retrieval default to 8,000 UAX #29 words when
+ `max_tokens` is absent or null (#2491). This is not a byte or model-token
+ cap. Explicit larger budgets remain available; errors and mandatory
+ search staleness hints stay outside the payload budget.
- GitHub field-report submission uses the supported explicit-repository
`gh repo fork --clone=false` invocation and adds the fork remote itself
(#2596). The rollback test now verifies that the report was committed
diff --git a/README.md b/README.md
index cae920eff..09e7e3383 100644
--- a/README.md
+++ b/README.md
@@ -521,7 +521,8 @@ browsing. Prefer an explicit `query_dsl` clause when browsing is intended.
Search tools and `xerj batch-search` default to an 8,000 UAX #29 word budget
when `max_tokens` is omitted or null. A larger integer requests more evidence;
accounting explains any trimming. This is not a byte cap or a model/BPE token
-limit. Errors remain untruncated.
+limit. Errors and mandatory `xerj_search` staleness hints are outside this
+budget and remain visible.
`xerj_def` finds local definitions with the same query, ranking and location deduplication
as `xerj def --json`. For example, `{"symbol":"Cache","prefix":"project","kind":"struct","lang":"rust","k":5}`
diff --git a/engine/crates/xerj-mcp/src/lib.rs b/engine/crates/xerj-mcp/src/lib.rs
index 5694c88cc..08902c2a6 100644
--- a/engine/crates/xerj-mcp/src/lib.rs
+++ b/engine/crates/xerj-mcp/src/lib.rs
@@ -95,8 +95,8 @@
//!
//! ## Token budgets (#1058, default since #2491)
//!
-//! The six search tools accept `max_tokens` and are held to it: the response
-//! text NEVER exceeds the budget. Enforcement lives in [`token_budget`],
+//! The six search tools accept `max_tokens` for successful search payloads.
+//! Errors and the mandatory `xerj_search` staleness hint are outside the budget. Enforcement lives in [`token_budget`],
//! counted with one named tokenizer (XERJ's `StandardTokenizer`, UAX #29
//! words), with duplicate passages removed before budgeting and
//! over-budget hits reduced to locators. Omitting `max_tokens` applies
@@ -884,7 +884,8 @@ fn max_tokens_arg_schema() -> Value {
"minimum": token_budget::MIN_MAX_TOKENS,
"default": token_budget::DEFAULT_MAX_TOKENS,
"description": format!(
- "Token budget for this response: the returned text NEVER exceeds it. Tokens are \
+ "Word budget for the successful search payload. Errors and mandatory \
+ xerj_search staleness hints are outside this budget. Tokens are \
counted with {} — the same word split the engine's own analyzer uses for BM25. \
Honest scope: that under-counts model-context subwords (snake_case is ONE token \
here), so treat it as a deterministic proxy, not a BPE count. The budget covers \
@@ -949,8 +950,8 @@ pub fn tool_specs_with(include_hub: bool) -> Value {
(capped). Cite `path:match_line`. Both are omitted when no term is \
located or the passage start line is unknown. `start_line` remains \
the passage start. \
- Pass `max_tokens` to cap the response at a token budget (never exceeded; \
- see that argument for the tokenizer and the locator-only rule). \
+ Pass `max_tokens` to override the default word budget (see that argument \
+ for the tokenizer, exceptions and locator-only rule). \
If this prefix was autoindexed from a folder on this machine and a file \
there is newer than the last run, or an indexed file has been removed, \
the response adds `_xerj.hints` code `index_stale` with the last index \
@@ -1011,8 +1012,8 @@ pub fn tool_specs_with(include_hub: bool) -> Value {
embedded SERVER-SIDE by XERJ's built-in lexical embedder (no external \
API key), then matched by vector similarity. Proxies POST /{index}/_search \
with {\"query\":{\"semantic\":{...}}}. \
- Pass `max_tokens` to cap the response at a token budget (never exceeded; \
- see that argument for the tokenizer and the locator-only rule).",
+ Pass `max_tokens` to override the default word budget (see that argument \
+ for the tokenizer, exceptions and locator-only rule).",
"inputSchema": {
"type": "object",
"properties": {
@@ -1036,8 +1037,8 @@ pub fn tool_specs_with(include_hub: bool) -> Value {
800). Filtered kNN, non-cosine metrics, SQ8 fields, and small indexes \
run an exact brute-force scan. \
Proxies POST /{index}/_search with a top-level {\"knn\":{...}}. \
- Pass `max_tokens` to cap the response at a token budget (never exceeded; \
- see that argument for the tokenizer and the locator-only rule).",
+ Pass `max_tokens` to override the default word budget (see that argument \
+ for the tokenizer, exceptions and locator-only rule).",
"inputSchema": {
"type": "object",
"properties": {
@@ -1069,8 +1070,8 @@ pub fn tool_specs_with(include_hub: bool) -> Value {
Optional `judge` re-judges the fused hits LOCALLY (zero tokens, \
nothing leaves the node); `judge.query` is REQUIRED for the same \
reason. \
- Pass `max_tokens` to cap the response at a token budget (never exceeded; \
- see that argument for the tokenizer and the locator-only rule).",
+ Pass `max_tokens` to override the default word budget (see that argument \
+ for the tokenizer, exceptions and locator-only rule).",
"inputSchema": {
"type": "object",
"properties": {
@@ -3505,9 +3506,9 @@ fn search_match_query(name: &str, args: &Value) -> Option<String> {
if name != "xerj_search" {
return None;
}
- match args.get("query") {
+ match search_query(args).ok().flatten() {
Some(Value::String(q)) if !q.trim().is_empty() => Some(q.clone()),
- Some(v @ (Value::Object(_) | Value::Array(_))) => Some(v.to_string()),
+ Some(v @ Value::Object(_)) => Some(v.to_string()),
_ => None,
}
}
@@ -4218,6 +4219,11 @@ mod tests {
let (_, _, new) = built(build_search(
&json!({"index":"docs","query_dsl":clause,"size":2,"from":1}),
));
+ assert_eq!(
+ search_match_query("xerj_search", &json!({"query":clause})),
+ search_match_query("xerj_search", &json!({"query_dsl":clause})),
+ "both DSL forms must retain passage match-line annotation"
+ );
assert_eq!(old["query"], clause);
assert_eq!(
new, old,
@@ -5091,13 +5097,18 @@ mod tests {
if search_tools.contains(&name) {
assert!(prop.is_object(), "{name} must advertise `max_tokens`");
let d = prop["description"].as_str().expect("described");
- // One NAMED tokenizer, stated in the description, plus the
- // never-exceed promise and the floor.
+ // Name the tokenizer, default and floor, and the existing
+ // staleness-warning exception instead of promising a total cap.
assert!(
d.contains(token_budget::TOKENIZER_NAME),
"{name}: the tokenizer must be named: {d}"
);
- assert!(d.contains("NEVER exceeds"), "{name}: {d}");
+ assert!(d.contains("successful search payload"), "{name}: {d}");
+ assert!(
+ d.contains("staleness hints are outside this budget"),
+ "{name}: {d}"
+ );
+ assert_eq!(prop["default"], token_budget::DEFAULT_MAX_TOKENS);
assert!(d.contains("LOCATORS ONLY"), "{name}: {d}");
assert!(
d.contains(&token_budget::MIN_MAX_TOKENS.to_string()),
@@ -6245,6 +6256,14 @@ mod tests {
assert_eq!(parsed["_xerj"]["hints"][0]["path"], "docs/page.md");
assert!(parsed["hits"]["hits"].as_array().unwrap().is_empty());
+ let tight = token_budget::budget_json_response(r#"{"hits":{"hits":[]}}"#, 32);
+ assert!(token_budget::count_tokens(&tight) <= 32);
+ let warned = apply_staleness(tool_text(tight, false), Some(hint.clone()));
+ assert!(
+ token_budget::count_tokens(warned["content"][0]["text"].as_str().unwrap()) > 32,
+ "the mandatory warning deliberately remains outside the payload budget"
+ );
+
let err = tool_text("XERJ returned HTTP 400: no", true);
let kept = apply_staleness(err.clone(), Some(hint));
assert_eq!(kept, err);
diff --git a/engine/crates/xerj-mcp/src/token_budget.rs b/engine/crates/xerj-mcp/src/token_budget.rs
index bcdfbbde7..7bedfe946 100644
--- a/engine/crates/xerj-mcp/src/token_budget.rs
+++ b/engine/crates/xerj-mcp/src/token_budget.rs
@@ -1,12 +1,12 @@
//! Token budgets for MCP search-tool responses (#1058).
//!
-//! The five search tools (`xerj_search`, `xerj_hybrid_search`,
-//! `xerj_semantic_search`, `xerj_vector_search`, `xerj_code_search`) are thin
-//! proxies: until now they returned whatever the engine sent, so an agent
-//! that asked for 50 hits got all 50 verbatim, context cost unknown and
-//! unbounded. `max_tokens` generalises the char-level cap `xerj_code_search`
-//! already had (its `full` argument — max chars *per passage*) to a token
-//! budget over the **whole response**.
+//! Six individual search tools share this output budget: `xerj_def`,
+//! `xerj_search`, `xerj_hybrid_search`, `xerj_semantic_search`,
+//! `xerj_vector_search` and `xerj_code_search`. `max_tokens` covers their
+//! successful search payload, including keys and syntax. Errors and the
+//! mandatory `xerj_search` staleness hint remain outside that budget.
+//! The batch adapter shares the tokenizer and default but preserves every
+//! item outcome using its own whole-response budgeting algorithm.
//!
//! ## The one tokenizer
//!
@@ -47,7 +47,7 @@ use serde_json::{json, Value};
use unicode_segmentation::UnicodeSegmentation;
/// How the budget counts tokens, named in tool descriptions so an agent knows
-/// what a "token" is before spending one. Kept as a constant so the five
+/// what a "token" is before spending one. Kept as a constant so the shared
/// descriptions cannot drift.
pub const TOKENIZER_NAME: &str = "XERJ's StandardTokenizer (UAX #29 word boundaries)";
diff --git a/landing/docs/agents/schemas/mcp-tools.json b/landing/docs/agents/schemas/mcp-tools.json
index 0af59d9b1..321ab577e 100644
--- a/landing/docs/agents/schemas/mcp-tools.json
+++ b/landing/docs/agents/schemas/mcp-tools.json
@@ -37,7 +37,8 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
+ "default": 8000,
+ "description": "Word budget for the successful search payload. Errors and mandatory xerj_search staleness hints are outside this budget. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 UAX #29 words. This is not a byte cap or a guarantee against host output limits. Pass a larger integer to request more."
}
},
"required": [
@@ -47,7 +48,7 @@
},
{
"name": "xerj_batch_search",
- "description": "Search 1..50 IDs or questions in one call, in input order. Shares the CLI xerj batch-search adapter. Each result has its zero-based position, optional caller key, and status hits, abstain (only a complete empty search), or error. Hits carry citations: index/id plus file/line or record locator when available. This retrieves evidence, not generated answers or calibrated confidence. Use identifier plus an explicitly keyword-mapped field for exact IDs; query takes plain text or an ES query clause. One bad item never discards its siblings; identical inputs remain separate. Uses one _msearch request, plus at most one mapping read. Optional max_tokens caps the entire successful JSON response using StandardTokenizer UAX #29 words, not BPE. Budgeting removes raw hit bodies, then extra hits, retaining each item's status and at least one citation per matched item. If that minimum cannot fit, returns an actionable tool error, never truncated JSON or a false abstention. Errors are not truncated.",
+ "description": "Search 1..50 IDs or questions in one call, in input order. Shares the CLI xerj batch-search adapter. Each result has its zero-based position, optional caller key, and status hits, abstain (only a complete empty search), or error. Hits carry citations: index/id plus file/line or record locator when available. This retrieves evidence, not generated answers or calibrated confidence. Use identifier plus an explicitly keyword-mapped field for exact IDs; query takes plain text or an ES query clause. One bad item never discards its siblings; identical inputs remain separate. Uses one _msearch request, plus at most one mapping read. Omitted or null max_tokens defaults to 8000 UAX #29 words and caps the entire successful JSON response using StandardTokenizer UAX #29 words, not BPE. Budgeting removes raw hit bodies, then extra hits, retaining each item's status and at least one citation per matched item. If that minimum cannot fit, returns an actionable tool error, never truncated JSON or a false abstention. Errors are not truncated.",
"inputSchema": {
"type": "object",
"additionalProperties": false,
@@ -129,7 +130,8 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Whole-response budget counted with XERJ's StandardTokenizer (UAX #29 word boundaries) (not BPE); minimum 32. A successful response NEVER exceeds it. Preserves ordered item outcomes and at least one citation per matched item, removing raw bodies first. Refuses if that minimum cannot fit; split the batch or increase the budget. Errors are outside this budget."
+ "default": 8000,
+ "description": "Whole-response budget counted with XERJ's StandardTokenizer (UAX #29 word boundaries) (not BPE or a byte cap); minimum 32. Omitted or null defaults to 8000 words; pass a larger integer for more. A successful response NEVER exceeds it. Preserves ordered item outcomes and at least one citation per matched item, removing raw bodies first. Refuses if that minimum cannot fit; split the batch or increase the budget. Errors are outside this budget."
}
},
"required": [
@@ -139,7 +141,7 @@
},
{
"name": "xerj_search",
- "description": "Full-text / keyword / structured search over a XERJ index. Proxies POST /{index}/_search. Pass `prefix` (a `xerj search --prefix` value, resolved by that command) or `index` (a name or pattern) — not both; with neither, searches `ax-*`. Pass `query` as a PLAIN STRING for the definition-first code search (ranks the file that defines a symbol above files that merely mention it — the right default for code), or as an ES query-DSL object (e.g. {\"term\":{\"status\":\"open\"}}, or a bool clause) for structured queries. Omit `query` for match_all. Optional `rerank` adds a second stage that re-judges the top hits with an external relevance model — read that argument's description before using it: it needs a provider the operator configured, and it sends document text off the machine. Optional `judge` re-judges the returned hits LOCALLY (zero tokens, nothing leaves the node) and adds a `_p_relevant` probability per hit — see that argument's description for its honest ceiling. Each hit carries `match_line` when a query term sits in the passage: the absolute 1-based source line of the best-scoring match (a tie takes the earliest line), and `match_lines` for every matched line (capped). Cite `path:match_line`. Both are omitted when no term is located or the passage start line is unknown. `start_line` remains the passage start. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule). If this prefix was autoindexed from a folder on this machine and a file there is newer than the last run, or an indexed file has been removed, the response adds `_xerj.hints` code `index_stale` with the last index time and a reindex command. Hits are returned with that hint. The check stats at most 4096 entries, stops after 100ms, and skips hidden and ignored paths.",
+ "description": "Full-text / keyword / structured search over a XERJ index. Proxies POST /{index}/_search. Pass `prefix` (a `xerj search --prefix` value, resolved by that command) or `index` (a name or pattern) — not both; with neither, searches `ax-*`. Pass `query` as a PLAIN STRING for the definition-first code search (ranks the file that defines a symbol above files that merely mention it — the right default for code), or as an ES query-DSL object (e.g. {\"term\":{\"status\":\"open\"}}, or a bool clause) for structured queries. Omit `query` for match_all. Optional `rerank` adds a second stage that re-judges the top hits with an external relevance model — read that argument's description before using it: it needs a provider the operator configured, and it sends document text off the machine. Optional `judge` re-judges the returned hits LOCALLY (zero tokens, nothing leaves the node) and adds a `_p_relevant` probability per hit — see that argument's description for its honest ceiling. Each hit carries `match_line` when a query term sits in the passage: the absolute 1-based source line of the best-scoring match (a tie takes the earliest line), and `match_lines` for every matched line (capped). Cite `path:match_line`. Both are omitted when no term is located or the passage start line is unknown. `start_line` remains the passage start. Pass `max_tokens` to override the default word budget (see that argument for the tokenizer, exceptions and locator-only rule). If this prefix was autoindexed from a folder on this machine and a file there is newer than the last run, or an indexed file has been removed, the response adds `_xerj.hints` code `index_stale` with the last index time and a reindex command. Hits are returned with that hint. The check stats at most 4096 entries, stops after 100ms, and skips hidden and ignored paths.",
"inputSchema": {
"type": "object",
"properties": {
@@ -153,7 +155,7 @@
},
"query": {
"type": "string",
- "description": "What to search for, in plain words — the definition-first code search `xerj search` runs. REQUIRED unless you pass `query_dsl` instead."
+ "description": "What to search for, in plain words — the definition-first code search `xerj search` runs. Use `query_dsl` for a structured clause. Omit both for legacy match_all browsing; explicit query_dsl: {match_all: {}} is clearer."
},
"query_dsl": {
"type": "object",
@@ -190,14 +192,15 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
+ "default": 8000,
+ "description": "Word budget for the successful search payload. Errors and mandatory xerj_search staleness hints are outside this budget. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 UAX #29 words. This is not a byte cap or a guarantee against host output limits. Pass a larger integer to request more."
}
}
}
},
{
"name": "xerj_semantic_search",
- "description": "Meaning-based search over a `semantic_text` field. The query text is embedded SERVER-SIDE by XERJ's built-in lexical embedder (no external API key), then matched by vector similarity. Proxies POST /{index}/_search with {\"query\":{\"semantic\":{...}}}. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule).",
+ "description": "Meaning-based search over a `semantic_text` field. The query text is embedded SERVER-SIDE by XERJ's built-in lexical embedder (no external API key), then matched by vector similarity. Proxies POST /{index}/_search with {\"query\":{\"semantic\":{...}}}. Pass `max_tokens` to override the default word budget (see that argument for the tokenizer, exceptions and locator-only rule).",
"inputSchema": {
"type": "object",
"properties": {
@@ -223,7 +226,8 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
+ "default": 8000,
+ "description": "Word budget for the successful search payload. Errors and mandatory xerj_search staleness hints are outside this budget. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 UAX #29 words. This is not a byte cap or a guarantee against host output limits. Pass a larger integer to request more."
}
},
"required": [
@@ -235,7 +239,7 @@
},
{
"name": "xerj_vector_search",
- "description": "K-nearest-neighbour search over a `dense_vector` field, given a caller-supplied query vector. NOTE: unfiltered kNN is HNSW-served (approximate) with exact rescoring — measured recall@10 1.00 on the official bench query; num_candidates sets the beam width (floored at 800). Filtered kNN, non-cosine metrics, SQ8 fields, and small indexes run an exact brute-force scan. Proxies POST /{index}/_search with a top-level {\"knn\":{...}}. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule).",
+ "description": "K-nearest-neighbour search over a `dense_vector` field, given a caller-supplied query vector. NOTE: unfiltered kNN is HNSW-served (approximate) with exact rescoring — measured recall@10 1.00 on the official bench query; num_candidates sets the beam width (floored at 800). Filtered kNN, non-cosine metrics, SQ8 fields, and small indexes run an exact brute-force scan. Proxies POST /{index}/_search with a top-level {\"knn\":{...}}. Pass `max_tokens` to override the default word budget (see that argument for the tokenizer, exceptions and locator-only rule).",
"inputSchema": {
"type": "object",
"properties": {
@@ -268,7 +272,8 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
+ "default": 8000,
+ "description": "Word budget for the successful search payload. Errors and mandatory xerj_search staleness hints are outside this budget. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 UAX #29 words. This is not a byte cap or a guarantee against host output limits. Pass a larger integer to request more."
}
},
"required": [
@@ -280,7 +285,7 @@
},
{
"name": "xerj_hybrid_search",
- "description": "Hybrid search: fuse several sub-queries (e.g. a lexical `match` plus a vector `knn`) into one ranked list. Fusion is `rrf` (reciprocal-rank) or `linear` (weighted). Proxies POST /{index}/_search with {\"query\":{\"hybrid\":{\"queries\":[...],\"fusion\":...}}}. Optional `rerank` re-judges the fused top hits with an external relevance model; here `rerank.query` is REQUIRED, because a hybrid search has several sub-queries and no single question to read. Optional `judge` re-judges the fused hits LOCALLY (zero tokens, nothing leaves the node); `judge.query` is REQUIRED for the same reason. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule).",
+ "description": "Hybrid search: fuse several sub-queries (e.g. a lexical `match` plus a vector `knn`) into one ranked list. Fusion is `rrf` (reciprocal-rank) or `linear` (weighted). Proxies POST /{index}/_search with {\"query\":{\"hybrid\":{\"queries\":[...],\"fusion\":...}}}. Optional `rerank` re-judges the fused top hits with an external relevance model; here `rerank.query` is REQUIRED, because a hybrid search has several sub-queries and no single question to read. Optional `judge` re-judges the fused hits LOCALLY (zero tokens, nothing leaves the node); `judge.query` is REQUIRED for the same reason. Pass `max_tokens` to override the default word budget (see that argument for the tokenizer, exceptions and locator-only rule).",
"inputSchema": {
"type": "object",
"properties": {
@@ -334,7 +339,8 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
+ "default": 8000,
+ "description": "Word budget for the successful search payload. Errors and mandatory xerj_search staleness hints are outside this budget. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 UAX #29 words. This is not a byte cap or a guarantee against host output limits. Pass a larger integer to request more."
}
},
"required": [
@@ -734,7 +740,7 @@
},
{
"name": "xerj_status",
- "description": "Is this node up, and what is on it? Answers the \"is it running\" question in one call, without a search and without reading the autoindex catalog: three GETs, a few lines of text. Reports the Elasticsearch-compatible version the node claims, cluster health, the index count and names, and whether `autoindex-catalog` exists. Call this before concluding that a 404 from xerj_map or xerj_search means the node is down: on a running but unindexed node that 404 is the empty case, and this tool says so in words. Also distinguishes a refused key (the node is up, the credentials are not accepted) from nothing listening at all.",
+ "description": "Is this node up, and what is on it? Answers the \"is it running\" question in one call, without a search and without reading the autoindex catalog: up to three GETs with bounded response bodies. Reports the Elasticsearch-compatible version the node claims, cluster health, visible index count and up to 20 names, and whether `autoindex-catalog` appears in that list. A missing catalog does not mean other indices are empty. Call this before concluding that a 404 from xerj_map or xerj_search means the node is down. Failed, refused, malformed or oversized probes remain errors with unknown fields, rather than successful empty results.",
"inputSchema": {
"type": "object",
"properties": {}
@@ -794,7 +800,8 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
+ "default": 8000,
+ "description": "Word budget for the successful search payload. Errors and mandatory xerj_search staleness hints are outside this budget. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 UAX #29 words. This is not a byte cap or a guarantee against host output limits. Pass a larger integer to request more."
}
},
"required": [
diff --git a/scripts/mcp-search-contract-smoke.py b/scripts/mcp-search-contract-smoke.py
index 911d818e2..2b7ab0518 100644
--- a/scripts/mcp-search-contract-smoke.py
+++ b/scripts/mcp-search-contract-smoke.py
@@ -16,7 +16,7 @@ def main():
binary = str(Path(sys.argv[1]).resolve(strict=True))
requests = []
hits = [{"_index": "docs", "_id": str(i), "_source": {
- "body": "search evidence " * 500, "path": f"source/{i}.txt",
+ "body": "search evidence " * 500, "path": f"source/{i}.txt", "start_line": 1,
}} for i in range(20)]
class Handler(BaseHTTPRequestHandler):
@@ -66,11 +66,17 @@ def main():
return json.loads(result["content"][0]["text"])
try:
- clause = {"bool": {"filter": {"term": {"kind": "guide"}}}}
+ clause = {"bool": {"must": {"match": {"body": "search"}}}}
+ prior = None
for field in ("query_dsl", "query"):
response = successful({"index": "docs", field: clause, "max_tokens": 100000})
assert requests == [("POST", "/docs/_search", {"query": clause})], requests
- assert response["hits"]["hits"] == hits, "nonbinding budget must retain full evidence"
+ returned = response["hits"]["hits"]
+ assert [h["_source"] for h in returned] == [h["_source"] for h in hits], "nonbinding budget must retain full evidence"
+ assert returned[0]["match_line"] == 1, "structured queries must retain passage locations"
+ if prior is not None:
+ assert returned == prior, "legacy and named DSL must return identical evidence"
+ prior = returned
print("PASS new query_dsl and legacy object query send identical structured requests")
for args, path in [({"index": "docs"}, "/docs/_search"), ({}, "/ax-*/_search")]: