Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(mcp): bound search responses with a default token budget (#2491)

snmsoodane467047e2fb8
diff --git a/engine/crates/xerj-mcp/src/lib.rs b/engine/crates/xerj-mcp/src/lib.rs
index 638b882a4..617ba0318 100644
--- a/engine/crates/xerj-mcp/src/lib.rs
+++ b/engine/crates/xerj-mcp/src/lib.rs
@@ -93,13 +93,16 @@
//! Diagnostics go to **stderr**; stdout is reserved exclusively for the
//! JSON-RPC stream.
//!
-//! ## Token budgets (#1058)
+//! ## Token budgets (#1058, default since #2491)
//!
//! The six search tools accept `max_tokens` and are held to it: the response
//! text NEVER exceeds the budget. Enforcement lives in [`token_budget`],
//! counted with one named tokenizer (XERJ's `StandardTokenizer`, UAX #29
//! words), with duplicate passages removed before budgeting and
-//! over-budget hits reduced to locators.
+//! over-budget hits reduced to locators. Omitting `max_tokens` applies
+//! [`token_budget::DEFAULT_MAX_TOKENS`] rather than leaving the response
+//! unbounded, so a result too large for the host's tool-output limit is
+//! trimmed with an accounting block instead of being spilled to a file.
mod batch;
mod batch_cli;
@@ -877,9 +880,12 @@ fn max_tokens_arg_schema() -> Value {
listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the \
response's `_token_budget` block accounts for every hit: kept, locator-only, \
deduped, dropped. Errors are never truncated — a refusal you cannot read \
- end-to-end is worse than a long response. Minimum {}.",
+ end-to-end is worse than a long response. Minimum {}; when omitted the budget is \
+ {} (about 45 KB), so a response stays inside a host's tool-output limit unless you \
+ ask for more.",
token_budget::TOKENIZER_NAME,
token_budget::MIN_MAX_TOKENS,
+ token_budget::DEFAULT_MAX_TOKENS,
)
})
}
diff --git a/engine/crates/xerj-mcp/src/token_budget.rs b/engine/crates/xerj-mcp/src/token_budget.rs
index 3ebd4e822..192f3f30d 100644
--- a/engine/crates/xerj-mcp/src/token_budget.rs
+++ b/engine/crates/xerj-mcp/src/token_budget.rs
@@ -60,6 +60,18 @@ pub const TOKENIZER_ID: &str = "standard-uax29-words";
/// budget it cannot honor.
pub const MIN_MAX_TOKENS: usize = 32;
+/// Budget applied when the caller passes no `max_tokens` (#2491).
+///
+/// Without a default the response was unbounded, and an agent whose host caps
+/// tool output had the result spilled to a temp file instead of shown inline —
+/// it then spent extra steps reading the file back, which is exactly the cost
+/// the budget exists to avoid. ~8000 word tokens is roughly 45 KB of text:
+/// far more than any useful search response (a typical 10-hit page is a few
+/// hundred), and comfortably inside the tool-output limits MCP hosts impose.
+/// A caller that wants more says so — any value above [`MIN_MAX_TOKENS`] is
+/// accepted — so this bounds the default, not the ceiling.
+pub const DEFAULT_MAX_TOKENS: usize = 8_000;
+
/// Tokens reserved for the `_token_budget` accounting block before any hit is
/// admitted. The block's fixed fields (keys + counts) measure ~14 word tokens;
/// locator stubs inside it are charged against the content budget separately,
@@ -81,9 +93,15 @@ const TRUNCATION_MARKER: &str = "\n[truncated at the max_tokens budget]";
/// floor is an error, never a silent drop — the `opt_typed` policy: an agent
/// that asked for a budget and silently got none would trust a response whose
/// size it cannot predict.
+///
+/// Absent means [`DEFAULT_MAX_TOKENS`], not unbounded (#2491): a response that
+/// overflows the host's tool-output limit is one the agent cannot read inline,
+/// so "no budget asked for" must not degrade into "no budget applied". The
+/// return is still an `Option` because a caller may want to skip budgeting
+/// entirely for a response it already knows is short.
pub fn opt_max_tokens(args: &Value) -> Result<Option<usize>, String> {
match args.get("max_tokens") {
- None | Some(Value::Null) => Ok(None),
+ None | Some(Value::Null) => Ok(Some(DEFAULT_MAX_TOKENS)),
Some(v) => match v.as_u64() {
Some(n) if n as usize >= MIN_MAX_TOKENS => Ok(Some(n as usize)),
_ => Err(format!(
@@ -623,8 +641,15 @@ mod tests {
#[test]
fn opt_max_tokens_validates() {
- assert_eq!(opt_max_tokens(&json!({})), Ok(None));
- assert_eq!(opt_max_tokens(&json!({ "max_tokens": null })), Ok(None));
+ assert_eq!(
+ opt_max_tokens(&json!({})),
+ Ok(Some(DEFAULT_MAX_TOKENS)),
+ "an omitted budget must fall back to the default, not to unbounded"
+ );
+ assert_eq!(
+ opt_max_tokens(&json!({ "max_tokens": null })),
+ Ok(Some(DEFAULT_MAX_TOKENS))
+ );
assert_eq!(opt_max_tokens(&json!({ "max_tokens": 500 })), Ok(Some(500)));
for bad in [
json!({ "max_tokens": 0 }),
@@ -639,6 +664,45 @@ mod tests {
}
}
+ /// #2491: a search that passes no `max_tokens` used to return whatever the
+ /// engine produced, and a response past the host's tool-output limit was
+ /// spilled to a temp file instead of shown to the agent. Absent now means
+ /// `DEFAULT_MAX_TOKENS`, so the same call is bounded and still readable.
+ #[test]
+ fn an_unbudgeted_search_response_is_bounded_by_default() {
+ // Wide enough that an unbounded response overruns any plausible host
+ // limit: 400 hits carrying 400 lines of body each.
+ let resp = search_response(
+ (1..=400)
+ .map(|i| code_hit(&format!("repo/src/module_{i}.rs"), 10 * i, 400))
+ .collect(),
+ );
+ let unbounded = count_tokens(&resp);
+ assert!(
+ unbounded > DEFAULT_MAX_TOKENS,
+ "the fixture must be big enough to need the default cap: {unbounded}"
+ );
+
+ let budget = opt_max_tokens(&json!({}))
+ .expect("absent max_tokens is not an error")
+ .expect("absent max_tokens must apply the default budget");
+ assert_eq!(budget, DEFAULT_MAX_TOKENS);
+
+ let out = budget_json_response(&resp, budget);
+ let used = count_tokens(&out);
+ assert!(
+ used <= DEFAULT_MAX_TOKENS,
+ "default budget {DEFAULT_MAX_TOKENS} exceeded: {used}\n---\n{out}\n---"
+ );
+ // Bounded by dropping content, not by dropping the answer: the
+ // accounting block has to say what went, or the agent cannot tell a
+ // short corpus from a truncated one.
+ assert!(
+ out.contains("_token_budget"),
+ "the response must account for what the budget cut:\n{out}"
+ );
+ }
+
// ── JSON surface: per-tool budget enforcement ──────────────────────────
/// xerj_search shape: definition-first hits over an autoindex corpus,
diff --git a/landing/docs/agents/schemas/mcp-tools.json b/landing/docs/agents/schemas/mcp-tools.json
index 8d3ad02c5..3bca056df 100644
--- a/landing/docs/agents/schemas/mcp-tools.json
+++ b/landing/docs/agents/schemas/mcp-tools.json
@@ -37,7 +37,7 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32."
+ "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
}
},
"required": [
@@ -189,7 +189,7 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32."
+ "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
}
}
}
@@ -222,7 +222,7 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32."
+ "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
}
},
"required": [
@@ -267,7 +267,7 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32."
+ "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
}
},
"required": [
@@ -333,7 +333,7 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32."
+ "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
}
},
"required": [
@@ -785,7 +785,7 @@
"max_tokens": {
"type": "integer",
"minimum": 32,
- "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32."
+ "description": "Token budget for this response: the returned text NEVER exceeds it. Tokens are counted with XERJ's StandardTokenizer (UAX #29 word boundaries) — the same word split the engine's own analyzer uses for BM25. Honest scope: that under-counts model-context subwords (snake_case is ONE token here), so treat it as a deterministic proxy, not a BPE count. The budget covers the whole response text (JSON keys and syntax included). JSON responses dedupe overlapping file:line passages first (higher-ranked wins); rendered code dedupes only identical locators with identical displayed passages. Hits that no longer fit are listed as LOCATORS ONLY (file:line, index/id) rather than dropped, and the response's `_token_budget` block accounts for every hit: kept, locator-only, deduped, dropped. Errors are never truncated — a refusal you cannot read end-to-end is worse than a long response. Minimum 32; when omitted the budget is 8000 (about 45 KB), so a response stays inside a host's tool-output limit unless you ask for more."
}
},
"required": [