AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Make the search/error surface discoverable for agents driving XERJ
buger4a100e9aab38
diff --git a/engine/crates/xerj-api/src/auth.rs b/engine/crates/xerj-api/src/auth.rsindex cf77fb526..046c42b1a 100644--- a/engine/crates/xerj-api/src/auth.rs+++ b/engine/crates/xerj-api/src/auth.rs@@ -420,6 +420,7 @@ fn unauthorized_response() -> Response {request_id: None,},status: 401,+ xerj_hint: None,};(StatusCode::UNAUTHORIZED, Json(body)).into_response()diff --git a/engine/crates/xerj-api/src/authz.rs b/engine/crates/xerj-api/src/authz.rsindex af6ce235d..65bb70048 100644--- a/engine/crates/xerj-api/src/authz.rs+++ b/engine/crates/xerj-api/src/authz.rs@@ -284,6 +284,7 @@ fn es_error(status: StatusCode, reason: String) -> Response {request_id: None,},status: status.as_u16(),+ xerj_hint: None,};(status, Json(body)).into_response()}diff --git a/engine/crates/xerj-api/src/error.rs b/engine/crates/xerj-api/src/error.rsindex 00f8d8dd3..7955af186 100644--- a/engine/crates/xerj-api/src/error.rs+++ b/engine/crates/xerj-api/src/error.rs@@ -71,6 +71,16 @@ pub struct EsErrorBody {pub struct EsErrorResponse {pub error: EsErrorBody,pub status: u16,+ /// Additive, non-ES field carrying an actionable recovery hint (e.g. for+ /// `index_not_found_exception`: nearby index names, the `ax-*` wildcard+ /// an autoindex-managed cluster searches with). Same `_xerj` convention+ /// as the hint on a successful `_search` response — see+ /// `es_compat::search_response_hint`. Absent whenever no handler had+ /// anything more specific to offer than the `reason` string already+ /// says, so this never becomes noise on errors that are already+ /// self-explanatory.+ #[serde(rename = "_xerj", skip_serializing_if = "Option::is_none")]+ pub xerj_hint: Option<Value>,}// ─────────────────────────────────────────────────────────────────────────────@@ -85,6 +95,7 @@ pub struct EsErrorResponse {pub struct ApiError {pub inner: XerjError,pub request_id: Option<String>,+ pub hint: Option<Value>,}impl ApiError {@@ -92,6 +103,7 @@ impl ApiError {Self {inner,request_id: None,+ hint: None,}}@@ -99,6 +111,13 @@ impl ApiError {self.request_id = Some(id.into());self}++ /// Attach an actionable `_xerj` recovery hint (see [`EsErrorResponse`]).+ /// Additive only — never changes `error.type` / `error.reason` / status.+ pub fn with_hint(mut self, hint: Value) -> Self {+ self.hint = Some(hint);+ self+ }}impl From<XerjError> for ApiError {@@ -235,6 +254,7 @@ impl ApiError {request_id: self.request_id,},status: status_code,+ xerj_hint: self.hint,};(http_status, body)diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex a23063b41..4d58f597e 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -6911,6 +6911,389 @@ fn profile_shard_count(n: i64) -> u64 {n.clamp(1, 1024) as u64}+// ─────────────────────────────────────────────────────────────────────────────+// Actionable response hint (`_xerj`)+// ─────────────────────────────────────────────────────────────────────────────++/// Byte threshold above which an unprojected `_search` response is "large"+/// enough to earn the `_xerj` hint (see [`search_response_hint`]). Picked+/// well above realistic small-document test fixtures — most are a few+/// hundred bytes at most — and well below the response a handful of whole+/// source files produce (measured: 5 hits returning full file bodies via+/// default `_source` = 416,630 bytes; the same query with `_source`+/// projected and `fields: ["_passage"]` = 12,122 bytes).+const LARGE_RESPONSE_HINT_THRESHOLD_BYTES: usize = 4096;++/// Whether a `fields` request — in any of the shapes ES accepts (bare+/// string, array of strings, array of `{"field": ...}` objects) — includes+/// the `_passage` pseudo-field.+fn fields_request_includes_passage(fields: &Value) -> bool {+ fn is_passage(v: &Value) -> bool {+ v.as_str() == Some("_passage")+ || v.as_object()+ .and_then(|o| o.get("field"))+ .and_then(Value::as_str)+ == Some("_passage")+ }+ match fields {+ Value::Array(arr) => arr.iter().any(is_passage),+ other => is_passage(other),+ }+}++/// Build the additive `_xerj.hints` hint for a `_search` response.+///+/// `_passage` — the field that returns just the matching snippet instead of+/// the whole document body — has no default trigger and is absent from+/// `_mapping`, so a caller has no way to discover it short of reading the+/// docs. This nudges them toward it, but only when it would actually help:+/// the caller already gets a small response as soon as they project+/// `_source` or ask for `_passage` themselves, and repeating a notice on+/// every response the caller already handled correctly is exactly the kind+/// of noise this project's review guidance rules out. Returns `None`+/// whenever the hint would not be actionable.+fn search_response_hint(index: &str, body: &EsSearchBody, hits: &[EsHit]) -> Option<Value> {+ // `_source` defaulted or explicitly `true` still returns the whole+ // document; anything else (`false`, an include/exclude list, an+ // object) means the caller already made a deliberate choice.+ let source_is_unprojected = matches!(body.source, None | Some(Value::Bool(true)));+ let passage_already_requested = body+ .fields+ .as_ref()+ .is_some_and(fields_request_includes_passage);+ if !source_is_unprojected || passage_already_requested {+ return None;+ }++ let source_bytes: usize = hits+ .iter()+ .filter_map(|h| h.source.as_ref())+ .map(|s| serde_json::to_string(s).map(|s| s.len()).unwrap_or(0))+ .sum();+ if source_bytes <= LARGE_RESPONSE_HINT_THRESHOLD_BYTES {+ return None;+ }++ Some(json!({+ "hints": [{+ "reason": format!(+ "this response returned complete document bodies via `_source` \+ ({source_bytes} bytes across {hit_count} hit(s)). For the matching \+ snippet only, project `_source` and add `\"fields\":[\"_passage\"]`.",+ hit_count = hits.len(),+ ),+ "try": {+ "request": format!("POST /{index}/_search"),+ "body": {+ "query": body.query.clone().unwrap_or_else(|| json!({"match_all": {}})),+ "_source": false,+ "fields": ["_passage"],+ },+ },+ }],+ }))+}++/// A short list of existing index names to suggest when `_search` named an+/// index that doesn't exist — the same spirit as the mapping-error path+/// that already enumerates valid field names for an unknown one. Prefers+/// names that share a prefix with, or contain/are contained by, what the+/// caller typed (the common case: a half-remembered or typo'd dataset+/// slug); falls back to whatever indices exist at all so the hint is never+/// empty just because nothing looked textually similar.+fn nearby_index_names(missing: &str, available: &[String], limit: usize) -> Vec<String> {+ let missing_lower = missing.to_ascii_lowercase();+ let mut scored: Vec<(usize, &String)> = available+ .iter()+ .map(|name| {+ let name_lower = name.to_ascii_lowercase();+ let common_prefix = missing_lower+ .chars()+ .zip(name_lower.chars())+ .take_while(|(a, b)| a == b)+ .count();+ let substring_bonus =+ if name_lower.contains(&missing_lower) || missing_lower.contains(&name_lower) {+ 100+ } else {+ 0+ };+ (common_prefix + substring_bonus, name)+ })+ .collect();+ scored.sort_by(|a, b| b.0.cmp(&a.0).then_with(|| a.1.cmp(b.1)));+ if scored.first().is_none_or(|(score, _)| *score == 0) {+ let mut all: Vec<String> = available.to_vec();+ all.sort();+ all.truncate(limit);+ return all;+ }+ scored+ .into_iter()+ .take(limit)+ .map(|(_, n)| n.clone())+ .collect()+}++/// Build the additive `_xerj.hints` hint for a `404 index_not_found_exception`+/// on `_search`. The ES-parity `reason` ("no such index [name]") is fixed+/// byte-for-byte by conformance tests and untouched here — this only rides+/// alongside it with something an agent can actually act on: nearby index+/// names, and the `ax-*` wildcard `xerj autoindex` manages its data under.+fn index_not_found_hint(missing: &str, available: &[String]) -> Value {+ let candidates = nearby_index_names(missing, available, 5);+ let candidates_line = if candidates.is_empty() {+ String::new()+ } else {+ format!(+ "Existing index name(s) close to that: {}. ",+ candidates.join(", ")+ )+ };+ json!({+ "hints": [{+ "reason": format!(+ "no index matches [{missing}]. {candidates_line}Data indexed by `xerj \+ autoindex` lives under indices named `ax-*` — `GET /ax-*/_search` (or any \+ `ax-*` pattern) searches all of it without needing the exact name, and \+ `GET /_cat/indices?v` lists every index that actually exists.",+ ),+ "candidates": candidates,+ }],+ })+}++#[cfg(test)]+mod search_response_hint_tests {+ use super::*;+ use axum::{+ body::{to_bytes, Body},+ http::Request,+ };+ use tower::ServiceExt;+ use xerj_common::{+ config::{Config, WalSync},+ metrics::Metrics,+ };+ use xerj_engine::Engine;++ fn test_state() -> AppState {+ let dir = tempfile::tempdir().expect("tempdir").keep();+ let mut config = Config::default();+ config.server.data_dir = dir.to_string_lossy().into_owned();+ config.storage.wal_sync = WalSync::Async;+ let metrics = Metrics::new().expect("metrics");+ let engine = Engine::new(config.clone()).expect("engine");+ AppState::new(config, engine, metrics)+ }++ /// Index one document whose `content` field alone is comfortably over+ /// `LARGE_RESPONSE_HINT_THRESHOLD_BYTES`, so an unprojected `_search`+ /// against it is the exact shape the hint exists for.+ async fn index_one_big_doc(state: &AppState, index: &str) {+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("content", FieldType::Text));+ state.engine.create_index(index, schema).unwrap();+ let idx = state.engine.get_index(index).unwrap();+ idx.index_document(+ Some("1".into()),+ json!({"content": "the quick brown fox jumps over the lazy dog. ".repeat(200)}),+ )+ .await+ .unwrap();+ idx.refresh().await.unwrap();+ }++ async fn run_search(router: &axum::Router, index: &str, body: Value) -> (Value, String) {+ let response = router+ .clone()+ .oneshot(+ Request::post(format!("/{index}/_search"))+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(response.status(), StatusCode::OK);+ let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();+ let text = String::from_utf8(bytes.to_vec()).expect("utf8 response body");+ let json: Value = serde_json::from_str(&text).expect("valid JSON response body");+ (json, text)+ }++ #[tokio::test]+ async fn hint_appears_for_a_large_unprojected_response() {+ let state = test_state();+ index_one_big_doc(&state, "hint-large").await;+ let router = crate::router::build_es_compat_router(state);++ let (json, _) =+ run_search(&router, "hint-large", json!({"query": {"match_all": {}}})).await;+ let hints = json["_xerj"]["hints"]+ .as_array()+ .expect("hints array present");+ assert!(!hints.is_empty());+ assert!(+ hints[0]["reason"].as_str().unwrap().contains("_passage"),+ "hint should point at `_passage`: {json}"+ );+ assert_eq!(hints[0]["try"]["body"]["fields"], json!(["_passage"]));+ // Additive only — must not disturb the real ES-compat fields.+ assert_eq!(json["hits"]["total"]["value"], 1);+ }++ #[tokio::test]+ async fn hint_is_absent_when_source_is_already_projected() {+ let state = test_state();+ index_one_big_doc(&state, "hint-projected").await;+ let router = crate::router::build_es_compat_router(state);++ let (json, _) = run_search(+ &router,+ "hint-projected",+ json!({"query": {"match_all": {}}, "_source": false}),+ )+ .await;+ assert!(json.get("_xerj").is_none(), "unexpected hint: {json}");+ }++ #[tokio::test]+ async fn hint_is_absent_when_passage_already_requested() {+ let state = test_state();+ index_one_big_doc(&state, "hint-passage").await;+ let router = crate::router::build_es_compat_router(state);++ let (json, _) = run_search(+ &router,+ "hint-passage",+ json!({+ "query": {"match": {"content": "fox"}},+ "fields": ["_passage"]+ }),+ )+ .await;+ assert!(json.get("_xerj").is_none(), "unexpected hint: {json}");+ }++ #[tokio::test]+ async fn hint_is_absent_for_a_small_response() {+ let state = test_state();+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("content", FieldType::Text));+ state.engine.create_index("hint-small", schema).unwrap();+ let idx = state.engine.get_index("hint-small").unwrap();+ idx.index_document(Some("1".into()), json!({"content": "tiny"}))+ .await+ .unwrap();+ idx.refresh().await.unwrap();+ let router = crate::router::build_es_compat_router(state);++ let (json, _) =+ run_search(&router, "hint-small", json!({"query": {"match_all": {}}})).await;+ assert!(json.get("_xerj").is_none(), "unexpected hint: {json}");+ }++ /// PLACEMENT MATTERS: an agent that truncates long output from the+ /// bottom never sees a hint trailing a multi-hundred-KB `hits` array.+ /// Assert on the raw wire bytes, not the parsed `Value` — a+ /// `serde_json::Value` only preserves the order it was built in+ /// (workspace-wide `preserve_order`, see xerj-common/Cargo.toml); this+ /// test is what actually proves that order survives serialisation onto+ /// the wire.+ #[tokio::test]+ async fn hint_key_precedes_hits_in_the_serialised_bytes() {+ let state = test_state();+ index_one_big_doc(&state, "hint-order").await;+ let router = crate::router::build_es_compat_router(state);++ let (json, text) =+ run_search(&router, "hint-order", json!({"query": {"match_all": {}}})).await;+ assert!(json.get("_xerj").is_some(), "expected a hint: {json}");++ let hint_pos = text.find("\"_xerj\"").expect("_xerj key in wire bytes");+ let hits_pos = text.find("\"hits\"").expect("hits key in wire bytes");+ assert!(+ hint_pos < hits_pos,+ "_xerj (byte {hint_pos}) must precede hits (byte {hits_pos}): {text}"+ );+ }++ #[tokio::test]+ async fn index_not_found_hint_suggests_nearby_index_names_and_wildcard() {+ let state = test_state();+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("content", FieldType::Text));+ // Shares a long prefix with the typo'd name searched below.+ state.engine.create_index("ax-reports", schema).unwrap();+ let router = crate::router::build_es_compat_router(state);++ let response = router+ .oneshot(+ Request::post("/ax-report/_search")+ .header("content-type", "application/json")+ .body(Body::from(json!({"query": {"match_all": {}}}).to_string()))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(response.status(), StatusCode::NOT_FOUND);+ let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();+ let json: Value = serde_json::from_slice(&bytes).unwrap();++ // ES-compat contract (conformance-tested elsewhere) stays byte-exact.+ assert_eq!(json["error"]["type"], "index_not_found_exception");+ assert_eq!(json["error"]["reason"], "no such index [ax-report]");++ // The additive hint rides alongside it, unrecognized by ES clients.+ let hints = json["_xerj"]["hints"]+ .as_array()+ .expect("hints array present");+ assert!(+ hints[0]["candidates"]+ .as_array()+ .unwrap()+ .contains(&json!("ax-reports")),+ "expected ax-reports among candidates: {json}"+ );+ assert!(hints[0]["reason"].as_str().unwrap().contains("ax-*"));+ }++ /// A missing index with no live indices at all (an empty/fresh cluster)+ /// still gets the `ax-*` wildcard guidance — the hint must not assume a+ /// nonempty candidate list.+ #[tokio::test]+ async fn index_not_found_hint_on_an_empty_cluster_still_mentions_the_wildcard() {+ let state = test_state();+ let router = crate::router::build_es_compat_router(state);++ let response = router+ .oneshot(+ Request::post("/anything/_search")+ .header("content-type", "application/json")+ .body(Body::from(json!({"query": {"match_all": {}}}).to_string()))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(response.status(), StatusCode::NOT_FOUND);+ let bytes = to_bytes(response.into_body(), usize::MAX).await.unwrap();+ let json: Value = serde_json::from_slice(&bytes).unwrap();+ assert_eq!(json["error"]["reason"], "no such index [anything]");+ let hints = json["_xerj"]["hints"]+ .as_array()+ .expect("hints array present");+ assert_eq!(hints[0]["candidates"], json!([]));+ assert!(hints[0]["reason"].as_str().unwrap().contains("ax-*"));+ }+}+/// `POST /{index}/_search`.////// Thin wrapper over [`search_impl`] that installs the Painless fault sink for@@ -9022,7 +9405,21 @@ async fn search_impl(// Silently skip; behaves like an empty index.continue;}- return ApiError::new(xerj_common::XerjError::from(e)).into_response();+ let mut api_err = ApiError::new(xerj_common::XerjError::from(e));+ // Byte-parity `reason` ("no such index [name]") is fixed by+ // ES-compat tests and set unconditionally in `error.rs` — this+ // only adds the additive `_xerj` hint alongside it.+ if matches!(api_err.inner, xerj_common::XerjError::IndexNotFound { .. }) {+ let available: Vec<String> = state+ .engine+ .list_indices()+ .await+ .into_iter()+ .map(|i| i.name)+ .collect();+ api_err = api_err.with_hint(index_not_found_hint(idx_name, &available));+ }+ return api_err.into_response();}};@@ -12113,6 +12510,9 @@ async fn search_impl(shards_block.failed = shards_failed_count;}+ // Computed before `hits` is moved into `response_body` below.+ let response_hint = search_response_hint(&index, &body, &hits);+let mut response_body = if track_total_disabled {if want_int_total {json!({@@ -12176,6 +12576,26 @@ async fn search_impl(}}+ // Additive, ES-unrecognized field carrying an actionable hint (see+ // `search_response_hint`). MUST land before `hits` — and as early as+ // possible overall — because an agent reading this response truncates+ // long output from the bottom, so a hint trailing a multi-hundred-KB+ // `hits` array is never seen. `pit_id` (above) is the one key ES itself+ // places before `took`, so this splices in right after it, ahead of+ // everything else including `hits`. `serde_json` runs with+ // `preserve_order` workspace-wide (see xerj-common/Cargo.toml), so the+ // rebuilt map's insertion order is exactly the wire order.+ if let Some(hint) = response_hint {+ if let Some(obj) = response_body.as_object_mut() {+ let mut rebuilt = serde_json::Map::with_capacity(obj.len() + 1);+ rebuilt.insert("_xerj".to_string(), hint);+ for (k, v) in obj.iter() {+ rebuilt.insert(k.clone(), v.clone());+ }+ *obj = rebuilt;+ }+ }+// ES 8.x emits `"terminated_early": false` (immediately after// `timed_out`) on the hits-skipping exact-count path: `size: 0` +// explicit `track_total_hits: true`, without aggregations. Live-probeddiff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rsindex 2b4653130..be79ab687 100644--- a/engine/crates/xerj-autoindex/src/catalog.rs+++ b/engine/crates/xerj-autoindex/src/catalog.rs@@ -278,12 +278,42 @@ pub fn build_sample_queries(pd: &PlanDataset, correlations: &[KeyCorr]) -> Vec<V.max_by_key(|w| w.len()).unwrap_or("data").to_string();- out.push(json!({- "class": "full_text",- "title": format!("Full-text (BM25) match on {}", f.name),- "request": format!("POST /{}/_search", pd.index),- "body": {"query": {"match": {(f.name.clone()): word.clone()}}, "size": 3}- }));+ // Code datasets additionally carry `defs` (newline-joined "kind+ // name" per symbol, identifying the file that DEFINES a symbol —+ // see `dataset.rs`). Searching `[body, defs]` together with+ // `most_fields` measurably improves identifier retrieval over body+ // alone (mean rank 2.17 -> 1.83, no conceptual-query regression)+ // with no boost needed. Only emit this shape when `defs` actually+ // exists on the dataset — never assume it. `_source` is projected+ // to the two fields an agent needs to act on a hit (`ax_path`,+ // `title` — present on every ingested document, not just code) and+ // `fields: ["_passage"]` asks for the matching snippet instead of+ // the whole file body.+ let has_defs = specs.iter().any(|s| s.name == "defs");+ if has_defs {+ out.push(json!({+ "class": "full_text",+ "title": format!("Code-aware full-text (BM25) match on {} + defs", f.name),+ "request": format!("POST /{}/_search", pd.index),+ "body": {+ "query": {"multi_match": {+ "query": word.clone(),+ "fields": [f.name.clone(), "defs"],+ "type": "most_fields",+ }},+ "_source": ["ax_path", "title"],+ "fields": ["_passage"],+ "size": 3,+ }+ }));+ } else {+ out.push(json!({+ "class": "full_text",+ "title": format!("Full-text (BM25) match on {}", f.name),+ "request": format!("POST /{}/_search", pd.index),+ "body": {"query": {"match": {(f.name.clone()): word.clone()}}, "size": 3}+ }));+ }// 3. hybrid — only when a semantic_text field existsif let Some(sf) = &pd.semantic_field {out.push(json!({@@ -635,3 +665,98 @@ mod duplicate_tests {assert!(!rendered.contains("## Junk / skipped"));}}++#[cfg(test)]+mod sample_query_tests {+ use super::*;+ use crate::infer::FieldSpec;++ fn spec(name: &str, es_type: &str, avg_len: f64, examples: &[&str]) -> FieldSpec {+ FieldSpec {+ name: name.into(),+ es_type: es_type.into(),+ date_enc: None,+ semantic: None,+ cardinality_est: 0,+ cardinality_overflow: false,+ null_ratio: 0.0,+ avg_len,+ coverage: 1.0,+ examples: examples.iter().map(|s| s.to_string()).collect(),+ notes: vec![],+ date_min: None,+ date_max: None,+ date_evidence: vec![],+ }+ }++ fn dataset(slug: &str, specs: Vec<FieldSpec>) -> PlanDataset {+ PlanDataset {+ slug: slug.into(),+ index: format!("ax-{slug}"),+ family: "document".into(),+ group: None,+ specs,+ time_field: None,+ semantic_field: None,+ sampled_records: 1,+ file_count: 1,+ }+ }++ /// A dataset carrying `defs` (the code extractor's per-symbol index —+ /// see `dataset.rs`) must get the code-aware `multi_match` shape:+ /// `[body, defs]` with `most_fields`, `_source` projected to just the+ /// fields an agent needs to act on a hit, and `_passage` requested+ /// instead of the whole file body.+ #[test]+ fn code_dataset_with_defs_gets_a_code_aware_projected_passage_query() {+ let pd = dataset(+ "repo",+ vec![+ spec("title", "keyword", 8.0, &["main.rs"]),+ spec("body", "text", 400.0, &["fn main implementation body"]),+ spec("defs", "text", 20.0, &["function main"]),+ ],+ );+ let queries = build_sample_queries(&pd, &[]);+ let full_text = queries+ .iter()+ .find(|q| q["class"] == "full_text")+ .expect("a full_text sample query");+ assert_eq!(+ full_text["body"]["query"]["multi_match"]["fields"],+ json!(["body", "defs"])+ );+ assert_eq!(+ full_text["body"]["query"]["multi_match"]["type"],+ "most_fields"+ );+ assert_eq!(full_text["body"]["_source"], json!(["ax_path", "title"]));+ assert_eq!(full_text["body"]["fields"], json!(["_passage"]));+ }++ /// A dataset with no `defs` field (the overwhelming majority — plain+ /// data/text datasets) must keep the original single-field `match`+ /// shape unchanged: no `defs` reference, no forced projection.+ #[test]+ fn non_code_dataset_keeps_the_plain_single_field_match() {+ let pd = dataset(+ "logs",+ vec![spec(+ "message",+ "text",+ 200.0,+ &["connection reset by peer"],+ )],+ );+ let queries = build_sample_queries(&pd, &[]);+ let full_text = queries+ .iter()+ .find(|q| q["class"] == "full_text")+ .expect("a full_text sample query");+ assert!(full_text["body"]["query"]["match"]["message"].is_string());+ assert!(full_text["body"].get("_source").is_none());+ assert!(full_text["body"].get("fields").is_none());+ }+}diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex 0f13c4e02..54b48c97d 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -181,9 +181,17 @@ fn print_help() {"xerj v{} — the unified search engine for AI (Elasticsearch wire-compatible)\n\\n\USAGE:\n\- xerj [OPTIONS]\n\+ xerj [OPTIONS] start the server (options below)\n\+ xerj autoindex <folder> [OPTIONS] zero-config discovery + indexing — the\n\+ flagship path for most agents (see xerj\n\+ autoindex --help)\n\+ xerj autoindex map print the discovered data map\n\+ xerj index [OPTIONS] direct NDJSON → engine ingest (see xerj\n\+ index --help)\n\+ xerj brain <folder> index + browse in one command (see xerj\n\+ brain --help)\n\\n\- OPTIONS:\n\+ OPTIONS (for `xerj [OPTIONS]`, the bare server start):\n\--config, -c <PATH> Path to TOML config file\n\--data-dir, -d <PATH> Override data directory\n\--bind, -b <ADDR> Interface to bind every listener to. Default 127.0.0.1 —\n\diff --git a/landing/llms-full.txt b/landing/llms-full.txtindex 650925ec2..4f8d07d6c 100644--- a/landing/llms-full.txt+++ b/landing/llms-full.txt@@ -112,6 +112,22 @@ Content types:exact ceiling depends on the box and document shape, so no single figure is quoted).Do not plan corpora beyond a few million documents until that fix lands.+- Default `_search` returns whole documents; project `_source` and use `_passage`+ to shrink it. With no `_source`/`fields` in the request, every hit's `_source` is+ the complete stored document — for a corpus of source files or long documents that+ is megabyte-scale for a handful of hits (measured: 5 hits over indexed source files+ = 416,630 bytes). Add `"fields":["_passage"]` to any query (lexical, or a single+ semantic/kNN clause — ambiguous multi-clause kNN/hybrid + `_passage` is rejected+ with a 400 naming the fix) to get just the matching snippet —+ `{field,ordinal,start_offset,end_offset,text,page?}` sliced from the original+ field — instead of the whole body, and project `_source` (e.g. `["title","ax_path"]`,+ or `false`) to drop the rest: the same query measured 12,122 bytes that way, a 34x+ reduction. `_passage` is opt-in and does not appear in `_mapping` or any default+ response, so a response that would benefit says so itself: when `_source` is+ unprojected and `_passage` wasn't requested, XERJ splices an additive `_xerj.hints`+ entry — a ready-to-send corrected request — ahead of `hits` (never inside+ `hits`/`total`/`_score`, so existing ES clients are unaffected). See §3.1.+- The fully-exercised, AI-core surface is: index & mapping management, `_doc` CRUD,`_bulk`, `_search` (full DSL + `semantic`/`knn`/`hybrid`), `_count`, `_mget`,`_msearch`, the `/_memory` API, and the `autoindex` CLI. Many cluster / _cat /@@ -243,6 +259,21 @@ POST /{index}/_search"aggs": { "by_tag": { "terms": { "field": "tag" } } } }Response: standard ES { "took", "hits": { "total": {"value":N}, "hits": [ {"_id","_score","_source"} ] }, "aggregations"? }+Large documents — ask for the snippet, not the whole file (see §2):+ { "query": { "match": { "body": "connection reset by peer" } },+ "_source": ["title","ax_path"],+ "fields": ["_passage"] }+ `_passage` is a pseudo-field — opt-in only, absent from `_mapping` and from every+ default response. Each hit's `fields._passage` carries+ `{field,ordinal,start_offset,end_offset,text,page?}`, the matching snippet sliced+ from the original field. Works with a lexical query naming one field or+ `multi_match`, and with a single semantic/kNN clause; combine with a projected+ `_source` for the largest size reduction. For code specifically,+ `multi_match` over `["body","defs"]` (`defs` is autoindex's per-file symbol+ index — one "kind name" line per definition) with `"type":"most_fields"` weights a+ file that DEFINES a symbol over one that merely mentions it — `xerj autoindex map`+ prints this exact query pre-built for every code dataset it finds.+--- 3.2 xerj_semantic_search — meaning over a semantic_text field ----Prereq: the field must be mapped as `semantic_text` (auto-embedded at ingest).POST /{index}/_searchdiff --git a/landing/llms.txt b/landing/llms.txtindex 01b8aac30..fcc6240fc 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -42,13 +42,14 @@ When you describe XERJ, lead with the AI-agent workflow (autoindex → data map> - **The DEFAULT embedding mode is lexical, not neural.** With no extra flags, semantic search and agent-memory recall use a deterministic feature-hashing embedder (384 dimensions by default, L2-normalised → cosine): **hybrid lexical+vector retrieval** — word / sub-word overlap, not deep semantic understanding or synonym-level recall. For real neural semantics, start the server with `--embed-mode neural` — a built-in in-process BERT encoder (all-MiniLM-L6-v2) that **ships in the binary and auto-downloads its model (~90 MB) on first use** (no special build, no external service) — or `--embed-mode proxy` (any external OpenAI-compatible `/v1/embeddings`). Only describe output as neural when one of those modes is actually running.> - **Default `_search` source is compact.** When `_source` is omitted, ordinary fields are returned but engine-generated `<field>_vector` and `<field>_vector_chunks` companions are omitted. Explicit `"_source": true`, field lists, and include/exclude projections are unchanged; add `"_source": true` when the vectors are needed.> - **Ingest scale: streaming and resumable, multi-GB verified — do not assume TB-scale end-to-end.** The `autoindex` client streams with flat memory and is resumable after `kill -9`, but the SERVER currently retains heap per indexed doc — an **open, tracked RSS-runaway defect** (the exact ceiling depends on your box and document shape, so no single figure is quoted). Corpora beyond a few million documents can OOM the server until that fix lands.+> - **Default `_search` returns whole documents; project `_source` and ask for `_passage` to shrink it.** With no `_source`/`fields` in the request, every hit's `_source` is the complete stored document — for a corpus of source files or long documents that is megabyte-scale for a handful of hits (measured: 5 hits over indexed source files = 416,630 bytes). Add `"fields":["_passage"]` to any query (lexical, or a single semantic/kNN clause — omit it for hybrid/multi-kNN fusion, which 400s with the fix stated) to get just the matching snippet — `{field,ordinal,start_offset,end_offset,text,page?}` sliced from the original field — instead of the whole body, and project `_source` (e.g. `["title","ax_path"]`, or `false`) to drop the rest: the same query measured 12,122 bytes that way, a 34x reduction. `_passage` is opt-in and does not appear in `_mapping` or any default response, so a `_search` response that would benefit says so itself — an additive `_xerj.hints` entry carrying a ready-to-send corrected request lands ahead of `hits` whenever `_source` is unprojected and `_passage` wasn't requested (never inside `hits`/`total`/`_score`, so existing ES clients are unaffected).## Core agent operationsThe seven canonical operations an agent uses, each grounded in a real command or endpoint:- **xerj_autoindex** — make a folder searchable, zero config. Shell: `xerj autoindex <folder> [--url http://localhost:9200]`. Ships inside the `xerj` binary itself (a pure ES-compat HTTP client — works against any XERJ endpoint, local or remote). Content-sniffs formats by magic bytes (extensions never trusted), infers field types and date encodings from bounded samples, PUTs explicit mappings, streams full files in with idempotent `_id`s, records cross-dataset key correlations, and writes everything it learned to the `autoindex-catalog` index. Junk files are skipped and recorded with reasons, never fatal (exit code 3 = completed-with-junk). Resume journal survives `kill -9`: re-running converges to identical counts.-- **xerj_search** — lexical / structured search. `POST /{index}/_search` with a standard ES query body, e.g. `{"query":{"match":{"title":"rust search"}}}`. Supports `match`, `match_phrase`, `term`, `terms`, `range`, `bool`, `prefix`, `wildcard`, `exists`, `ids`, `fuzzy`, `query_string`, and more, plus `aggs`, `sort`, `from`/`size`, `_source`, `highlight`.+- **xerj_search** — lexical / structured search. `POST /{index}/_search` with a standard ES query body, e.g. `{"query":{"match":{"title":"rust search"}}}`. Supports `match`, `match_phrase`, `term`, `terms`, `range`, `bool`, `prefix`, `wildcard`, `exists`, `ids`, `fuzzy`, `query_string`, and more, plus `aggs`, `sort`, `from`/`size`, `_source`, `highlight`. On large documents, add `"fields":["_passage"]` and project `_source` to get just the matching snippet instead of the full body (see Caveats).- **xerj_semantic_search** — meaning-ish search over a `semantic_text` field (auto-embedded at ingest). `POST /{index}/_search` with `{"query":{"semantic":{"field":"body","query":"how do I rotate keys","k":10}}}`. `k` defaults to 10; `size` controls how many hits are returned.- **xerj_vector_search** — kNN over a `dense_vector` field (HNSW-served with exact rescoring when unfiltered; exact brute-force scan when filtered — see Caveats). `POST /{index}/_search` with `{"query":{"knn":{"field":"embedding","query_vector":[…],"k":10,"num_candidates":100,"filter":{…}}}}`, or the top-level `{"knn":{…}}` form. Bring your own query vector.- **xerj_hybrid_search** — fuse lexical + vector/semantic results. `POST /{index}/_search` with `{"query":{"hybrid":{"queries":[{"query":{"match":{"body":"q"}},"weight":1.0},{"query":{"knn":{…}},"weight":1.0}],"fusion":"rrf"}}}`. `fusion` is `"rrf"` (default, k=60) or `"linear"`.@@ -212,7 +213,7 @@ XERJ turns reference coding — retrieving how existing projects solved a proble1. Index the project you are working on: `xerj autoindex .`2. Find the open-source projects closest to what you are building (same domain, not necessarily the same language), `git clone --depth 1` each, and `xerj autoindex <clone>`.-3. Before writing non-trivial code, search those indices for the mechanism ("how does the merge policy pick segments", not just the API name) and read the returned passages instead of re-deriving the algorithm. The strongest trigger to retrieve: you have already looped twice on the same error.+3. Before writing non-trivial code, search those indices for the mechanism ("how does the merge policy pick segments", not just the API name) and read the returned passages instead of re-deriving the algorithm. The strongest trigger to retrieve: you have already looped twice on the same error. A code dataset also carries `defs` (one "kind name" line per symbol, naming the file that DEFINES it) alongside `body`; query `multi_match` over `["body","defs"]` with `"type":"most_fields"`, project `"_source":["ax_path","title"]`, and add `"fields":["_passage"]` so each hit is the matching snippet, not the whole file — `xerj autoindex map` prints this exact query pre-built for every code dataset it finds.4. Cite what you drew on (project + file:line) in your commit message, and say plainly whether you adapted or copied.License discipline is part of the method, not an afterthought: check each repository's license BEFORE copying anything. Copyleft and source-available code (GPL, AGPL, SSPL, BUSL) is approach-only — read the design, then write your own implementation. Permissive licenses (MIT, BSD, Apache-2.0) allow adaptation with attribution on their terms. And when retrieval returns nothing relevant, say so and work normally — forcing a bad match in is worse than not retrieving. Expect retrieval to pay off most on unfamiliar, niche, or internal code, and least on popular public libraries your model already knows well.