Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(search): gate code definitions with a phrase clause

Leonid Bugaev57e5ddc5fdd1
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 846187039..293e6061f 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -7202,7 +7202,17 @@ fn get_doc_response_hint(index: &str, doc_bytes: usize) -> Value {
"try": {
"request": format!("POST /{index}/_search"),
"body": {
- "query": {"match": {"body": "<the symbol or phrase you want>"}},
+ "query": {"bool": {"should": [
+ {"multi_match": {
+ "query": "<the symbol or phrase you want>",
+ "fields": ["body", "defs"],
+ "type": "most_fields",
+ }},
+ {"match_phrase": {"defs": {
+ "query": "<the symbol or phrase you want>",
+ "boost": 8,
+ }}},
+ ]}},
// Project the path rather than dropping `_source`: a blind
// run got the passage, could not tell which file it came
// from, and spent 6 MB on follow-ups recovering it. Costs
diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rs
index be79ab687..06b0cfe65 100644
--- a/engine/crates/xerj-autoindex/src/catalog.rs
+++ b/engine/crates/xerj-autoindex/src/catalog.rs
@@ -280,15 +280,16 @@ pub fn build_sample_queries(pd: &PlanDataset, correlations: &[KeyCorr]) -> Vec<V
.to_string();
// Code datasets additionally carry `defs` (newline-joined "kind
// name" per symbol, identifying the file that DEFINES a symbol —
- // see `dataset.rs`). Searching `[body, defs]` together with
- // `most_fields` measurably improves identifier retrieval over body
- // alone (mean rank 2.17 -> 1.83, no conceptual-query regression)
- // with no boost needed. Only emit this shape when `defs` actually
- // exists on the dataset — never assume it. `_source` is projected
- // to the two fields an agent needs to act on a hit (`ax_path`,
- // `title` — present on every ingested document, not just code) and
- // `fields: ["_passage"]` asks for the matching snippet instead of
- // the whole file body.
+ // see `dataset.rs`). `defs` is analyzed text, so `strip_shortcodes`
+ // tokenizes to `strip` + `shortcodes`; boosting it alone amplifies
+ // conceptual noise. The phrase clause requires the token sequence,
+ // so it contributes for a symbol definition but essentially never
+ // for a multi-word conceptual query. Only emit this shape when `defs`
+ // actually exists on the dataset — never assume it. `_source` is
+ // projected to the two fields an agent needs to act on a hit
+ // (`ax_path`, `title` — present on every ingested document, not just
+ // code) and `fields: ["_passage"]` asks for the matching snippet
+ // instead of the whole file body.
let has_defs = specs.iter().any(|s| s.name == "defs");
if has_defs {
out.push(json!({
@@ -296,11 +297,17 @@ pub fn build_sample_queries(pd: &PlanDataset, correlations: &[KeyCorr]) -> Vec<V
"title": format!("Code-aware full-text (BM25) match on {} + defs", f.name),
"request": format!("POST /{}/_search", pd.index),
"body": {
- "query": {"multi_match": {
- "query": word.clone(),
- "fields": [f.name.clone(), "defs"],
- "type": "most_fields",
- }},
+ "query": {"bool": {"should": [
+ {"multi_match": {
+ "query": word.clone(),
+ "fields": [f.name.clone(), "defs"],
+ "type": "most_fields",
+ }},
+ {"match_phrase": {"defs": {
+ "query": word.clone(),
+ "boost": 8,
+ }}},
+ ]}},
"_source": ["ax_path", "title"],
"fields": ["_passage"],
"size": 3,
@@ -705,10 +712,9 @@ mod sample_query_tests {
}
/// A dataset carrying `defs` (the code extractor's per-symbol index —
- /// see `dataset.rs`) must get the code-aware `multi_match` shape:
- /// `[body, defs]` with `most_fields`, `_source` projected to just the
- /// fields an agent needs to act on a hit, and `_passage` requested
- /// instead of the whole file body.
+ /// see `dataset.rs`) must get the code-aware bool shape: a `[body, defs]`
+ /// `most_fields` query plus a self-gating `match_phrase` on `defs`, with
+ /// `_source` projected and `_passage` requested instead of the whole file.
#[test]
fn code_dataset_with_defs_gets_a_code_aware_projected_passage_query() {
let pd = dataset(
@@ -724,14 +730,17 @@ mod sample_query_tests {
.iter()
.find(|q| q["class"] == "full_text")
.expect("a full_text sample query");
+ let should = full_text["body"]["query"]["bool"]["should"]
+ .as_array()
+ .expect("code-aware query should have bool.should");
+ assert_eq!(should.len(), 2);
+ assert_eq!(should[0]["multi_match"]["fields"], json!(["body", "defs"]));
+ assert_eq!(should[0]["multi_match"]["type"], "most_fields");
assert_eq!(
- full_text["body"]["query"]["multi_match"]["fields"],
- json!(["body", "defs"])
- );
- assert_eq!(
- full_text["body"]["query"]["multi_match"]["type"],
- "most_fields"
+ should[1]["match_phrase"]["defs"]["query"],
+ should[0]["multi_match"]["query"]
);
+ assert_eq!(should[1]["match_phrase"]["defs"]["boost"], json!(8));
assert_eq!(full_text["body"]["_source"], json!(["ax_path", "title"]));
assert_eq!(full_text["body"]["fields"], json!(["_passage"]));
}
@@ -756,6 +765,8 @@ mod sample_query_tests {
.find(|q| q["class"] == "full_text")
.expect("a full_text sample query");
assert!(full_text["body"]["query"]["match"]["message"].is_string());
+ assert!(full_text["body"]["query"].get("bool").is_none());
+ assert!(full_text["body"]["query"].get("match_phrase").is_none());
assert!(full_text["body"].get("_source").is_none());
assert!(full_text["body"].get("fields").is_none());
}
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index ba4588620..d6ae41fcd 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -4709,10 +4709,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
println!(
" code was indexed — for a function/class instead of a whole file:\n\
\x20 curl -s $URL/{}-*/_search -H 'Content-Type: application/json' \\\n\
- \x20 -d '{{\"query\":{{\"multi_match\":{{\"query\":\"<symbol or phrase>\",\
- \"fields\":[\"body\",\"defs\"],\"type\":\"most_fields\"}}}},\
+ \x20 -d '{{\"query\":{{\"bool\":{{\"should\":[{{\"multi_match\":{{\"query\":\"<symbol or phrase>\",\
+ \"fields\":[\"body\",\"defs\"],\"type\":\"most_fields\"}}}},{{\"match_phrase\":{{\"defs\":{{\"query\":\"<symbol or phrase>\",\"boost\":8}}}}}}]}}}},\
\"_source\":[\"ax_path\",\"title\"],\"fields\":[\"_passage\"]}}'\n\
- \x20 (`defs` matches the file that DEFINES a symbol; `_passage` returns \
+ \x20 (the `match_phrase` clause ranks the file that DEFINES a symbol above files that merely call it; `_passage` returns \
the enclosing block, not the file)",
cfg.prefix
);
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index d97b5a62d..7743e7208 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -260,11 +260,11 @@ fn print_help() {
whole file. To get only the matching function or class, ask for the passage:\n\
\n\
curl -s $URL/<index>/_search -H 'Content-Type: application/json' \\\n\
- -d '{{\"query\":{{\"multi_match\":{{\"query\":\"<symbol or phrase>\",\n\
- \"fields\":[\"body\",\"defs\"],\"type\":\"most_fields\"}}}},\n\
+ -d '{{\"query\":{{\"bool\":{{\"should\":[{{\"multi_match\":{{\"query\":\"<symbol or phrase>\",\n\
+ \"fields\":[\"body\",\"defs\"],\"type\":\"most_fields\"}}}},{{\"match_phrase\":{{\"defs\":{{\"query\":\"<symbol or phrase>\",\"boost\":8}}}}}}]}}}},\n\
\"_source\":[\"ax_path\",\"title\"],\"fields\":[\"_passage\"]}}'\n\
\n\
- `defs` matches the file that DEFINES a symbol rather than one that mentions it;\n\
+ the `match_phrase` clause ranks the file that DEFINES a symbol above files that merely call it;\n\
`_passage` returns the enclosing block. Without them you get whole files.\n\
\n\
ENVIRONMENT:\n\
diff --git a/landing/llms-full.txt b/landing/llms-full.txt
index 4f8d07d6c..94d0077f5 100644
--- a/landing/llms-full.txt
+++ b/landing/llms-full.txt
@@ -269,10 +269,11 @@ Large documents — ask for the snippet, not the whole file (see §2):
from the original field. Works with a lexical query naming one field or
`multi_match`, and with a single semantic/kNN clause; combine with a projected
`_source` for the largest size reduction. For code specifically,
- `multi_match` over `["body","defs"]` (`defs` is autoindex's per-file symbol
- index — one "kind name" line per definition) with `"type":"most_fields"` weights a
- file that DEFINES a symbol over one that merely mentions it — `xerj autoindex map`
- prints this exact query pre-built for every code dataset it finds.
+ `bool.should` with `multi_match` over `["body","defs"]` and a
+ `match_phrase` on `defs` with "boost":8 (`defs` is autoindex's per-file symbol
+ index — one "kind name" line per definition) ranks a file that DEFINES a symbol
+ over one that merely mentions it — `xerj autoindex map` prints this exact query
+ pre-built for every code dataset it finds.
--- 3.2 xerj_semantic_search — meaning over a semantic_text field ----
Prereq: the field must be mapped as `semantic_text` (auto-embedded at ingest).
diff --git a/landing/llms.txt b/landing/llms.txt
index fcc6240fc..e28713ad5 100644
--- a/landing/llms.txt
+++ b/landing/llms.txt
@@ -213,7 +213,7 @@ XERJ turns reference coding — retrieving how existing projects solved a proble
1. Index the project you are working on: `xerj autoindex .`
2. Find the open-source projects closest to what you are building (same domain, not necessarily the same language), `git clone --depth 1` each, and `xerj autoindex <clone>`.
-3. Before writing non-trivial code, search those indices for the mechanism ("how does the merge policy pick segments", not just the API name) and read the returned passages instead of re-deriving the algorithm. The strongest trigger to retrieve: you have already looped twice on the same error. A code dataset also carries `defs` (one "kind name" line per symbol, naming the file that DEFINES it) alongside `body`; query `multi_match` over `["body","defs"]` with `"type":"most_fields"`, project `"_source":["ax_path","title"]`, and add `"fields":["_passage"]` so each hit is the matching snippet, not the whole file — `xerj autoindex map` prints this exact query pre-built for every code dataset it finds.
+3. Before writing non-trivial code, search those indices for the mechanism ("how does the merge policy pick segments", not just the API name) and read the returned passages instead of re-deriving the algorithm. The strongest trigger to retrieve: you have already looped twice on the same error. A code dataset also carries `defs` (one "kind name" line per symbol, naming the file that DEFINES it) alongside `body`; query a `bool.should` with `multi_match` over `["body","defs"]` and `match_phrase` on `defs` with "boost":8, project `"_source":["ax_path","title"]`, and add `"fields":["_passage"]` so each hit is the matching snippet, not the whole file — `xerj autoindex map` prints this exact query pre-built for every code dataset it finds.
4. Cite what you drew on (project + file:line) in your commit message, and say plainly whether you adapted or copied.
License discipline is part of the method, not an afterthought: check each repository's license BEFORE copying anything. Copyleft and source-available code (GPL, AGPL, SSPL, BUSL) is approach-only — read the design, then write your own implementation. Permissive licenses (MIT, BSD, Apache-2.0) allow adaptation with attribution on their terms. And when retrieval returns nothing relevant, say so and work normally — forcing a bad match in is worse than not retrieving. Expect retrieval to pay off most on unfamiliar, niche, or internal code, and least on popular public libraries your model already knows well.