AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
#825 knn beside query scores as a sum over the union, aggs included (#879)
Onyeka Obid2f68019fc8f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 063714f4e..b6f6feb0d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **A top-level `knn` beside a `query` now scores as a sum over the union of+ both halves, aggregations included**+ ([#825](https://github.com/xerj-org/xerj/issues/825)). ES 8.x defines this+ shape as a disjunction: the hit set is the union of the lexical matches and+ the global top-`k` neighbours, each document scores `query_score ++ knn_score`, and aggregations are computed over that union. XERJ folded the+ request to `bool.should[query, knn]` but the generic matcher/scorer has no+ `knn` arm, so a vector-only document vanished and a lexical match lost its+ vector score — silently, with `_shards.failed: 0`. The+ [#458](https://github.com/xerj-org/xerj/issues/458) carve-out covered one+ narrow shape with RRF rank fusion (itself a scoring divergence from ES) and+ left every request carrying `aggs`, `sort`, `collapse` or `rescore` on the+ silent drop. The engine now pre-executes the `knn` leg and pins its top-`k`+ into the tree as per-document constant scores, so one path serves the union+ with summed scores and every request feature applies to it.++ Two behaviour changes worth knowing before you upgrade:++ - `{query, knn:{field: <not a vector field>}, aggs}` now answers **400**+ (`the field is not answerable by knn`), matching ES and the existing+ [#498](https://github.com/xerj-org/xerj/issues/498) rule. It previously+ answered 200 with lexical-only hits — the wrong answer, quietly.+ - **This shape is slower.** A pinned tree contains `ids` clauses, which the+ FTS projection cannot lift, so the request is answered by a stored-document+ scan instead of the inverted index. Measured on 100 000 documents (`text` ++ 8-dim `dense_vector`, ~10 % lexical selectivity, closed-loop, fresh query+ vector per request): `query` + `knn` k=10 goes from 2.5 ms (the old RRF+ route, which returned ES-divergent scores) to ~208 ms; k=1000 to ~385 ms;+ k=10000 to ~5.1 s. The shapes that previously carried the silent drop —+ anything with `aggs`/`sort`/`collapse`/`rescore` — already took the scan+ and cost about a quarter more (476 ms → 605 ms with a terms agg),+ but they now return the right documents. Restoring an indexed route is+ tracked as [#892](https://github.com/xerj-org/xerj/issues/892);+ correctness landed first.+- **`match_phrase` `slop` now admits transposed terms at Lucene's cost of 2**([#830](https://github.com/xerj-org/xerj/issues/830)). The sloppy-phrasewalk was in-order only — each next term matched strictly after thediff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex e68bbc6cb..396eab9c5 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -7486,16 +7486,25 @@ fn vector_query_fields(q: &Value, out: &mut Vec<String>) {}/// Fields whose vector clause reaches the vector for this query, approximating-/// the engine's `peel_semantic_query` / `peel_knn_query` dispatch (index.rs) for-/// the shapes that can co-exist with a lexical `semantic_text` clause. A-/// `semantic`/`knn` clause consults the vector only as the whole query, as the-/// SOLE `must`/`should` candidate of a `bool` with no `must_not`, or inside a-/// `hybrid` (which fuses and runs every branch). A `semantic`/`knn` clause-/// sitting BESIDE a sibling in a multi-clause `bool` falls through to the lexical-/// path and never consults the vector (#394) — so it must NOT count as "reached-/// the vector" here, or the hint gets suppressed on a query that ran BM25-only,+/// the engine's `peel_semantic_query` / `peel_knn_query` /+/// `compound_bool_direct_knn` dispatch (index.rs) for the shapes that can+/// co-exist with a lexical `semantic_text` clause.+///+/// A `semantic` clause consults the vector only as the whole query, as the SOLE+/// `must`/`should` candidate of a `bool` with no `must_not`, or inside a+/// `hybrid` (which fuses and runs every branch). A `semantic` clause sitting+/// BESIDE a sibling in a multi-clause `bool` falls through to the lexical path+/// and never consults the vector (#394) — so it must NOT count as "reached the+/// vector" here, or the hint gets suppressed on a query that ran BM25-only,/// reintroducing the exact silence the hint exists to break.///+/// A `knn` clause is no longer bound by that (#825). A `knn` that is a DIRECT+/// child of a compound bool's scoring lists is pre-executed by the engine and+/// its top-k pinned into the tree, so it dispatches whatever its siblings are —+/// and that is precisely the tree this module's `knn`-beside-`query` fold+/// emits, so the hand-rolled ES hybrid must not be nagged for running BM25+/// "only".+////// This is NOT a byte-exact mirror of `peel`, and does not need to be (#777):/// where it diverges the shapes are contrived or carry no lexical clause, so none/// can produce an observable wrong hint — a pure multi-`knn` or `nested` bool has@@ -7506,6 +7515,14 @@ fn vector_query_fields(q: &Value, out: &mut Vec<String>) {/// `boost`/`_name` is an inline leaf param, not a wrapper key, so nothing/// dispatching is missed.///+/// The `knn` arm inherits exactly that one known divergence and no other. A+/// one-clause `bool` is erased before the engine reads the tree (#399), so+/// descending through a sole candidate into a compound bool tracks+/// `compound_bool_direct_knn` faithfully — except in the same contrived+/// `bool{should:[bool{should:[knn, match]}], filter:[…]}` shape, where the+/// wrapper is not erased, the engine does NOT pin, and this over-suppresses.+/// Same class, same contrivance, no new one.+////// Contrast [`vector_query_fields`], which collects every mention regardless of/// shape and is right only for the top-level `knn` block (which always dispatches).fn dispatching_vector_fields(q: &Value, out: &mut Vec<String>) {@@ -7539,27 +7556,48 @@ fn dispatching_vector_fields(q: &Value, out: &mut Vec<String>) {if let Some(inner) = o.get("constant_score").and_then(|c| c.get("filter")) {dispatching_vector_fields(inner, out);}- // `bool` reaches its vector clause ONLY in the single-`must`/`should`+ // `bool` reaches a `semantic` clause ONLY in the single-`must`/`should`// candidate, no-`must_not` shape the engine peels; every other bool falls// through to the lexical path and consults no vector.+ //+ // `knn` is no longer bound by that (#825): a `knn` clause sitting DIRECTLY+ // in a compound bool's SCORING lists is now pre-executed by the engine and+ // its top-k pinned into the tree (`compound_bool_direct_knn` /+ // `pin_knn_clause` in xerj-engine), so it genuinely dispatches however many+ // siblings it has — which is exactly the tree this module's own+ // `knn`-beside-`query` fold emits. Mirror the engine's gate: exactly one+ // direct `Knn` child across `must` + `should`, `must_not`/`filter`+ // irrelevant. `semantic` keeps the #394 rule, which still holds for it.if let Some(Value::Object(b)) = o.get("bool") {+ let mut scoring: Vec<&Value> = Vec::new();+ for key in ["must", "should"] {+ match b.get(key) {+ Some(Value::Array(a)) => scoring.extend(a.iter()),+ Some(Value::Null) | None => {}+ Some(other) => scoring.push(other),+ }+ }+ let mut direct_knn = scoring.iter().filter(|c| c.get("knn").is_some());+ if let (Some(clause), None) = (direct_knn.next(), direct_knn.next()) {+ if let Some(spec) = clause.get("knn") {+ let specs: Vec<&Value> = match spec {+ Value::Array(a) => a.iter().collect(),+ single => vec![single],+ };+ for spec in specs {+ if let Some(f) = spec.get("field").and_then(Value::as_str) {+ out.push(f.to_string());+ }+ }+ }+ }let must_not_empty = match b.get("must_not") {None | Some(Value::Null) => true,Some(Value::Array(a)) => a.is_empty(),Some(_) => false,};- if must_not_empty {- let mut candidates: Vec<&Value> = Vec::new();- for key in ["must", "should"] {- match b.get(key) {- Some(Value::Array(a)) => candidates.extend(a.iter()),- Some(Value::Null) | None => {}- Some(other) => candidates.push(other),- }- }- if candidates.len() == 1 {- dispatching_vector_fields(candidates[0], out);- }+ if must_not_empty && scoring.len() == 1 {+ dispatching_vector_fields(scoring[0], out);}}}@@ -9558,61 +9596,25 @@ async fn search_impl(};let merged_query = if let Some(ref existing_q) = body.query {// ES 8.x canonical hybrid: a top-level `knn` block beside a- // `query`. This used to fold to `{"bool":{"should":[query, knn]}}`- // — and the engine never dispatched it to the vector path, because- // `peel_knn_query` only peels a bool holding exactly ONE clause.- // Two clauses, so it fell through to the generic lexical path and- // the kNN half contributed NOTHING: the request answered 200 with- // the purely lexical result set, `_shards.failed: 0`, no warning- // (#395). Measured on a 3-document index: `query` alone returned- // documents 0 and 2, `knn` alone returned 0, 1 and 2, and the two- // together returned exactly 0 and 2 — the lexical answer wearing a- // hybrid request's clothes, which is a wire-compat divergence from- // ES 8.x and not merely an internal limitation.+ // `query` folds to `{"bool":{"should":[query, knn]}}` — a+ // disjunction, which is exactly ES's documented contract for the+ // shape: the knn side contributes the global top-k neighbours,+ // the hit set is the union of both halves, a document reached by+ // both scores the SUM `query_score + knn_score` (`boost`+ // weighting each side), and aggregations run over the union.//- // `hybrid` is the shape this engine actually combines (it fans each- // sub-query out and RRF-fuses the ranked lists), so it dispatches- // the kNN half correctly. BUT the hybrid executor REJECTS- // aggregations (returns 400) and ignores sort/collapse/search_after.- // Folding EVERY knn-beside-query request to hybrid would therefore- // turn a valid faceted request (knn + query + aggs) from 200 into a- // 400, and silently drop a caller's sort/collapse — trading one- // wrong answer for another. So fold to `hybrid` only when the- // request carries none of those; otherwise keep the lexical- // `bool.should`. Its kNN-dropped hits are the PRE-EXISTING behaviour- // (strictly no worse than before this fix), and it still honours- // aggs/sort/collapse and returns 200. In the hybrid case scores are- // RRF, not a BM25 sum — a visible, documented difference, still- // better than silently returning the wrong documents.- let hybrid_safe = body.aggs.is_none()- && body.aggregations.is_none()- && body.sort.is_none()- && body.collapse.is_none()- && body.search_after.is_none()- // The fusion executor also hard-drops rescore and highlight and- // ignores explain/profile (run_hybrid_with_deadline zeroes them on- // every sub-query) — keep bool.should when any is present so they- // are honoured rather than silently dropped (#458).- && body.rescore.is_none()- && body.highlight.is_none()- && !body.explain- && !body.profile- // A BM25-tuned `min_score` (e.g. 2.0) applied post-fusion to RRF- // micro-scores (~1/61) would drop EVERY hit — keep bool.should,- // where min_score compares against the BM25 sum it was tuned for.- && body.min_score.is_none();- if hybrid_safe {- json!({- "hybrid": {- "queries": [- { "query": existing_q },- { "query": knn_query_json }- ]- }- })- } else {- json!({ "bool": { "should": [existing_q, knn_query_json] } })- }+ // The engine pre-executes a `knn` clause it finds inside a+ // compound bool and pins its top-k documents into the tree+ // (#825, `pin_knn_clause` in xerj-engine), so this fold carries+ // BOTH halves through the NORMAL search path — aggs, sort,+ // collapse, search_after, rescore, highlight, explain, profile+ // and min_score all apply to the union. That retires the #458+ // `hybrid_safe` special case, which routed the no-extras case to+ // the XERJ-native `hybrid` (RRF) executor — itself a scoring+ // divergence from ES (RRF fuses ranks; ES sums raw scores) — and+ // kept the kNN-dropped lexical `bool.should` whenever any of+ // those features was present (#395's silent drop).+ json!({ "bool": { "should": [existing_q, knn_query_json] } })} else {knn_query_json};diff --git a/engine/crates/xerj-api/tests/knn_beside_query_with_aggs_stays_200.rs b/engine/crates/xerj-api/tests/knn_beside_query_with_aggs_stays_200.rsindex 20bbcf4b2..66a360c94 100644--- a/engine/crates/xerj-api/tests/knn_beside_query_with_aggs_stays_200.rs+++ b/engine/crates/xerj-api/tests/knn_beside_query_with_aggs_stays_200.rs@@ -1,10 +1,15 @@-//! Issue #458 (rework guard): the ES 8.x "faceted hybrid" shape is a top-level-//! `knn` beside a `query` AND an `aggs` block. Folding EVERY knn-beside-query-//! request to `hybrid` regressed this from 200 to 400, because the hybrid/fusion-//! executor rejects aggregations. The rework folds to `hybrid` only when the-//! request is hybrid-safe (no aggs/sort/collapse/search_after/min_score);-//! otherwise it keeps the lexical `bool.should`, which still computes the-//! aggregations and returns 200. This guards that regression.+//! Issues #458/#825: the ES 8.x hybrid shape is a top-level `knn` beside a+//! `query`, optionally with `aggs`/`sort`/`rescore`/… ES's contract for it:+//! the knn side contributes the global top-k neighbours, the hit set is the+//! UNION of both halves, a document reached by both scores the SUM+//! `query_score + knn_score`, and aggregations run over the union.+//!+//! History: the original fold to `bool.should` dropped the kNN half entirely+//! (#395); #458 routed the no-extras case to the XERJ-native `hybrid` (RRF)+//! executor and kept the kNN-dropped lexical bool whenever aggs/sort/…+//! were present; #825 retires both halves of that split — the engine now+//! pre-executes the `knn` clause and pins its top-k into the tree, so the+//! `bool.should` fold serves the full ES contract for every request shape.//!//! Elasticsearch is referenced for wire semantics only; no ES code is here.@@ -53,7 +58,12 @@ async fn json_req(}/// `knn` + `query` + `aggs` (faceted hybrid) must return 200 with its facets —-/// not the 400 an unconditional hybrid fold produced.+/// not the 400 an unconditional hybrid fold produced — AND (#825) the kNN half+/// must actually contribute: a document reachable only through the vector side+/// appears in the hits, its category shows up in the aggregation buckets+/// (ES computes aggs over the combined knn+query match set), and the two+/// lexical docs are separated by their vector scores instead of answering+/// with identical pure-BM25 scores.#[tokio::test]async fn knn_beside_query_with_aggs_stays_200_with_facets() {let (app, _dir) = app().await;@@ -71,9 +81,14 @@ async fn knn_beside_query_with_aggs_stays_200_with_facets() {.await;assert!(st.is_success(), "create index: {st} {b}");+ // Docs 1 and 2 match the lexical query ("alpha", identical tf and field+ // length, so identical BM25); doc 3 shares no term with it and is+ // reachable ONLY through the vector side (its vector IS the query+ // vector). knn k=3 puts all three in the vector top-k.for (id, body, vec, cat) in [("1", "alpha beta", [0.1_f32, 0.2, 0.3], "x"),("2", "alpha gamma", [0.2_f32, 0.1, 0.4], "y"),+ ("3", "zzz qqq", [0.1_f32, 0.2, 0.3], "z"),] {let (st, b) = json_req(&app,@@ -113,18 +128,64 @@ async fn knn_beside_query_with_aggs_stays_200_with_facets() {buckets.is_some_and(|b| !b.is_empty()),"the aggregation must be computed and returned, not dropped: {body}");+ // #825: aggs run over the UNION of both halves — the vector-only doc's+ // category must be bucketed even though the lexical query never+ // matches it.+ let bucket_keys: Vec<String> = buckets+ .into_iter()+ .flatten()+ .filter_map(|b| b.get("key").and_then(Value::as_str).map(String::from))+ .collect();+ assert!(+ bucket_keys.iter().any(|k| k == "z"),+ "#825: the aggregation must cover the vector-only document (cat 'z') — \+ aggs are computed over the combined knn+query match set, not the \+ lexical half alone: buckets={bucket_keys:?} body={body}"+ );+ let hits: Vec<(String, f64)> = body+ .pointer("/hits/hits")+ .and_then(Value::as_array)+ .map(|hits| {+ hits.iter()+ .filter_map(|h| {+ h.get("_id")+ .and_then(Value::as_str)+ .map(String::from)+ .zip(h.get("_score").and_then(Value::as_f64))+ })+ .collect()+ })+ .unwrap_or_default();+ assert!(+ hits.iter().any(|(id, _)| id == "3"),+ "#825: the vector-only document must be in the hit union: {hits:?} {body}"+ );+ // Doc 1's vector IS the query vector while doc 2's is not; their BM25+ // halves are identical. With the kNN half contributing, doc 1 must+ // outrank doc 2 — under the old silent drop both answered the same+ // pure-BM25 score.+ let score = |wanted: &str| {+ hits.iter()+ .filter(|(id, _)| id == wanted)+ .map(|(_, s)| *s)+ .next()+ };+ let (s1, s2) = (score("1"), score("2"));assert!(- body.pointer("/hits/hits")- .and_then(Value::as_array)- .is_some_and(|h| !h.is_empty()),- "hits must still be returned alongside the aggregation: {body}"+ s1.zip(s2).is_some_and(|(a, b)| a > b),+ "#825: the vector contribution must separate the lexically-tied docs \+ (doc 1 carries the exact query vector): s1={s1:?} s2={s2:?} {body}");}-/// The core fix: a hybrid-safe `knn` beside a `query` must return the UNION of-/// both halves — a document reachable only via the vector side must appear. The-/// old fold to a two-clause `bool.should` never dispatched the kNN half, so that-/// document was silently dropped.+/// The core contract: `knn` beside a `query` must return the UNION of both+/// halves — a document reachable only via the vector side must appear — and a+/// document reached by BOTH halves scores the SUM `query_score + knn_score`+/// (#825, ES semantics), so the lexical+vector doc outranks the vector-only+/// doc even though the latter carries the stronger vector match (its vector+/// IS the query vector). The original fold to a two-clause `bool.should` never+/// dispatched the kNN half at all; the #458 interim fix served this shape with+/// RRF rank fusion instead of ES's score sum.#[tokio::test]async fn knn_beside_query_returns_the_vector_only_document() {let (app, _dir) = app().await;@@ -188,15 +249,27 @@ async fn knn_beside_query_returns_the_vector_only_document() {ids.iter().any(|id| id == "lex"),"the lexical match must still be present: hits={ids:?} body={body}");+ // #825: summed scores, not rank fusion. Both docs are in the vector+ // top-k (k=2 of 2), so "lex" scores BM25 + its (weaker) vector score+ // while "vec" scores its (perfect) vector match alone — the sum must+ // put "lex" first. Under RRF both halves' ranks fused to ~1/61-scale+ // micro-scores instead of the ES score sum.+ assert_eq!(+ ids.first().map(String::as_str),+ Some("lex"),+ "#825: the doc matching both halves must outrank the vector-only doc \+ (score = query_score + knn_score): hits={ids:?} body={body}"+ );}-/// A hybrid-incompatible feature (here `rescore`, which the fusion executor-/// silently drops) must keep the request on the lexical `bool.should` path where-/// rescore is honoured — NOT route it to `hybrid`. Observable proxy: the-/// vector-only document that ONLY the hybrid path surfaces must be ABSENT, so a-/// present-`rescore` request is provably not on the hybrid branch.+/// #825: a feature the fusion executor could not serve (here `rescore`) no+/// longer costs the request its vector half. The engine pins the knn top-k+/// into the `bool.should` tree, so the vector-only document appears in the+/// union AND rescore is honoured on the normal path. Under #458 this exact+/// shape silently dropped the kNN contribution (this test used to assert the+/// vector doc was ABSENT — that was the documented cost of keeping rescore).#[tokio::test]-async fn knn_beside_query_with_rescore_stays_on_the_lexical_path() {+async fn knn_beside_query_with_rescore_keeps_the_vector_contribution() {let (app, _dir) = app().await;let (st, b) = json_req(@@ -252,9 +325,107 @@ async fn knn_beside_query_with_rescore_stays_on_the_lexical_path() {}).unwrap_or_default();assert!(- !ids.iter().any(|id| id == "vec"),- "#458: knn+query+rescore must stay on bool.should (rescore honoured), not route to the \- hybrid executor that silently drops rescore — the hybrid-only vector doc must be \- absent: hits={ids:?} body={body}"+ ids.iter().any(|id| id == "vec"),+ "#825: knn+query+rescore must serve the UNION — the vector-only doc must appear \+ even with rescore present (the old bool.should fallback silently dropped the \+ kNN half for every rescore-bearing request): hits={ids:?} body={body}"+ );+ assert!(+ ids.iter().any(|id| id == "lex"),+ "the lexical (and rescored) match must still be present: hits={ids:?} body={body}"+ );+}++/// #825 (review follow-up): the summed-score contract must also hold when the+/// lexical half is itself a `bool` carrying two or more scoring text clauses —+/// the shape that reaches the post-scan IDF heuristic rescore, whose gate+/// `query_uses_bool_text` counts the user's inner bool and is blind to the+/// pinned kNN sub-tree. That pass rewrites `hit.score` from term frequencies+/// alone, so run blind it discards the vector contribution for exactly the+/// documents reached by BOTH halves — the same silent drop #825 exists to+/// close, one layer down. A bare `match`/`term` query half returns+/// text_children = 1 and never reaches it, which is why every other test here+/// misses this.+///+/// The two documents carry identical lexical content (identical tf and field+/// lengths, so identical BM25 *and* identical heuristic scores) and differ+/// only in their vector: the one whose vector IS the query vector must rank+/// strictly above the other. `near` is indexed SECOND, so if the scores+/// collapsed to a tie the arrival-order tie-break would put it last.+#[tokio::test]+async fn knn_beside_bool_query_keeps_the_vector_contribution() {+ let (app, _dir) = app().await;++ let (st, b) = json_req(+ &app,+ "PUT",+ "/bq",+ json!({ "mappings": { "properties": {+ "title": { "type": "text" },+ "body": { "type": "text" },+ "v": { "type": "dense_vector", "dims": 3 }+ } } }),+ )+ .await;+ assert!(st.is_success(), "create index: {st} {b}");++ for (id, vec) in [("far", [1.0_f32, 0.0, 0.0]), ("near", [0.1_f32, 0.2, 0.3])] {+ let (st, b) = json_req(+ &app,+ "POST",+ &format!("/bq/_doc/{id}"),+ json!({ "title": "alpha", "body": "beta", "v": vec }),+ )+ .await;+ assert!(st.is_success(), "index {id}: {st} {b}");+ }+ let (_s, _b) = json_req(&app, "POST", "/bq/_refresh", json!({})).await;++ let (status, body) = json_req(+ &app,+ "POST",+ "/bq/_search",+ json!({+ "query": { "bool": { "must": [+ { "match": { "title": "alpha" } },+ { "match": { "body": "beta" } }+ ] } },+ "knn": { "field": "v", "query_vector": [0.1, 0.2, 0.3], "k": 2, "num_candidates": 10 }+ }),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "_search status: {status} {body}");++ let hits: Vec<(String, f64)> = body+ .pointer("/hits/hits")+ .and_then(Value::as_array)+ .map(|hits| {+ hits.iter()+ .filter_map(|h| {+ h.get("_id")+ .and_then(Value::as_str)+ .map(String::from)+ .zip(h.get("_score").and_then(Value::as_f64))+ })+ .collect()+ })+ .unwrap_or_default();+ let score = |wanted: &str| {+ hits.iter()+ .filter(|(id, _)| id == wanted)+ .map(|(_, s)| *s)+ .next()+ };+ let (near, far) = (score("near"), score("far"));+ assert!(+ near.zip(far).is_some_and(|(n, f)| n > f),+ "#825: with a multi-clause bool as the lexical half the kNN score must still be \+ summed in — the post-scan IDF rescore must not overwrite the pinned vector \+ contribution: near={near:?} far={far:?} {body}"+ );+ assert_eq!(+ hits.first().map(|(id, _)| id.as_str()),+ Some("near"),+ "#825: the better vector match must lead the page: hits={hits:?} {body}");}diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 5a2eaa733..ab16f910a 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -14237,6 +14237,158 @@ impl Index {})}+ /// #825: execute one `Knn` clause standalone and return a replacement+ /// sub-tree that pins its top-k results as per-document constant scores:+ /// `bool.should = [Constant{score, Ids{[id]}}, …]` (or `MatchNone` when+ /// the leg returns nothing). The generic matcher/scorer has no `Knn`+ /// arm, so this is what lets a `knn` clause inside a compound `bool`+ /// contribute at all — and it reproduces ES's hybrid contract exactly:+ /// the knn side contributes the GLOBAL top-k neighbours, a document in+ /// both halves scores the SUM (Bool sums clause scores), and every+ /// request feature (aggs over the union, sort, collapse, highlight,+ /// rescore, min_score) applies through the normal path afterwards.+ ///+ /// The leg runs on the SAME executors as the single-clause peel in+ /// `search_inner`: HNSW only for the plain case (exact-rescored scores,+ /// but `aggs` on the OUTER request forces exact brute force — the rc.6+ /// rule that ANN approximation must not leak into agg buckets), brute+ /// force otherwise. Boost and the raw-similarity cutoff are applied by+ /// those executors before the scores reach the pinned constants.+ async fn pin_knn_clause(+ &self,+ request: &SearchRequest,+ deadline: std::time::Instant,+ knn: &QueryNode,+ ) -> Result<QueryNode> {+ if let QueryNode::Knn {+ field,+ vector,+ k,+ num_candidates,+ filter,+ boost,+ similarity: min_similarity,+ } = knn+ {+ let similarity = {+ let schema = self.schema.read().await;+ lookup_vector_similarity(&schema.schema, field)+ };+ // The leg needs the FULL top-k regardless of the caller's page:+ // ES unions the global k nearest into the candidate set even when+ // `size` is smaller, so pagination happens after the merge.+ let sub_request = SearchRequest {+ query: knn.clone(),+ from: 0,+ size: *k,+ sort: Vec::new(),+ search_after: None,+ aggs: None,+ explain: false,+ source: request.source.clone(),+ timeout_ms: request.timeout_ms,+ highlight: None,+ track_total_hits: request.track_total_hits,+ script_fields: None,+ fields: Vec::new(),+ profile: false,+ collapse: None,+ rescore: Vec::new(),+ min_score: None,+ leaf_ts_field: None,+ savings: request.savings,+ };+ let plain = filter.is_none()+ && boost.is_none()+ && min_similarity.is_none()+ && request.aggs.is_none();+ let hnsw = if plain {+ self.run_knn_hnsw(+ &sub_request,+ deadline,+ field,+ vector,+ *k,+ *num_candidates,+ &similarity,+ )+ .await+ } else {+ None+ };+ let leg = match hnsw {+ Some(result) => result,+ None => {+ self.run_knn_brute_force_with_deadline(+ &sub_request,+ deadline,+ field,+ vector,+ *k,+ filter.clone(),+ &similarity,+ *boost,+ *min_similarity,+ )+ .await?+ }+ };+ // Same execution-truth check as the single-knn and multi-knn+ // arms (#498): an empty leg on an unanswerable field must fail+ // loud, not degrade to lexical-only.+ if leg.hits.is_empty() && !leg.timed_out {+ self.reject_unanswerable_knn_field(field).await?;+ }+ let pinned: Vec<QueryNode> = leg+ .hits+ .iter()+ .map(|hit| QueryNode::Constant {+ score: hit.score,+ query: Box::new(QueryNode::Ids {+ values: vec![hit.id.clone()],+ }),+ })+ .collect();+ Ok(if pinned.is_empty() {+ QueryNode::MatchNone+ } else {+ // The `filter` is a REDUNDANT accelerator, not semantics: it+ // holds exactly the union of the `should` clauses' ids, so+ // `filter ∧ (≥1 should)` and `(≥1 should)` have the same match+ // set, and `filter` contributes nothing to `_score` (only+ // must/should are summed — `BooleanClause.isScoring()`).+ // What it buys is the shape of the DOC-SCAN cost. The scan+ // evaluates must/must_not/filter BEFORE should, so a document+ // outside the top-k is now rejected by ONE `_id` lookup plus a+ // memcmp sweep, instead of recursing into all k+ // `Constant{Ids}` clauses and re-reading `_id` from the source+ // map k times. Measured on a 100 k-doc index, together with+ // the `should` short-circuit in `doc_matches_query_typed`+ // (see the PR's measurement table): hybrid k=1000+ // 2309 ms → 385 ms, k=10000 24 437 ms → 5 071 ms.+ // `minimum_should_match: 1` is written out rather than left+ // implicit: with a non-empty `filter` the default would fall+ // to 0 ("filter alone decides"), which is the same match set+ // here but is NOT the same statement, and any other reader of+ // this tree should see the disjunction that was meant.+ QueryNode::Bool {+ must: Vec::new(),+ should: pinned,+ filter: vec![QueryNode::Ids {+ values: leg.hits.iter().map(|hit| hit.id.clone()).collect(),+ }],+ must_not: Vec::new(),+ minimum_should_match: Some(MinShouldMatch::Fixed(1)),+ }+ })+ } else {+ // Caller contract: `knn` is a `QueryNode::Knn` (see+ // `compound_bool_direct_knn`). Anything else passes through+ // untouched rather than panicking.+ Ok(knn.clone())+ }+ }+/// `field` is the full dotted path as the ES query provides it/// (e.g. `nested.vector`). The element inside the nested array is/// the sub-field after the path prefix (`vector`).@@ -16409,6 +16561,36 @@ impl Index {.await;}+ // ── #825: `knn` clause inside a compound `bool` ────────────────────────+ // Every vector-aware short-circuit above has declined by now, so a+ // `Knn` node still in the tree is about to hit the generic path,+ // where it matches nothing and scores 0.0 (silent drop). The ES 8.x+ // canonical hybrid — a top-level `knn` beside a `query`, folded by+ // the compat layer into exactly this `bool.should` — lands here for+ // EVERY request since the #458 hybrid_safe fold was retired, and+ // previously landed here whenever aggs/sort/collapse/rescore/… were+ // present. Pre-execute the knn leg and splice its top-k back in as+ // pinned per-document constants; the generic path then serves the+ // union with summed scores and full feature support (see+ // `pin_knn_clause`).+ //+ // `pinned_knn_scores` is the id → knn-score map of the clauses we+ // just spliced in. Everything downstream that RE-WRITES `_score`+ // rather than reading it has to add the vector half back, or the+ // sum contract is lost exactly where it was hardest to see — see+ // the IDF heuristic rescore further down.+ let mut pinned_knn_scores: HashMap<String, f32> = HashMap::new();+ let pinned_query = if let Some(knn_clause) = compound_bool_direct_knn(query) {+ let pinned = self+ .pin_knn_clause(request, search_deadline, knn_clause)+ .await?;+ collect_pinned_knn_scores(&pinned, &mut pinned_knn_scores);+ Some(replace_direct_knn_with_pinned(query, &pinned))+ } else {+ None+ };+ let query = pinned_query.as_ref().unwrap_or(query);+// ── Max result window enforcement ──────────────────────────────────────// Default max_result_window is 10,000 (matches ES default).// Allow override via index setting `index.max_result_window`.@@ -19908,7 +20090,27 @@ impl Index {}}if score > 0.0 {- hit.score = score;+ // #825: this pass derives a score from term+ // frequencies ALONE, so on a #825-pinned tree it+ // would overwrite the summed `query_score ++ // knn_score` with the lexical half only — silently+ // dropping the vector contribution for exactly the+ // documents reached by BOTH halves, while a+ // vector-only doc (tf 0, heuristic 0) keeps its knn+ // score and can then outrank them. `query_uses_bool_text`+ // above cannot see the difference: the user's inner+ // bool supplies the ≥2 text clauses and the pinned+ // sub-tree contributes 0 text children, so it neither+ // disqualifies nor suppresses. Add the pinned constant+ // back rather than suppressing the rescore, so the+ // lexical half still gets its IDF weighting.+ // (Empty map — the overwhelmingly common case — costs+ // one hash lookup that always misses.)+ hit.score = score+ + pinned_knn_scores+ .get(hit.id.as_str())+ .copied()+ .unwrap_or(0.0);}}// Re-sort by the new scores (#270 — full page-order key, so@@ -33826,6 +34028,106 @@ fn peel_multi_knn_query(q: &QueryNode) -> Option<Vec<PeeledKnn>> {}}+/// #825: locate the single direct `Knn` clause inside a compound `bool`.+///+/// `peel_knn_query` serves the one-scoring-clause wrapper shapes; a `Knn`+/// beside OTHER clauses falls through to the generic path, whose+/// matcher/scorer has no `Knn` arm (`doc_matches_query_typed` answers+/// `false`, `score_query_against_doc` contributes `0.0`) — so the ES-compat+/// fold of a top-level `knn` beside a `query` silently returned the lexical+/// half alone whenever the request was not routed elsewhere (#395/#458).+///+/// Returns the clause only when the bool carries EXACTLY ONE direct `Knn`+/// child (in `must` or `should`). Several sibling `Knn` clauses would each+/// need their own pinned executor pass, and no supported request shape+/// produces that tree: the ES-compat layer answers 400 to a `knn` array+/// beside a `query` before folding, and the pure `knn` array form is served+/// by `peel_multi_knn_query` above.+///+/// Everything else keeps the pre-existing (dropped) behaviour: a `Knn`+/// nested deeper than a direct child, AND a `Knn` sitting directly in+/// `filter` or `must_not` — only the SCORING lists are scanned here, and+/// `replace_direct_knn_with_pinned` likewise only swaps those two. That is+/// exact rather than tidy: `filter`/`must_not` are the non-scoring lists+/// (`BooleanClause.isScoring()` is `MUST || SHOULD`), a pinned constant+/// there would contribute nothing anyway, and no supported request shape+/// produces such a tree — the ES-compat fold only ever emits+/// `bool.should[query, knn]`.+fn compound_bool_direct_knn(q: &QueryNode) -> Option<&QueryNode> {+ if let QueryNode::Bool { must, should, .. } = q {+ let mut knns = must+ .iter()+ .chain(should.iter())+ .filter(|clause| matches!(clause, QueryNode::Knn { .. }));+ let first = knns.next();+ let surplus = knns.next();+ first.filter(|_| surplus.is_none())+ } else {+ None+ }+}++/// #825: read the id → knn-score map back out of the sub-tree+/// `pin_knn_clause` just produced (`Bool{should:[Constant{score,+/// Ids{[id]}}, …]}`, or `MatchNone` for an empty leg).+///+/// Deliberately scoped to that exact shape and called ONLY on+/// `pin_knn_clause`'s own output, never on the user's tree: a+/// user-supplied `constant_score{filter:{ids}}` is an ordinary scoring+/// clause whose interaction with the heuristic rescore is pre-existing+/// behaviour, and folding it in here would change it.+fn collect_pinned_knn_scores(pinned: &QueryNode, out: &mut HashMap<String, f32>) {+ if let QueryNode::Bool { should, .. } = pinned {+ for clause in should {+ if let QueryNode::Constant { score, query } = clause {+ if let QueryNode::Ids { values } = query.as_ref() {+ for id in values {+ // A document can be pinned once only (the leg+ // returns distinct ids), so `insert` is total.+ out.insert(id.clone(), *score);+ }+ }+ }+ }+ }+}++/// #825: rebuild the compound bool with its direct `Knn` clause(s) swapped+/// for the pre-executed pinned sub-tree. Pure shape surgery — every other+/// clause, the filters, and `minimum_should_match` pass through unchanged.+fn replace_direct_knn_with_pinned(q: &QueryNode, pinned: &QueryNode) -> QueryNode {+ if let QueryNode::Bool {+ must,+ should,+ filter,+ must_not,+ minimum_should_match,+ } = q+ {+ let swap = |clauses: &[QueryNode]| -> Vec<QueryNode> {+ clauses+ .iter()+ .map(|clause| {+ if matches!(clause, QueryNode::Knn { .. }) {+ pinned.clone()+ } else {+ clause.clone()+ }+ })+ .collect()+ };+ QueryNode::Bool {+ must: swap(must),+ should: swap(should),+ filter: filter.clone(),+ must_not: must_not.clone(),+ minimum_should_match: minimum_should_match.clone(),+ }+ } else {+ q.clone()+ }+}+// Peel a nested-KNN query and partition surrounding clauses into// pre-filters (applied BEFORE top-k ranking — align with ES `knn.filter`// and `bool.filter`) and post-filters (applied AFTER top-k — align with@@ -38552,11 +38854,28 @@ fn doc_matches_query_typed(q: &QueryNode, source: &Value, schema: &Schema) -> bo}};if min > 0 {- let matched = should- .iter()- .filter(|q| doc_matches_query_typed(q, source, schema))- .count();- if matched < min {+ // Stop as soon as the requirement is met. The old+ // `.filter(…).count()` evaluated EVERY should-clause on+ // EVERY document even when the first one already settled+ // it — O(clauses) per doc with no upside. That is free+ // ordinarily and expensive on a #825-pinned tree, whose+ // should-list is one clause per pinned kNN neighbour:+ // a document the lexical clause already matched no longer+ // walks the pinned sub-tree at all. Semantics are+ // unchanged — `matched >= min` is the same predicate,+ // reached without counting past `min`.+ let mut matched = 0usize;+ let mut satisfied = false;+ for q in should {+ if doc_matches_query_typed(q, source, schema) {+ matched += 1;+ if matched >= min {+ satisfied = true;+ break;+ }+ }+ }+ if !satisfied {return false;}}diff --git a/engine/tests/es-compat-yaml/yaml/vectors/96_knn_beside_query_aggs.yml b/engine/tests/es-compat-yaml/yaml/vectors/96_knn_beside_query_aggs.ymlnew file mode 100644index 000000000..5e2039f0a--- /dev/null+++ b/engine/tests/es-compat-yaml/yaml/vectors/96_knn_beside_query_aggs.yml@@ -0,0 +1,82 @@+setup:++ - do:+ indices.create:+ index: knn_union_test+ body:+ settings:+ number_of_shards: 1+ mappings:+ properties:+ title:+ type: text+ category:+ type: keyword+ vector:+ type: dense_vector+ dims: 3+ index: true+ similarity: cosine++ - do:+ bulk:+ refresh: true+ body:+ - '{ "index": { "_index": "knn_union_test", "_id": "1" } }'+ - '{ "title": "quick brown fox", "category": "a", "vector": [0.9, 0.1, 0.0] }'+ - '{ "index": { "_index": "knn_union_test", "_id": "2" } }'+ - '{ "title": "quick brown dog", "category": "a", "vector": [0.0, 1.0, 0.0] }'+ - '{ "index": { "_index": "knn_union_test", "_id": "3" } }'+ - '{ "title": "zzz qqq", "category": "z", "vector": [1.0, 0.0, 0.0] }'++---+"knn beside query with aggs returns the union and buckets over it":++ # Issue #825: the knn contribution must not be dropped when aggs are+ # present. The hit set is the union of both halves (ES 8.x semantics)+ # and the aggregations run over that union.+ - do:+ search:+ rest_total_hits_as_int: true+ index: knn_union_test+ body:+ size: 10+ query:+ match:+ title: "quick"+ knn:+ field: vector+ query_vector: [1.0, 0.0, 0.0]+ k: 2+ num_candidates: 10+ aggs:+ by_category:+ terms:+ field: category++ - match: { hits.total: 3 }+ - length: { aggregations.by_category.buckets: 2 }+ - match: { aggregations.by_category.buckets.0.key: "a" }+ - match: { aggregations.by_category.buckets.0.doc_count: 2 }+ - match: { aggregations.by_category.buckets.1.key: "z" }+ - match: { aggregations.by_category.buckets.1.doc_count: 1 }++---+"knn beside query without aggs is the same union":++ - do:+ search:+ rest_total_hits_as_int: true+ index: knn_union_test+ body:+ size: 10+ query:+ match:+ title: "quick"+ knn:+ field: vector+ query_vector: [1.0, 0.0, 0.0]+ k: 2+ num_candidates: 10++ - match: { hits.total: 3 }