Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

#825 knn beside query scores as a sum over the union, aggs included (#879)

Onyeka Obid2f68019fc8f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 063714f4e..b6f6feb0d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **A top-level `knn` beside a `query` now scores as a sum over the union of
+ both halves, aggregations included**
+ ([#825](https://github.com/xerj-org/xerj/issues/825)). ES 8.x defines this
+ shape as a disjunction: the hit set is the union of the lexical matches and
+ the global top-`k` neighbours, each document scores `query_score +
+ knn_score`, and aggregations are computed over that union. XERJ folded the
+ request to `bool.should[query, knn]` but the generic matcher/scorer has no
+ `knn` arm, so a vector-only document vanished and a lexical match lost its
+ vector score — silently, with `_shards.failed: 0`. The
+ [#458](https://github.com/xerj-org/xerj/issues/458) carve-out covered one
+ narrow shape with RRF rank fusion (itself a scoring divergence from ES) and
+ left every request carrying `aggs`, `sort`, `collapse` or `rescore` on the
+ silent drop. The engine now pre-executes the `knn` leg and pins its top-`k`
+ into the tree as per-document constant scores, so one path serves the union
+ with summed scores and every request feature applies to it.
+
+ Two behaviour changes worth knowing before you upgrade:
+
+ - `{query, knn:{field: <not a vector field>}, aggs}` now answers **400**
+ (`the field is not answerable by knn`), matching ES and the existing
+ [#498](https://github.com/xerj-org/xerj/issues/498) rule. It previously
+ answered 200 with lexical-only hits — the wrong answer, quietly.
+ - **This shape is slower.** A pinned tree contains `ids` clauses, which the
+ FTS projection cannot lift, so the request is answered by a stored-document
+ scan instead of the inverted index. Measured on 100 000 documents (`text` +
+ 8-dim `dense_vector`, ~10 % lexical selectivity, closed-loop, fresh query
+ vector per request): `query` + `knn` k=10 goes from 2.5 ms (the old RRF
+ route, which returned ES-divergent scores) to ~208 ms; k=1000 to ~385 ms;
+ k=10000 to ~5.1 s. The shapes that previously carried the silent drop —
+ anything with `aggs`/`sort`/`collapse`/`rescore` — already took the scan
+ and cost about a quarter more (476 ms → 605 ms with a terms agg),
+ but they now return the right documents. Restoring an indexed route is
+ tracked as [#892](https://github.com/xerj-org/xerj/issues/892);
+ correctness landed first.
+
- **`match_phrase` `slop` now admits transposed terms at Lucene's cost of 2**
([#830](https://github.com/xerj-org/xerj/issues/830)). The sloppy-phrase
walk was in-order only — each next term matched strictly after the
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index e68bbc6cb..396eab9c5 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -7486,16 +7486,25 @@ fn vector_query_fields(q: &Value, out: &mut Vec<String>) {
}
/// Fields whose vector clause reaches the vector for this query, approximating
-/// the engine's `peel_semantic_query` / `peel_knn_query` dispatch (index.rs) for
-/// the shapes that can co-exist with a lexical `semantic_text` clause. A
-/// `semantic`/`knn` clause consults the vector only as the whole query, as the
-/// SOLE `must`/`should` candidate of a `bool` with no `must_not`, or inside a
-/// `hybrid` (which fuses and runs every branch). A `semantic`/`knn` clause
-/// sitting BESIDE a sibling in a multi-clause `bool` falls through to the lexical
-/// path and never consults the vector (#394) — so it must NOT count as "reached
-/// the vector" here, or the hint gets suppressed on a query that ran BM25-only,
+/// the engine's `peel_semantic_query` / `peel_knn_query` /
+/// `compound_bool_direct_knn` dispatch (index.rs) for the shapes that can
+/// co-exist with a lexical `semantic_text` clause.
+///
+/// A `semantic` clause consults the vector only as the whole query, as the SOLE
+/// `must`/`should` candidate of a `bool` with no `must_not`, or inside a
+/// `hybrid` (which fuses and runs every branch). A `semantic` clause sitting
+/// BESIDE a sibling in a multi-clause `bool` falls through to the lexical path
+/// and never consults the vector (#394) — so it must NOT count as "reached the
+/// vector" here, or the hint gets suppressed on a query that ran BM25-only,
/// reintroducing the exact silence the hint exists to break.
///
+/// A `knn` clause is no longer bound by that (#825). A `knn` that is a DIRECT
+/// child of a compound bool's scoring lists is pre-executed by the engine and
+/// its top-k pinned into the tree, so it dispatches whatever its siblings are —
+/// and that is precisely the tree this module's `knn`-beside-`query` fold
+/// emits, so the hand-rolled ES hybrid must not be nagged for running BM25
+/// "only".
+///
/// This is NOT a byte-exact mirror of `peel`, and does not need to be (#777):
/// where it diverges the shapes are contrived or carry no lexical clause, so none
/// can produce an observable wrong hint — a pure multi-`knn` or `nested` bool has
@@ -7506,6 +7515,14 @@ fn vector_query_fields(q: &Value, out: &mut Vec<String>) {
/// `boost`/`_name` is an inline leaf param, not a wrapper key, so nothing
/// dispatching is missed.
///
+/// The `knn` arm inherits exactly that one known divergence and no other. A
+/// one-clause `bool` is erased before the engine reads the tree (#399), so
+/// descending through a sole candidate into a compound bool tracks
+/// `compound_bool_direct_knn` faithfully — except in the same contrived
+/// `bool{should:[bool{should:[knn, match]}], filter:[…]}` shape, where the
+/// wrapper is not erased, the engine does NOT pin, and this over-suppresses.
+/// Same class, same contrivance, no new one.
+///
/// Contrast [`vector_query_fields`], which collects every mention regardless of
/// shape and is right only for the top-level `knn` block (which always dispatches).
fn dispatching_vector_fields(q: &Value, out: &mut Vec<String>) {
@@ -7539,27 +7556,48 @@ fn dispatching_vector_fields(q: &Value, out: &mut Vec<String>) {
if let Some(inner) = o.get("constant_score").and_then(|c| c.get("filter")) {
dispatching_vector_fields(inner, out);
}
- // `bool` reaches its vector clause ONLY in the single-`must`/`should`
+ // `bool` reaches a `semantic` clause ONLY in the single-`must`/`should`
// candidate, no-`must_not` shape the engine peels; every other bool falls
// through to the lexical path and consults no vector.
+ //
+ // `knn` is no longer bound by that (#825): a `knn` clause sitting DIRECTLY
+ // in a compound bool's SCORING lists is now pre-executed by the engine and
+ // its top-k pinned into the tree (`compound_bool_direct_knn` /
+ // `pin_knn_clause` in xerj-engine), so it genuinely dispatches however many
+ // siblings it has — which is exactly the tree this module's own
+ // `knn`-beside-`query` fold emits. Mirror the engine's gate: exactly one
+ // direct `Knn` child across `must` + `should`, `must_not`/`filter`
+ // irrelevant. `semantic` keeps the #394 rule, which still holds for it.
if let Some(Value::Object(b)) = o.get("bool") {
+ let mut scoring: Vec<&Value> = Vec::new();
+ for key in ["must", "should"] {
+ match b.get(key) {
+ Some(Value::Array(a)) => scoring.extend(a.iter()),
+ Some(Value::Null) | None => {}
+ Some(other) => scoring.push(other),
+ }
+ }
+ let mut direct_knn = scoring.iter().filter(|c| c.get("knn").is_some());
+ if let (Some(clause), None) = (direct_knn.next(), direct_knn.next()) {
+ if let Some(spec) = clause.get("knn") {
+ let specs: Vec<&Value> = match spec {
+ Value::Array(a) => a.iter().collect(),
+ single => vec![single],
+ };
+ for spec in specs {
+ if let Some(f) = spec.get("field").and_then(Value::as_str) {
+ out.push(f.to_string());
+ }
+ }
+ }
+ }
let must_not_empty = match b.get("must_not") {
None | Some(Value::Null) => true,
Some(Value::Array(a)) => a.is_empty(),
Some(_) => false,
};
- if must_not_empty {
- let mut candidates: Vec<&Value> = Vec::new();
- for key in ["must", "should"] {
- match b.get(key) {
- Some(Value::Array(a)) => candidates.extend(a.iter()),
- Some(Value::Null) | None => {}
- Some(other) => candidates.push(other),
- }
- }
- if candidates.len() == 1 {
- dispatching_vector_fields(candidates[0], out);
- }
+ if must_not_empty && scoring.len() == 1 {
+ dispatching_vector_fields(scoring[0], out);
}
}
}
@@ -9558,61 +9596,25 @@ async fn search_impl(
};
let merged_query = if let Some(ref existing_q) = body.query {
// ES 8.x canonical hybrid: a top-level `knn` block beside a
- // `query`. This used to fold to `{"bool":{"should":[query, knn]}}`
- // — and the engine never dispatched it to the vector path, because
- // `peel_knn_query` only peels a bool holding exactly ONE clause.
- // Two clauses, so it fell through to the generic lexical path and
- // the kNN half contributed NOTHING: the request answered 200 with
- // the purely lexical result set, `_shards.failed: 0`, no warning
- // (#395). Measured on a 3-document index: `query` alone returned
- // documents 0 and 2, `knn` alone returned 0, 1 and 2, and the two
- // together returned exactly 0 and 2 — the lexical answer wearing a
- // hybrid request's clothes, which is a wire-compat divergence from
- // ES 8.x and not merely an internal limitation.
+ // `query` folds to `{"bool":{"should":[query, knn]}}` — a
+ // disjunction, which is exactly ES's documented contract for the
+ // shape: the knn side contributes the global top-k neighbours,
+ // the hit set is the union of both halves, a document reached by
+ // both scores the SUM `query_score + knn_score` (`boost`
+ // weighting each side), and aggregations run over the union.
//
- // `hybrid` is the shape this engine actually combines (it fans each
- // sub-query out and RRF-fuses the ranked lists), so it dispatches
- // the kNN half correctly. BUT the hybrid executor REJECTS
- // aggregations (returns 400) and ignores sort/collapse/search_after.
- // Folding EVERY knn-beside-query request to hybrid would therefore
- // turn a valid faceted request (knn + query + aggs) from 200 into a
- // 400, and silently drop a caller's sort/collapse — trading one
- // wrong answer for another. So fold to `hybrid` only when the
- // request carries none of those; otherwise keep the lexical
- // `bool.should`. Its kNN-dropped hits are the PRE-EXISTING behaviour
- // (strictly no worse than before this fix), and it still honours
- // aggs/sort/collapse and returns 200. In the hybrid case scores are
- // RRF, not a BM25 sum — a visible, documented difference, still
- // better than silently returning the wrong documents.
- let hybrid_safe = body.aggs.is_none()
- && body.aggregations.is_none()
- && body.sort.is_none()
- && body.collapse.is_none()
- && body.search_after.is_none()
- // The fusion executor also hard-drops rescore and highlight and
- // ignores explain/profile (run_hybrid_with_deadline zeroes them on
- // every sub-query) — keep bool.should when any is present so they
- // are honoured rather than silently dropped (#458).
- && body.rescore.is_none()
- && body.highlight.is_none()
- && !body.explain
- && !body.profile
- // A BM25-tuned `min_score` (e.g. 2.0) applied post-fusion to RRF
- // micro-scores (~1/61) would drop EVERY hit — keep bool.should,
- // where min_score compares against the BM25 sum it was tuned for.
- && body.min_score.is_none();
- if hybrid_safe {
- json!({
- "hybrid": {
- "queries": [
- { "query": existing_q },
- { "query": knn_query_json }
- ]
- }
- })
- } else {
- json!({ "bool": { "should": [existing_q, knn_query_json] } })
- }
+ // The engine pre-executes a `knn` clause it finds inside a
+ // compound bool and pins its top-k documents into the tree
+ // (#825, `pin_knn_clause` in xerj-engine), so this fold carries
+ // BOTH halves through the NORMAL search path — aggs, sort,
+ // collapse, search_after, rescore, highlight, explain, profile
+ // and min_score all apply to the union. That retires the #458
+ // `hybrid_safe` special case, which routed the no-extras case to
+ // the XERJ-native `hybrid` (RRF) executor — itself a scoring
+ // divergence from ES (RRF fuses ranks; ES sums raw scores) — and
+ // kept the kNN-dropped lexical `bool.should` whenever any of
+ // those features was present (#395's silent drop).
+ json!({ "bool": { "should": [existing_q, knn_query_json] } })
} else {
knn_query_json
};
diff --git a/engine/crates/xerj-api/tests/knn_beside_query_with_aggs_stays_200.rs b/engine/crates/xerj-api/tests/knn_beside_query_with_aggs_stays_200.rs
index 20bbcf4b2..66a360c94 100644
--- a/engine/crates/xerj-api/tests/knn_beside_query_with_aggs_stays_200.rs
+++ b/engine/crates/xerj-api/tests/knn_beside_query_with_aggs_stays_200.rs
@@ -1,10 +1,15 @@
-//! Issue #458 (rework guard): the ES 8.x "faceted hybrid" shape is a top-level
-//! `knn` beside a `query` AND an `aggs` block. Folding EVERY knn-beside-query
-//! request to `hybrid` regressed this from 200 to 400, because the hybrid/fusion
-//! executor rejects aggregations. The rework folds to `hybrid` only when the
-//! request is hybrid-safe (no aggs/sort/collapse/search_after/min_score);
-//! otherwise it keeps the lexical `bool.should`, which still computes the
-//! aggregations and returns 200. This guards that regression.
+//! Issues #458/#825: the ES 8.x hybrid shape is a top-level `knn` beside a
+//! `query`, optionally with `aggs`/`sort`/`rescore`/… ES's contract for it:
+//! the knn side contributes the global top-k neighbours, the hit set is the
+//! UNION of both halves, a document reached by both scores the SUM
+//! `query_score + knn_score`, and aggregations run over the union.
+//!
+//! History: the original fold to `bool.should` dropped the kNN half entirely
+//! (#395); #458 routed the no-extras case to the XERJ-native `hybrid` (RRF)
+//! executor and kept the kNN-dropped lexical bool whenever aggs/sort/…
+//! were present; #825 retires both halves of that split — the engine now
+//! pre-executes the `knn` clause and pins its top-k into the tree, so the
+//! `bool.should` fold serves the full ES contract for every request shape.
//!
//! Elasticsearch is referenced for wire semantics only; no ES code is here.
@@ -53,7 +58,12 @@ async fn json_req(
}
/// `knn` + `query` + `aggs` (faceted hybrid) must return 200 with its facets —
-/// not the 400 an unconditional hybrid fold produced.
+/// not the 400 an unconditional hybrid fold produced — AND (#825) the kNN half
+/// must actually contribute: a document reachable only through the vector side
+/// appears in the hits, its category shows up in the aggregation buckets
+/// (ES computes aggs over the combined knn+query match set), and the two
+/// lexical docs are separated by their vector scores instead of answering
+/// with identical pure-BM25 scores.
#[tokio::test]
async fn knn_beside_query_with_aggs_stays_200_with_facets() {
let (app, _dir) = app().await;
@@ -71,9 +81,14 @@ async fn knn_beside_query_with_aggs_stays_200_with_facets() {
.await;
assert!(st.is_success(), "create index: {st} {b}");
+ // Docs 1 and 2 match the lexical query ("alpha", identical tf and field
+ // length, so identical BM25); doc 3 shares no term with it and is
+ // reachable ONLY through the vector side (its vector IS the query
+ // vector). knn k=3 puts all three in the vector top-k.
for (id, body, vec, cat) in [
("1", "alpha beta", [0.1_f32, 0.2, 0.3], "x"),
("2", "alpha gamma", [0.2_f32, 0.1, 0.4], "y"),
+ ("3", "zzz qqq", [0.1_f32, 0.2, 0.3], "z"),
] {
let (st, b) = json_req(
&app,
@@ -113,18 +128,64 @@ async fn knn_beside_query_with_aggs_stays_200_with_facets() {
buckets.is_some_and(|b| !b.is_empty()),
"the aggregation must be computed and returned, not dropped: {body}"
);
+ // #825: aggs run over the UNION of both halves — the vector-only doc's
+ // category must be bucketed even though the lexical query never
+ // matches it.
+ let bucket_keys: Vec<String> = buckets
+ .into_iter()
+ .flatten()
+ .filter_map(|b| b.get("key").and_then(Value::as_str).map(String::from))
+ .collect();
+ assert!(
+ bucket_keys.iter().any(|k| k == "z"),
+ "#825: the aggregation must cover the vector-only document (cat 'z') — \
+ aggs are computed over the combined knn+query match set, not the \
+ lexical half alone: buckets={bucket_keys:?} body={body}"
+ );
+ let hits: Vec<(String, f64)> = body
+ .pointer("/hits/hits")
+ .and_then(Value::as_array)
+ .map(|hits| {
+ hits.iter()
+ .filter_map(|h| {
+ h.get("_id")
+ .and_then(Value::as_str)
+ .map(String::from)
+ .zip(h.get("_score").and_then(Value::as_f64))
+ })
+ .collect()
+ })
+ .unwrap_or_default();
+ assert!(
+ hits.iter().any(|(id, _)| id == "3"),
+ "#825: the vector-only document must be in the hit union: {hits:?} {body}"
+ );
+ // Doc 1's vector IS the query vector while doc 2's is not; their BM25
+ // halves are identical. With the kNN half contributing, doc 1 must
+ // outrank doc 2 — under the old silent drop both answered the same
+ // pure-BM25 score.
+ let score = |wanted: &str| {
+ hits.iter()
+ .filter(|(id, _)| id == wanted)
+ .map(|(_, s)| *s)
+ .next()
+ };
+ let (s1, s2) = (score("1"), score("2"));
assert!(
- body.pointer("/hits/hits")
- .and_then(Value::as_array)
- .is_some_and(|h| !h.is_empty()),
- "hits must still be returned alongside the aggregation: {body}"
+ s1.zip(s2).is_some_and(|(a, b)| a > b),
+ "#825: the vector contribution must separate the lexically-tied docs \
+ (doc 1 carries the exact query vector): s1={s1:?} s2={s2:?} {body}"
);
}
-/// The core fix: a hybrid-safe `knn` beside a `query` must return the UNION of
-/// both halves — a document reachable only via the vector side must appear. The
-/// old fold to a two-clause `bool.should` never dispatched the kNN half, so that
-/// document was silently dropped.
+/// The core contract: `knn` beside a `query` must return the UNION of both
+/// halves — a document reachable only via the vector side must appear — and a
+/// document reached by BOTH halves scores the SUM `query_score + knn_score`
+/// (#825, ES semantics), so the lexical+vector doc outranks the vector-only
+/// doc even though the latter carries the stronger vector match (its vector
+/// IS the query vector). The original fold to a two-clause `bool.should` never
+/// dispatched the kNN half at all; the #458 interim fix served this shape with
+/// RRF rank fusion instead of ES's score sum.
#[tokio::test]
async fn knn_beside_query_returns_the_vector_only_document() {
let (app, _dir) = app().await;
@@ -188,15 +249,27 @@ async fn knn_beside_query_returns_the_vector_only_document() {
ids.iter().any(|id| id == "lex"),
"the lexical match must still be present: hits={ids:?} body={body}"
);
+ // #825: summed scores, not rank fusion. Both docs are in the vector
+ // top-k (k=2 of 2), so "lex" scores BM25 + its (weaker) vector score
+ // while "vec" scores its (perfect) vector match alone — the sum must
+ // put "lex" first. Under RRF both halves' ranks fused to ~1/61-scale
+ // micro-scores instead of the ES score sum.
+ assert_eq!(
+ ids.first().map(String::as_str),
+ Some("lex"),
+ "#825: the doc matching both halves must outrank the vector-only doc \
+ (score = query_score + knn_score): hits={ids:?} body={body}"
+ );
}
-/// A hybrid-incompatible feature (here `rescore`, which the fusion executor
-/// silently drops) must keep the request on the lexical `bool.should` path where
-/// rescore is honoured — NOT route it to `hybrid`. Observable proxy: the
-/// vector-only document that ONLY the hybrid path surfaces must be ABSENT, so a
-/// present-`rescore` request is provably not on the hybrid branch.
+/// #825: a feature the fusion executor could not serve (here `rescore`) no
+/// longer costs the request its vector half. The engine pins the knn top-k
+/// into the `bool.should` tree, so the vector-only document appears in the
+/// union AND rescore is honoured on the normal path. Under #458 this exact
+/// shape silently dropped the kNN contribution (this test used to assert the
+/// vector doc was ABSENT — that was the documented cost of keeping rescore).
#[tokio::test]
-async fn knn_beside_query_with_rescore_stays_on_the_lexical_path() {
+async fn knn_beside_query_with_rescore_keeps_the_vector_contribution() {
let (app, _dir) = app().await;
let (st, b) = json_req(
@@ -252,9 +325,107 @@ async fn knn_beside_query_with_rescore_stays_on_the_lexical_path() {
})
.unwrap_or_default();
assert!(
- !ids.iter().any(|id| id == "vec"),
- "#458: knn+query+rescore must stay on bool.should (rescore honoured), not route to the \
- hybrid executor that silently drops rescore — the hybrid-only vector doc must be \
- absent: hits={ids:?} body={body}"
+ ids.iter().any(|id| id == "vec"),
+ "#825: knn+query+rescore must serve the UNION — the vector-only doc must appear \
+ even with rescore present (the old bool.should fallback silently dropped the \
+ kNN half for every rescore-bearing request): hits={ids:?} body={body}"
+ );
+ assert!(
+ ids.iter().any(|id| id == "lex"),
+ "the lexical (and rescored) match must still be present: hits={ids:?} body={body}"
+ );
+}
+
+/// #825 (review follow-up): the summed-score contract must also hold when the
+/// lexical half is itself a `bool` carrying two or more scoring text clauses —
+/// the shape that reaches the post-scan IDF heuristic rescore, whose gate
+/// `query_uses_bool_text` counts the user's inner bool and is blind to the
+/// pinned kNN sub-tree. That pass rewrites `hit.score` from term frequencies
+/// alone, so run blind it discards the vector contribution for exactly the
+/// documents reached by BOTH halves — the same silent drop #825 exists to
+/// close, one layer down. A bare `match`/`term` query half returns
+/// text_children = 1 and never reaches it, which is why every other test here
+/// misses this.
+///
+/// The two documents carry identical lexical content (identical tf and field
+/// lengths, so identical BM25 *and* identical heuristic scores) and differ
+/// only in their vector: the one whose vector IS the query vector must rank
+/// strictly above the other. `near` is indexed SECOND, so if the scores
+/// collapsed to a tie the arrival-order tie-break would put it last.
+#[tokio::test]
+async fn knn_beside_bool_query_keeps_the_vector_contribution() {
+ let (app, _dir) = app().await;
+
+ let (st, b) = json_req(
+ &app,
+ "PUT",
+ "/bq",
+ json!({ "mappings": { "properties": {
+ "title": { "type": "text" },
+ "body": { "type": "text" },
+ "v": { "type": "dense_vector", "dims": 3 }
+ } } }),
+ )
+ .await;
+ assert!(st.is_success(), "create index: {st} {b}");
+
+ for (id, vec) in [("far", [1.0_f32, 0.0, 0.0]), ("near", [0.1_f32, 0.2, 0.3])] {
+ let (st, b) = json_req(
+ &app,
+ "POST",
+ &format!("/bq/_doc/{id}"),
+ json!({ "title": "alpha", "body": "beta", "v": vec }),
+ )
+ .await;
+ assert!(st.is_success(), "index {id}: {st} {b}");
+ }
+ let (_s, _b) = json_req(&app, "POST", "/bq/_refresh", json!({})).await;
+
+ let (status, body) = json_req(
+ &app,
+ "POST",
+ "/bq/_search",
+ json!({
+ "query": { "bool": { "must": [
+ { "match": { "title": "alpha" } },
+ { "match": { "body": "beta" } }
+ ] } },
+ "knn": { "field": "v", "query_vector": [0.1, 0.2, 0.3], "k": 2, "num_candidates": 10 }
+ }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "_search status: {status} {body}");
+
+ let hits: Vec<(String, f64)> = body
+ .pointer("/hits/hits")
+ .and_then(Value::as_array)
+ .map(|hits| {
+ hits.iter()
+ .filter_map(|h| {
+ h.get("_id")
+ .and_then(Value::as_str)
+ .map(String::from)
+ .zip(h.get("_score").and_then(Value::as_f64))
+ })
+ .collect()
+ })
+ .unwrap_or_default();
+ let score = |wanted: &str| {
+ hits.iter()
+ .filter(|(id, _)| id == wanted)
+ .map(|(_, s)| *s)
+ .next()
+ };
+ let (near, far) = (score("near"), score("far"));
+ assert!(
+ near.zip(far).is_some_and(|(n, f)| n > f),
+ "#825: with a multi-clause bool as the lexical half the kNN score must still be \
+ summed in — the post-scan IDF rescore must not overwrite the pinned vector \
+ contribution: near={near:?} far={far:?} {body}"
+ );
+ assert_eq!(
+ hits.first().map(|(id, _)| id.as_str()),
+ Some("near"),
+ "#825: the better vector match must lead the page: hits={hits:?} {body}"
);
}
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 5a2eaa733..ab16f910a 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -14237,6 +14237,158 @@ impl Index {
})
}
+ /// #825: execute one `Knn` clause standalone and return a replacement
+ /// sub-tree that pins its top-k results as per-document constant scores:
+ /// `bool.should = [Constant{score, Ids{[id]}}, …]` (or `MatchNone` when
+ /// the leg returns nothing). The generic matcher/scorer has no `Knn`
+ /// arm, so this is what lets a `knn` clause inside a compound `bool`
+ /// contribute at all — and it reproduces ES's hybrid contract exactly:
+ /// the knn side contributes the GLOBAL top-k neighbours, a document in
+ /// both halves scores the SUM (Bool sums clause scores), and every
+ /// request feature (aggs over the union, sort, collapse, highlight,
+ /// rescore, min_score) applies through the normal path afterwards.
+ ///
+ /// The leg runs on the SAME executors as the single-clause peel in
+ /// `search_inner`: HNSW only for the plain case (exact-rescored scores,
+ /// but `aggs` on the OUTER request forces exact brute force — the rc.6
+ /// rule that ANN approximation must not leak into agg buckets), brute
+ /// force otherwise. Boost and the raw-similarity cutoff are applied by
+ /// those executors before the scores reach the pinned constants.
+ async fn pin_knn_clause(
+ &self,
+ request: &SearchRequest,
+ deadline: std::time::Instant,
+ knn: &QueryNode,
+ ) -> Result<QueryNode> {
+ if let QueryNode::Knn {
+ field,
+ vector,
+ k,
+ num_candidates,
+ filter,
+ boost,
+ similarity: min_similarity,
+ } = knn
+ {
+ let similarity = {
+ let schema = self.schema.read().await;
+ lookup_vector_similarity(&schema.schema, field)
+ };
+ // The leg needs the FULL top-k regardless of the caller's page:
+ // ES unions the global k nearest into the candidate set even when
+ // `size` is smaller, so pagination happens after the merge.
+ let sub_request = SearchRequest {
+ query: knn.clone(),
+ from: 0,
+ size: *k,
+ sort: Vec::new(),
+ search_after: None,
+ aggs: None,
+ explain: false,
+ source: request.source.clone(),
+ timeout_ms: request.timeout_ms,
+ highlight: None,
+ track_total_hits: request.track_total_hits,
+ script_fields: None,
+ fields: Vec::new(),
+ profile: false,
+ collapse: None,
+ rescore: Vec::new(),
+ min_score: None,
+ leaf_ts_field: None,
+ savings: request.savings,
+ };
+ let plain = filter.is_none()
+ && boost.is_none()
+ && min_similarity.is_none()
+ && request.aggs.is_none();
+ let hnsw = if plain {
+ self.run_knn_hnsw(
+ &sub_request,
+ deadline,
+ field,
+ vector,
+ *k,
+ *num_candidates,
+ &similarity,
+ )
+ .await
+ } else {
+ None
+ };
+ let leg = match hnsw {
+ Some(result) => result,
+ None => {
+ self.run_knn_brute_force_with_deadline(
+ &sub_request,
+ deadline,
+ field,
+ vector,
+ *k,
+ filter.clone(),
+ &similarity,
+ *boost,
+ *min_similarity,
+ )
+ .await?
+ }
+ };
+ // Same execution-truth check as the single-knn and multi-knn
+ // arms (#498): an empty leg on an unanswerable field must fail
+ // loud, not degrade to lexical-only.
+ if leg.hits.is_empty() && !leg.timed_out {
+ self.reject_unanswerable_knn_field(field).await?;
+ }
+ let pinned: Vec<QueryNode> = leg
+ .hits
+ .iter()
+ .map(|hit| QueryNode::Constant {
+ score: hit.score,
+ query: Box::new(QueryNode::Ids {
+ values: vec![hit.id.clone()],
+ }),
+ })
+ .collect();
+ Ok(if pinned.is_empty() {
+ QueryNode::MatchNone
+ } else {
+ // The `filter` is a REDUNDANT accelerator, not semantics: it
+ // holds exactly the union of the `should` clauses' ids, so
+ // `filter ∧ (≥1 should)` and `(≥1 should)` have the same match
+ // set, and `filter` contributes nothing to `_score` (only
+ // must/should are summed — `BooleanClause.isScoring()`).
+ // What it buys is the shape of the DOC-SCAN cost. The scan
+ // evaluates must/must_not/filter BEFORE should, so a document
+ // outside the top-k is now rejected by ONE `_id` lookup plus a
+ // memcmp sweep, instead of recursing into all k
+ // `Constant{Ids}` clauses and re-reading `_id` from the source
+ // map k times. Measured on a 100 k-doc index, together with
+ // the `should` short-circuit in `doc_matches_query_typed`
+ // (see the PR's measurement table): hybrid k=1000
+ // 2309 ms → 385 ms, k=10000 24 437 ms → 5 071 ms.
+ // `minimum_should_match: 1` is written out rather than left
+ // implicit: with a non-empty `filter` the default would fall
+ // to 0 ("filter alone decides"), which is the same match set
+ // here but is NOT the same statement, and any other reader of
+ // this tree should see the disjunction that was meant.
+ QueryNode::Bool {
+ must: Vec::new(),
+ should: pinned,
+ filter: vec![QueryNode::Ids {
+ values: leg.hits.iter().map(|hit| hit.id.clone()).collect(),
+ }],
+ must_not: Vec::new(),
+ minimum_should_match: Some(MinShouldMatch::Fixed(1)),
+ }
+ })
+ } else {
+ // Caller contract: `knn` is a `QueryNode::Knn` (see
+ // `compound_bool_direct_knn`). Anything else passes through
+ // untouched rather than panicking.
+ Ok(knn.clone())
+ }
+ }
+
/// `field` is the full dotted path as the ES query provides it
/// (e.g. `nested.vector`). The element inside the nested array is
/// the sub-field after the path prefix (`vector`).
@@ -16409,6 +16561,36 @@ impl Index {
.await;
}
+ // ── #825: `knn` clause inside a compound `bool` ────────────────────────
+ // Every vector-aware short-circuit above has declined by now, so a
+ // `Knn` node still in the tree is about to hit the generic path,
+ // where it matches nothing and scores 0.0 (silent drop). The ES 8.x
+ // canonical hybrid — a top-level `knn` beside a `query`, folded by
+ // the compat layer into exactly this `bool.should` — lands here for
+ // EVERY request since the #458 hybrid_safe fold was retired, and
+ // previously landed here whenever aggs/sort/collapse/rescore/… were
+ // present. Pre-execute the knn leg and splice its top-k back in as
+ // pinned per-document constants; the generic path then serves the
+ // union with summed scores and full feature support (see
+ // `pin_knn_clause`).
+ //
+ // `pinned_knn_scores` is the id → knn-score map of the clauses we
+ // just spliced in. Everything downstream that RE-WRITES `_score`
+ // rather than reading it has to add the vector half back, or the
+ // sum contract is lost exactly where it was hardest to see — see
+ // the IDF heuristic rescore further down.
+ let mut pinned_knn_scores: HashMap<String, f32> = HashMap::new();
+ let pinned_query = if let Some(knn_clause) = compound_bool_direct_knn(query) {
+ let pinned = self
+ .pin_knn_clause(request, search_deadline, knn_clause)
+ .await?;
+ collect_pinned_knn_scores(&pinned, &mut pinned_knn_scores);
+ Some(replace_direct_knn_with_pinned(query, &pinned))
+ } else {
+ None
+ };
+ let query = pinned_query.as_ref().unwrap_or(query);
+
// ── Max result window enforcement ──────────────────────────────────────
// Default max_result_window is 10,000 (matches ES default).
// Allow override via index setting `index.max_result_window`.
@@ -19908,7 +20090,27 @@ impl Index {
}
}
if score > 0.0 {
- hit.score = score;
+ // #825: this pass derives a score from term
+ // frequencies ALONE, so on a #825-pinned tree it
+ // would overwrite the summed `query_score +
+ // knn_score` with the lexical half only — silently
+ // dropping the vector contribution for exactly the
+ // documents reached by BOTH halves, while a
+ // vector-only doc (tf 0, heuristic 0) keeps its knn
+ // score and can then outrank them. `query_uses_bool_text`
+ // above cannot see the difference: the user's inner
+ // bool supplies the ≥2 text clauses and the pinned
+ // sub-tree contributes 0 text children, so it neither
+ // disqualifies nor suppresses. Add the pinned constant
+ // back rather than suppressing the rescore, so the
+ // lexical half still gets its IDF weighting.
+ // (Empty map — the overwhelmingly common case — costs
+ // one hash lookup that always misses.)
+ hit.score = score
+ + pinned_knn_scores
+ .get(hit.id.as_str())
+ .copied()
+ .unwrap_or(0.0);
}
}
// Re-sort by the new scores (#270 — full page-order key, so
@@ -33826,6 +34028,106 @@ fn peel_multi_knn_query(q: &QueryNode) -> Option<Vec<PeeledKnn>> {
}
}
+/// #825: locate the single direct `Knn` clause inside a compound `bool`.
+///
+/// `peel_knn_query` serves the one-scoring-clause wrapper shapes; a `Knn`
+/// beside OTHER clauses falls through to the generic path, whose
+/// matcher/scorer has no `Knn` arm (`doc_matches_query_typed` answers
+/// `false`, `score_query_against_doc` contributes `0.0`) — so the ES-compat
+/// fold of a top-level `knn` beside a `query` silently returned the lexical
+/// half alone whenever the request was not routed elsewhere (#395/#458).
+///
+/// Returns the clause only when the bool carries EXACTLY ONE direct `Knn`
+/// child (in `must` or `should`). Several sibling `Knn` clauses would each
+/// need their own pinned executor pass, and no supported request shape
+/// produces that tree: the ES-compat layer answers 400 to a `knn` array
+/// beside a `query` before folding, and the pure `knn` array form is served
+/// by `peel_multi_knn_query` above.
+///
+/// Everything else keeps the pre-existing (dropped) behaviour: a `Knn`
+/// nested deeper than a direct child, AND a `Knn` sitting directly in
+/// `filter` or `must_not` — only the SCORING lists are scanned here, and
+/// `replace_direct_knn_with_pinned` likewise only swaps those two. That is
+/// exact rather than tidy: `filter`/`must_not` are the non-scoring lists
+/// (`BooleanClause.isScoring()` is `MUST || SHOULD`), a pinned constant
+/// there would contribute nothing anyway, and no supported request shape
+/// produces such a tree — the ES-compat fold only ever emits
+/// `bool.should[query, knn]`.
+fn compound_bool_direct_knn(q: &QueryNode) -> Option<&QueryNode> {
+ if let QueryNode::Bool { must, should, .. } = q {
+ let mut knns = must
+ .iter()
+ .chain(should.iter())
+ .filter(|clause| matches!(clause, QueryNode::Knn { .. }));
+ let first = knns.next();
+ let surplus = knns.next();
+ first.filter(|_| surplus.is_none())
+ } else {
+ None
+ }
+}
+
+/// #825: read the id → knn-score map back out of the sub-tree
+/// `pin_knn_clause` just produced (`Bool{should:[Constant{score,
+/// Ids{[id]}}, …]}`, or `MatchNone` for an empty leg).
+///
+/// Deliberately scoped to that exact shape and called ONLY on
+/// `pin_knn_clause`'s own output, never on the user's tree: a
+/// user-supplied `constant_score{filter:{ids}}` is an ordinary scoring
+/// clause whose interaction with the heuristic rescore is pre-existing
+/// behaviour, and folding it in here would change it.
+fn collect_pinned_knn_scores(pinned: &QueryNode, out: &mut HashMap<String, f32>) {
+ if let QueryNode::Bool { should, .. } = pinned {
+ for clause in should {
+ if let QueryNode::Constant { score, query } = clause {
+ if let QueryNode::Ids { values } = query.as_ref() {
+ for id in values {
+ // A document can be pinned once only (the leg
+ // returns distinct ids), so `insert` is total.
+ out.insert(id.clone(), *score);
+ }
+ }
+ }
+ }
+ }
+}
+
+/// #825: rebuild the compound bool with its direct `Knn` clause(s) swapped
+/// for the pre-executed pinned sub-tree. Pure shape surgery — every other
+/// clause, the filters, and `minimum_should_match` pass through unchanged.
+fn replace_direct_knn_with_pinned(q: &QueryNode, pinned: &QueryNode) -> QueryNode {
+ if let QueryNode::Bool {
+ must,
+ should,
+ filter,
+ must_not,
+ minimum_should_match,
+ } = q
+ {
+ let swap = |clauses: &[QueryNode]| -> Vec<QueryNode> {
+ clauses
+ .iter()
+ .map(|clause| {
+ if matches!(clause, QueryNode::Knn { .. }) {
+ pinned.clone()
+ } else {
+ clause.clone()
+ }
+ })
+ .collect()
+ };
+ QueryNode::Bool {
+ must: swap(must),
+ should: swap(should),
+ filter: filter.clone(),
+ must_not: must_not.clone(),
+ minimum_should_match: minimum_should_match.clone(),
+ }
+ } else {
+ q.clone()
+ }
+}
+
// Peel a nested-KNN query and partition surrounding clauses into
// pre-filters (applied BEFORE top-k ranking — align with ES `knn.filter`
// and `bool.filter`) and post-filters (applied AFTER top-k — align with
@@ -38552,11 +38854,28 @@ fn doc_matches_query_typed(q: &QueryNode, source: &Value, schema: &Schema) -> bo
}
};
if min > 0 {
- let matched = should
- .iter()
- .filter(|q| doc_matches_query_typed(q, source, schema))
- .count();
- if matched < min {
+ // Stop as soon as the requirement is met. The old
+ // `.filter(…).count()` evaluated EVERY should-clause on
+ // EVERY document even when the first one already settled
+ // it — O(clauses) per doc with no upside. That is free
+ // ordinarily and expensive on a #825-pinned tree, whose
+ // should-list is one clause per pinned kNN neighbour:
+ // a document the lexical clause already matched no longer
+ // walks the pinned sub-tree at all. Semantics are
+ // unchanged — `matched >= min` is the same predicate,
+ // reached without counting past `min`.
+ let mut matched = 0usize;
+ let mut satisfied = false;
+ for q in should {
+ if doc_matches_query_typed(q, source, schema) {
+ matched += 1;
+ if matched >= min {
+ satisfied = true;
+ break;
+ }
+ }
+ }
+ if !satisfied {
return false;
}
}
diff --git a/engine/tests/es-compat-yaml/yaml/vectors/96_knn_beside_query_aggs.yml b/engine/tests/es-compat-yaml/yaml/vectors/96_knn_beside_query_aggs.yml
new file mode 100644
index 000000000..5e2039f0a
--- /dev/null
+++ b/engine/tests/es-compat-yaml/yaml/vectors/96_knn_beside_query_aggs.yml
@@ -0,0 +1,82 @@
+setup:
+
+ - do:
+ indices.create:
+ index: knn_union_test
+ body:
+ settings:
+ number_of_shards: 1
+ mappings:
+ properties:
+ title:
+ type: text
+ category:
+ type: keyword
+ vector:
+ type: dense_vector
+ dims: 3
+ index: true
+ similarity: cosine
+
+ - do:
+ bulk:
+ refresh: true
+ body:
+ - '{ "index": { "_index": "knn_union_test", "_id": "1" } }'
+ - '{ "title": "quick brown fox", "category": "a", "vector": [0.9, 0.1, 0.0] }'
+ - '{ "index": { "_index": "knn_union_test", "_id": "2" } }'
+ - '{ "title": "quick brown dog", "category": "a", "vector": [0.0, 1.0, 0.0] }'
+ - '{ "index": { "_index": "knn_union_test", "_id": "3" } }'
+ - '{ "title": "zzz qqq", "category": "z", "vector": [1.0, 0.0, 0.0] }'
+
+---
+"knn beside query with aggs returns the union and buckets over it":
+
+ # Issue #825: the knn contribution must not be dropped when aggs are
+ # present. The hit set is the union of both halves (ES 8.x semantics)
+ # and the aggregations run over that union.
+ - do:
+ search:
+ rest_total_hits_as_int: true
+ index: knn_union_test
+ body:
+ size: 10
+ query:
+ match:
+ title: "quick"
+ knn:
+ field: vector
+ query_vector: [1.0, 0.0, 0.0]
+ k: 2
+ num_candidates: 10
+ aggs:
+ by_category:
+ terms:
+ field: category
+
+ - match: { hits.total: 3 }
+ - length: { aggregations.by_category.buckets: 2 }
+ - match: { aggregations.by_category.buckets.0.key: "a" }
+ - match: { aggregations.by_category.buckets.0.doc_count: 2 }
+ - match: { aggregations.by_category.buckets.1.key: "z" }
+ - match: { aggregations.by_category.buckets.1.doc_count: 1 }
+
+---
+"knn beside query without aggs is the same union":
+
+ - do:
+ search:
+ rest_total_hits_as_int: true
+ index: knn_union_test
+ body:
+ size: 10
+ query:
+ match:
+ title: "quick"
+ knn:
+ field: vector
+ query_vector: [1.0, 0.0, 0.0]
+ k: 2
+ num_candidates: 10
+
+ - match: { hits.total: 3 }