AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(fts): match_phrase slop admits transpositions at Lucene cost 2 (#830) (#882)
xerj-org66d5a2274a8e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 17ecffa01..063714f4e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **`match_phrase` `slop` now admits transposed terms at Lucene's cost of 2**+ ([#830](https://github.com/xerj-org/xerj/issues/830)). The sloppy-phrase+ walk was in-order only — each next term matched strictly after the+ previous one — so a reordered pair never matched at ANY slop:+ `{"match_phrase":{"t":{"query":"quick brown","slop":2}}}` returned zero+ hits on a document reading `brown quick`, where Lucene/ES match it at+ distance 2 (`SloppyPhraseMatcher`'s own javadoc example). The evaluator is+ now the Lucene move-distance semantics — pick one document position per+ phrase term; the distance is the span of the positions after subtracting+ each term's query offset — implemented once+ (`xerj_fts::search::phrase_positions_match`) and called by both the segment+ positional clause and the engine's memtable/stored-scan walk+ (`phrase_walk`), so slop is evaluated identically on either side of a+ flush. `match_phrase_prefix` and `multi_match` phrase/phrase_prefix go+ through the same evaluator and gain the same behavior. Documents that+ matched before still match — an in-order pick has strictly increasing+ positions, so its span telescopes to exactly the old summed-gaps value —+ and that is checked rather than assumed, by an exhaustive test over every+ document of length <= 5 and every phrase of length <= 3 from a 3-symbol+ alphabet at slop 0..3, comparing against both the old walk and a+ brute-force reference. A repeated phrase term still needs as many DISTINCT+ document positions as it has slots (`"a a"` does not match a doc holding+ one `a`).+- **Wrapping a query in a one-clause `bool` no longer changes its `_score` orits ranking** ([#399](https://github.com/xerj-org/xerj/issues/399)).`{"bool":{"must":[X]}}` and bare `X` are the same query — Lucene'sdiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 0814d574d..21a9f9d79 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -37476,20 +37476,15 @@ fn phrase_tokens(text: &str) -> Vec<String> {.analyze_to_terms(text)}-/// True when `query_tokens` occur in `field_tokens` in order with at most-/// `slop` intervening positions in total — the stored-scan twin of-/// `xerj_fts::search`'s `phrase_positions_match`, which evaluates the same+/// True when `query_tokens` occur in `field_tokens` within `slop` under+/// Lucene sloppy-phrase semantics — the stored-scan twin of+/// `xerj_fts::search::phrase_positions_match`, which evaluates the same/// predicate over the segment's real term positions.///-/// `slop == 0` is exact adjacency (a window compare); `slop > 0` anchors on-/// EVERY occurrence of the first term and walks each later term to its-/// earliest position after the previous match, summing the gaps — the same-/// greedy walk `phrase_positions_match` runs over segment positions.-///-/// Anchoring on every occurrence is load-bearing: the previous stored-scan-/// walk anchored only on the FIRST occurrence of the leading term, so-/// `[a, x, x, x, a, b]` rejected the slop-1 phrase `a b` that both ES and-/// the segment path accept (anchor at the second `a`).+/// `slop == 0` is exact adjacency; `slop > 0` is an edit distance in+/// single-position term moves (transpositions cost 2 — `"a b"~2` matches a+/// doc reading `b a`; see `phrase_positions_match`, which closed #830's+/// in-order-only divergence for both arms at once).fn phrase_positions_in_tokens(field_tokens: &[String], query_tokens: &[String], slop: u32) -> bool {phrase_walk(field_tokens, query_tokens, slop, false)}@@ -37502,23 +37497,25 @@ fn phrase_positions_in_tokens(field_tokens: &[String], query_tokens: &[String],/// query per expansion, and "some expansion term sits here" is the same/// predicate as "this token starts with the prefix".///-/// Keeping one walk (rather than two look-alike ones) is deliberate: the-/// two evaluators must not be able to drift apart the way the slop-0-only-/// prefix walk had already drifted from the sloppy phrase walk.+/// It materialises one token-index position list per query slot and hands+/// them to `xerj_fts::search::phrase_positions_match` — the SAME function+/// the segment positional clause runs over postings positions. Sharing the+/// evaluator (rather than mirroring it) is deliberate, and stronger than the+/// previous arrangement of two hand-synchronised walks: slop cannot be+/// evaluated differently on the two sides (#218/#222/#230 regression class),+/// and the #830 fix — Lucene `SloppyPhraseMatcher` distance semantics, where+/// slop admits transpositions at cost 2 instead of the old in-order-only gap+/// walk — lands in both arms at once.///-/// KNOWN DIVERGENCE FROM ES, not closed by #230: this walk is **in-order-/// only**. Lucene's sloppy phrase admits a transposition at a distance-/// cost — its own javadoc: «for query "a b"~2, a document "x a b a y" can-/// be matched twice: once for "a b" (distance=0), and once for "b a"-/// (distance=2)» (`lucene/core/.../search/SloppyPhraseMatcher.java`, class-/// javadoc; Apache-2.0) — so ES answers `match_phrase {"query": "policy-/// merge", "slop": 2}` on a document reading `merge policy`, and XERJ-/// answers it with zero hits. That is the pre-existing semantics of-/// `xerj_fts::search::phrase_positions_match`, which the segment side has-/// always used and which this walk deliberately mirrors: matching ES here-/// means changing BOTH evaluators together, and mirroring the existing one-/// is what keeps the hit set flush-invariant today. Measured, not assumed-/// (5-doc index, `"policy merge"` slop 2 and slop 3 → `[]` in both states).+/// What is shared is the EVALUATOR, not the position model, so this is not+/// flush-invariance "by construction": this arm re-analyses stored text with+/// the standard analyzer and feeds token INDICES, while the segment arm+/// feeds indexed postings positions. A custom analyzer (synonym+/// co-location, `position_increment_gap`) can therefore still hand the two+/// arms different position lists; multi-valued fields are handled per+/// element (#332). The tests in+/// `tests/match_phrase_slop_transposition.rs` pin the agreement for the+/// standard-analyzer cases that this planner actually routes both ways.fn phrase_walk(field_tokens: &[String],query_tokens: &[String],@@ -37532,56 +37529,30 @@ fn phrase_walk(return false;}let n = query_tokens.len();- let matches_at = |idx: usize, qi: usize| -> bool {- let ft = &field_tokens[idx];- if last_is_prefix && qi == n - 1 {- ft.starts_with(query_tokens[qi].as_str())- } else {- ft == &query_tokens[qi]- }- };- // Exact-adjacency fast path for the whole-term case (the common one).- if slop == 0 && !last_is_prefix {- return field_tokens.windows(n).any(|w| w == query_tokens);- }- for start in 0..field_tokens.len() {- if !matches_at(start, 0) {- continue;- }- let mut current = start;- let mut total_gaps: u32 = 0;- let mut ok = true;- for qi in 1..n {- // Earliest match after `current` minimises the running gap sum,- // so the greedy walk is optimal for an in-order phrase — the- // same walk `xerj_fts::search::phrase_positions_match` runs over- // segment positions.- match (current + 1..field_tokens.len()).find(|&i| matches_at(i, qi)) {- Some(next) => {- total_gaps += (next - current - 1) as u32;- if total_gaps > slop {- ok = false;- break;- }- current = next;- }- None => {- ok = false;- break;- }+ // Per-query-slot ascending token-index lists — the memtable analogue of+ // the segment's per-term postings position lists.+ let mut positions: Vec<Vec<u32>> = vec![Vec::new(); n];+ for (idx, ft) in field_tokens.iter().enumerate() {+ for (qi, qt) in query_tokens.iter().enumerate() {+ let hit = if last_is_prefix && qi == n - 1 {+ ft.starts_with(qt.as_str())+ } else {+ ft == qt+ };+ if hit {+ positions[qi].push(idx as u32);}}- if ok {- return true;- }}- false+ let lists: Vec<&[u32]> = positions.iter().map(|v| v.as_slice()).collect();+ xerj_fts::search::phrase_positions_match(&lists, slop)}-/// True when the leading `query_tokens[..n-1]` form an in-order phrase whose+/// True when the leading `query_tokens[..n-1]` form a phrase whose/// next position starts with `query_tokens[n-1]`, with at most `slop`-/// intervening positions in total — ES `match_phrase_prefix` semantics over-/// the token stream. A single token degrades to "any token starts with it".+/// (Lucene move-distance semantics) — ES `match_phrase_prefix` semantics+/// over the token stream. A single token degrades to "any token starts+/// with it".////// This is TERM-level, not substring-level: `merge poli` matches the token/// stream `[merge, policy]` (so it matches raw text `merge, policy` too),diff --git a/engine/crates/xerj-engine/tests/match_phrase_slop_transposition.rs b/engine/crates/xerj-engine/tests/match_phrase_slop_transposition.rsnew file mode 100644index 000000000..a5ca841a3--- /dev/null+++ b/engine/crates/xerj-engine/tests/match_phrase_slop_transposition.rs@@ -0,0 +1,262 @@+//! Regression tests for issue #830: `match_phrase` `slop` must follow+//! Lucene `SloppyPhraseMatcher` move-distance semantics — a transposed+//! (reordered) pair costs 2 — instead of the old in-order-only gap walk,+//! under which a reordering never matched at ANY slop.+//!+//! Lucene's own class javadoc: for query `"a b"~2`, a document `x a b a y`+//! matches once as `a b` (distance 0) and once as `b a` (distance 2), so+//! ES answers `{"match_phrase": {"t": {"query": "quick brown", "slop": 2}}}`+//! on a document reading `brown quick` — and pre-fix XERJ answered it with+//! zero hits at slop 2, 3, 4, ….+//!+//! Every case is asserted BOTH pre-flush and post-flush, which pins+//! flush-invariance (#218/#222/#230 regression class) as well as the+//! semantics. Note what the post-flush half does and does not exercise:+//! the planner keeps single-field `match_phrase`/`match_phrase_prefix` with+//! `slop > 0` on the STORED SCAN ("slop>0 and non-standard analyzers keep+//! the stored scan", index.rs), so those rows re-run `phrase_walk` after the+//! flush rather than the segment positional clause. The segment arm is+//! covered directly by the xerj-fts unit tests+//! (`sloppy_phrase_*` in `search.rs`) and by the `multi_match` case here.+//! Both arms call the one evaluator, `phrase_positions_match`.++use std::collections::BTreeSet;++use serde_json::{json, Value};+use tempfile::TempDir;+use xerj_common::config::Config;+use xerj_common::types::Schema;+use xerj_engine::{Engine, Index};+use xerj_query::parse_request;++fn make_engine(dir: &TempDir) -> Engine {+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ Engine::new(config).expect("engine::new")+}++fn req(q: Value) -> xerj_query::ast::SearchRequest {+ parse_request(&json!({ "query": q, "size": 50 })).expect("parse_request")+}++async fn ids(idx: &Index, q: &Value) -> BTreeSet<String> {+ idx.search(&req(q.clone()))+ .await+ .unwrap()+ .hits+ .iter()+ .map(|h| h.id.clone())+ .collect()+}++fn set(ids: &[&str]) -> BTreeSet<String> {+ ids.iter().map(|s| s.to_string()).collect()+}++/// Run every case against the memtable, flush once, run every case again+/// against the segment, and assert the hit set is the expected one in both+/// states — semantics AND flush-invariance in one assertion.+async fn assert_both_states(idx: &std::sync::Arc<Index>, cases: &[(Value, &[&str], &str)]) {+ for (q, exp, label) in cases {+ let pre = ids(idx, q).await;+ assert_eq!(+ pre,+ set(exp),+ "{label}: PRE-flush (memtable) hit set for {q}"+ );+ }+ idx.flush().await.unwrap();+ for (q, exp, label) in cases {+ let post = ids(idx, q).await;+ assert_eq!(+ post,+ set(exp),+ "{label}: POST-flush (segment) hit set for {q}"+ );+ }+}++/// Doc "adj" holds the phrase in order and adjacent; "rev" holds it+/// TRANSPOSED (the #830 case); "gap" holds it in order with one intervening+/// token; "ctrl" must never match.+async fn seed(engine: &Engine, name: &str) -> std::sync::Arc<Index> {+ engine.create_index(name, Schema::empty()).unwrap();+ let idx = engine.get_index(name).unwrap();+ for (id, text) in [+ ("adj", "quick brown fox"),+ ("rev", "brown quick fox"),+ ("gap", "quick lazy brown"),+ ("ctrl", "the lazy dog"),+ ] {+ idx.index_document(Some(id.into()), json!({ "t": text }))+ .await+ .unwrap();+ }+ idx+}++/// `match_phrase` "quick brown": slop 0 → adjacency only; slop 1 → the+/// forward gap joins; slop 2+ → the transposition joins (cost 2). The+/// slop-1 rows double as the negative case: the transposition must NOT be+/// admitted below 2, and `ctrl` never matches.+#[tokio::test]+async fn match_phrase_slop_admits_transposition_at_two() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ let idx = seed(&engine, "phrase_transposition").await;++ let q =+ |slop: u32| json!({ "match_phrase": { "t": { "query": "quick brown", "slop": slop } } });+ let cases: Vec<(Value, &[&str], &str)> = vec![+ (q(0), &["adj"], "slop 0: adjacency only"),+ (+ q(1),+ &["adj", "gap"],+ "slop 1: forward gap yes, transposition no",+ ),+ (+ q(2),+ &["adj", "gap", "rev"],+ "slop 2: transposition matches",+ ),+ (+ q(3),+ &["adj", "gap", "rev"],+ "slop 3: superset stays stable",+ ),+ ];+ assert_both_states(&idx, &cases).await;+}++/// Same semantics through `match_phrase_prefix` (the `last_is_prefix` arm of+/// the shared walk; the segment side evaluates it via per-expansion sloppy+/// phrase queries): "quick bro" finds `bro*` transposed against "quick" at+/// slop 2 and not below.+#[tokio::test]+async fn match_phrase_prefix_slop_admits_transposition_at_two() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ let idx = seed(&engine, "phrase_prefix_transposition").await;++ let q = |slop: u32| json!({ "match_phrase_prefix": { "t": { "query": "quick bro", "slop": slop } } });+ let cases: Vec<(Value, &[&str], &str)> = vec![+ (q(0), &["adj"], "prefix slop 0: adjacency only"),+ (+ q(1),+ &["adj", "gap"],+ "prefix slop 1: forward gap yes, transposition no",+ ),+ (+ q(2),+ &["adj", "gap", "rev"],+ "prefix slop 2: transposition matches",+ ),+ ];+ assert_both_states(&idx, &cases).await;+}++/// The `multi_match` phrase type lowers to the same per-field phrase+/// predicate — one case pins that arm too.+#[tokio::test]+async fn multi_match_phrase_slop_admits_transposition_at_two() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ let idx = seed(&engine, "multi_match_transposition").await;++ let q = |slop: u32| json!({ "multi_match": { "query": "quick brown", "fields": ["t"], "type": "phrase", "slop": slop } });+ let cases: Vec<(Value, &[&str], &str)> = vec![+ (+ q(1),+ &["adj", "gap"],+ "multi_match phrase slop 1: no transposition",+ ),+ (+ q(2),+ &["adj", "gap", "rev"],+ "multi_match phrase slop 2: transposition matches",+ ),+ ];+ assert_both_states(&idx, &cases).await;+}++/// Doc "twin" repeats a phrase term; "single" holds it only once. The+/// review of #882 found that the first minimal-window sweep lost repeated+/// terms: `"i said no no"~1` stopped matching `i said uh no no` at every+/// slop, because on a collision it advanced the pointer at the minimum+/// adjusted position (a non-colliding, single-occurrence slot) and exhausted+/// it before ever trying the assignment that separates the two `no`s.+async fn seed_repeats(engine: &Engine, name: &str) -> std::sync::Arc<Index> {+ engine.create_index(name, Schema::empty()).unwrap();+ let idx = engine.get_index(name).unwrap();+ for (id, text) in [+ ("twin", "i said uh no no"),+ ("single", "i said no thanks"),+ ("noway", "no way no no"),+ ("ctrl", "the lazy dog"),+ ] {+ idx.index_document(Some(id.into()), json!({ "t": text }))+ .await+ .unwrap();+ }+ idx+}++/// A repeated phrase term must not cost a document its match. `"i said no+/// no"` needs two DISTINCT `no` tokens: "twin" has them (adjusted 0,0,1,1 —+/// span 1), "single" has one `no` and must never match at any slop.+#[tokio::test]+async fn match_phrase_slop_repeated_term_still_matches() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ let idx = seed_repeats(&engine, "phrase_repeats").await;++ let q =+ |slop: u32| json!({ "match_phrase": { "t": { "query": "i said no no", "slop": slop } } });+ let cases: Vec<(Value, &[&str], &str)> = vec![+ (+ q(1),+ &["twin"],+ "slop 1: one intervening token, two distinct 'no' tokens",+ ),+ (q(2), &["twin"], "slop 2: still just the doc with two 'no's"),+ (+ q(3),+ &["twin"],+ "slop 3: 'single' has only one 'no' — never",+ ),+ (q(10), &["twin"], "slop 10: a repeat needs a second token"),+ ];+ assert_both_states(&idx, &cases).await;+}++/// Every slot the same term: `"no no no"~1` on "no way no no"+/// (no@[0,2,3] — pick 0,2,3, adjusted 0,1,1, span 1).+#[tokio::test]+async fn match_phrase_slop_all_slots_repeated() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ let idx = seed_repeats(&engine, "phrase_repeats_all").await;++ let q = |slop: u32| json!({ "match_phrase": { "t": { "query": "no no no", "slop": slop } } });+ let cases: Vec<(Value, &[&str], &str)> = vec![+ (q(1), &["noway"], "slop 1: three distinct 'no' positions"),+ (q(3), &["noway"], "slop 3: superset stays stable"),+ ];+ assert_both_states(&idx, &cases).await;+}++/// The same defect through `multi_match` — the arm that DOES reach the+/// segment positional clause after a flush.+#[tokio::test]+async fn multi_match_phrase_slop_repeated_term_still_matches() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ let idx = seed_repeats(&engine, "multi_match_repeats").await;++ let q = |slop: u32| json!({ "multi_match": { "query": "i said no no", "fields": ["t"], "type": "phrase", "slop": slop } });+ let cases: Vec<(Value, &[&str], &str)> = vec![+ (q(1), &["twin"], "multi_match phrase slop 1: repeat matches"),+ (q(3), &["twin"], "multi_match phrase slop 3: repeat matches"),+ ];+ assert_both_states(&idx, &cases).await;+}diff --git a/engine/crates/xerj-engine/tests/multi_match_phrase_positions.rs b/engine/crates/xerj-engine/tests/multi_match_phrase_positions.rsindex ba030e741..6b8711bad 100644--- a/engine/crates/xerj-engine/tests/multi_match_phrase_positions.rs+++ b/engine/crates/xerj-engine/tests/multi_match_phrase_positions.rs@@ -285,26 +285,18 @@ async fn phrase_slop_is_honoured() {.await;}-/// PINS A KNOWN DIVERGENCE FROM ES, so nothing in this PR can be read as-/// claiming it away: XERJ's sloppy phrase is **in-order only**, at every-/// entry point and in both states.-///-/// Lucene's is not. `SloppyPhraseMatcher`'s class javadoc: «for query "a-/// b"~2, a document "x a b a y" can be matched twice: once for "a b"-/// (distance=0), and once for "b a" (distance=2)» — so ES answers-/// `{"query": "policy merge", "slop": 2}` with docs 1 and 2, and XERJ-/// answers with none. That is the pre-existing behaviour of-/// `xerj_fts::search::phrase_positions_match` (which single-field-/// `match_phrase` has always used) and the stored-scan walk added by #230-/// mirrors it deliberately, because the two evaluators disagreeing is the-/// flush-variance class this PR exists to remove.-///-/// It is NOT a regression: before #230 the `multi_match` phrase arms tested-/// raw-substring containment, which does not match a transposition either.-/// When the walk learns transpositions, both evaluators change together and-/// this test flips — that is the point of pinning it.+/// The former in-order-only divergence pin, FLIPPED by the #830 fix — its+/// own doc said «when the walk learns transpositions, both evaluators+/// change together and this test flips». XERJ's sloppy phrase now follows+/// Lucene `SloppyPhraseMatcher` move-distance: `SloppyPhraseMatcher`'s+/// class javadoc — «for query "a b"~2, a document "x a b a y" can be+/// matched twice: once for "a b" (distance=0), and once for "b a"+/// (distance=2)» — so ES answers `{"query": "policy merge", "slop": 2}`+/// with docs 1 and 2 (both read `merge policy`), and XERJ now gives the+/// same answer, at every entry point and in both states. The dedicated+/// slop-1 negative cases live in `match_phrase_slop_transposition.rs`.#[tokio::test]-async fn sloppy_phrase_is_in_order_only_unlike_lucene() {+async fn sloppy_phrase_admits_transpositions_like_lucene() {let dir = TempDir::new().unwrap();let engine = make_engine(&dir);let idx = seed(&engine, "mmpp_inorder").await;@@ -315,16 +307,16 @@ async fn sloppy_phrase_is_in_order_only_unlike_lucene() {(json!({"multi_match": {"query": "policy merge", "fields": ["body"],"type": "phrase", "slop": 2}}),- &[],- "transposed phrase, slop 2: ES matches, XERJ does not",+ &["1", "2"],+ "transposed phrase, slop 2: matches like ES (cost 2)",),(json!({"match_phrase": {"body": {"query": "policy merge", "slop": 3}}}),- &[],+ &["1", "2"],"transposed phrase, slop 3, single-field spelling: same answer",),- // The in-order control, so the case above cannot pass by- // accident (e.g. if slop stopped being honoured at all).+ // The in-order control at the same slop, so the cases above+ // cannot pass by slop being ignored altogether.(json!({"multi_match": {"query": "log policy", "fields": ["body"],"type": "phrase", "slop": 2}}),diff --git a/engine/crates/xerj-fts/src/lib.rs b/engine/crates/xerj-fts/src/lib.rsindex f3d4cd533..f0c3d4560 100644--- a/engine/crates/xerj-fts/src/lib.rs+++ b/engine/crates/xerj-fts/src/lib.rs@@ -84,5 +84,6 @@ pub use postings::{};pub use search::{- search_segments, BoolQuery, FtsSearcher, PhraseQuery, PrefixQuery, Query, ScoredHit, TermQuery,+ phrase_positions_match, search_segments, BoolQuery, FtsSearcher, PhraseQuery, PrefixQuery,+ Query, ScoredHit, TermQuery,};diff --git a/engine/crates/xerj-fts/src/search.rs b/engine/crates/xerj-fts/src/search.rsindex 8012efcdd..1a2545ab1 100644--- a/engine/crates/xerj-fts/src/search.rs+++ b/engine/crates/xerj-fts/src/search.rs@@ -169,13 +169,15 @@ impl TermQuery {}}-/// An ordered phrase — terms must appear in the given order with adjacent positions.+/// A phrase — terms must appear with adjacent positions in the given order+/// (`slop == 0`, the default), or within `slop` under Lucene move-distance+/// semantics (transpositions cost 2; see `phrase_positions_match`).#[derive(Debug, Clone, Serialize, Deserialize)]pub struct PhraseQuery {pub field: String,/// Pre-analyzed phrase terms in order.pub terms: Vec<String>,- /// Allowed number of intervening positions (slop).+ /// Allowed Lucene move-distance (slop).#[serde(default)]pub slop: u32,#[serde(default = "default_boost")]@@ -753,10 +755,10 @@ impl FtsSearcher {'doc: for &doc_id in term_maps[min_idx].0.keys() {// Gather each term's positions for this doc, in TERM ORDER (raw —// no offset games; the matcher below handles adjacency + slop).- let mut all_positions: Vec<&Vec<u32>> = Vec::with_capacity(pq.terms.len());+ let mut all_positions: Vec<&[u32]> = Vec::with_capacity(pq.terms.len());for (map, _) in &term_maps {match map.get(&doc_id) {- Some((_, positions)) => all_positions.push(positions),+ Some((_, positions)) => all_positions.push(positions.as_slice()),None => continue 'doc, // term absent from this doc}}@@ -1479,18 +1481,71 @@ pub fn search_segments(// ── Phrase matching helper ────────────────────────────────────────────────────/// Returns `true` if the per-term **raw** position lists (in TERM ORDER)-/// contain an in-order occurrence of the phrase within `slop`.+/// contain an occurrence of the phrase within `slop` — Lucene sloppy-phrase+/// semantics, transpositions included.////// `all_positions[i]` is the ascending list of positions at which the i-th/// phrase term occurs in one document.////// - `slop == 0`: exact adjacency. There must be a start `p` such that term i/// occupies exactly `p + i` for every i (start-position intersection).-/// - `slop > 0`: greedy in-order walk — each next term is matched at the-/// earliest position `> ` the previous match, and the summed gaps between-/// adjacent matched terms must not exceed `slop` (the semantics the engine's-/// stored-scan `MatchPhrase` uses).-fn phrase_positions_match(all_positions: &[&Vec<u32>], slop: u32) -> bool {+/// - `slop > 0`: Lucene `SloppyPhraseMatcher` semantics. Slop is an edit+/// distance in single-position term MOVES: pick one document position per+/// phrase term, and the match distance is the span of the ADJUSTED+/// positions (document position minus the term's query offset), i.e.+/// `max(p_i - i) - min(p_i - i)`; the phrase matches when some choice keeps+/// that distance `<= slop`. An in-order pick has strictly increasing+/// `p_i`, so `p_i - i` is non-decreasing and the span telescopes to+/// `(p_last - p_first) - (n-1)` — exactly the summed-gaps value the+/// pre-#830 walk measured — and its positions are automatically distinct.+/// Every phrase that walk matched therefore still matches here, which is+/// a property of the DECISION PROCEDURE being complete, not of the cost+/// function alone: an incomplete search over assignments can lose an+/// in-order match it never tries (it did, for repeated terms, before this+/// evaluator got its SDR window test). Unlike that walk this also admits+/// REORDERED terms — an out-of-order pair costs at least 2, so `"a b"~2`+/// matches a document reading `b a` (Lucene's class javadoc example: for+/// query `"a b"~2`, document `x a b a y` matches once at distance 0 and+/// once, as `b a`, at distance 2). Issue #830.+///+/// Adapted (algorithm, not code) from Lucene's `SloppyPhraseMatcher`+/// (lucene/core/src/java/org/apache/lucene/search/SloppyPhraseMatcher.java,+/// Apache-2.0): `nextMatch` (:205-:237) pops the minimum `PhrasePositions`+/// from a priority queue ordered by adjusted position (`pp.position` is set+/// to `position - offset` in PhrasePositions.java `nextPosition`, :53-:59)+/// and measures `matchLength = end - pp.position` — max minus min adjusted+/// position. That plain "advance the minimum" walk is only complete once+/// REPEATS have been resolved: a repeated query term needs as many DISTINCT+/// document positions as it has slots, and Lucene therefore advances the+/// colliding repeat rather than the current minimum (`advanceRpts`, :317).+/// This implementation reaches the same completeness differently, because+/// it has no term identity to group by (see the distinctness note below):+/// it sweeps every candidate window floor `lo` — each adjusted position is+/// one — and asks whether the window `[lo, lo + slop]` admits a choice of+/// one position per slot with all positions DISTINCT, i.e. a system of+/// distinct representatives, decided by bipartite matching. The sweep is+/// complete: if any assignment has span `<= slop`, then the window floored+/// at that assignment's own minimum adjusted position contains all of it.+///+/// Distinctness is enforced pairwise over ALL slots rather than only over+/// repeated terms. Under the standard analyzer the two are the same thing+/// (one token per position, so only a repeated term can collide). It is+/// stricter than Lucene on a field whose analyzer co-locates DIFFERENT+/// terms — XERJ's `SynonymFilter` emits synonyms at the same position —+/// where Lucene would let two distinct query terms share one position and+/// this does not. That is a deliberate, conservative divergence: it can+/// only withhold a match, never invent one, and it is what the pre-#830+/// walk did too (it required strictly increasing positions).+///+/// Both arms share this evaluator: the segment positional clause calls it+/// with real postings positions, and the engine's stored-scan/memtable arm+/// (`phrase_walk` in xerj-engine's `index.rs`) materialises token-index+/// lists and calls it too, so slop is evaluated identically on both sides.+/// What they do NOT share is the position model — the memtable arm+/// re-analyses text with the standard analyzer while the segment arm reads+/// indexed positions — so a custom analyzer (synonyms,+/// `position_increment_gap`) can still hand the two arms different inputs.+pub fn phrase_positions_match(all_positions: &[&[u32]], slop: u32) -> bool {if all_positions.iter().any(|p| p.is_empty()) {return false;}@@ -1517,28 +1572,161 @@ fn phrase_positions_match(all_positions: &[&Vec<u32>], slop: u32) -> bool {return false;}- // Sloppy phrase: greedy earliest-match walk, accumulating the gaps.- 'outer: for &start_pos in all_positions[0] {- let mut current_pos = start_pos;- let mut total_gaps: u32 = 0;- for positions in &all_positions[1..] {- // Earliest position strictly after the previous matched term.- match positions.iter().copied().find(|&p| p > current_pos) {- Some(next_pos) => {- total_gaps += next_pos - current_pos - 1;- if total_gaps > slop {- continue 'outer;- }- current_pos = next_pos;- }- None => continue 'outer,+ sloppy_phrase_match(all_positions, slop)+}++/// Is there a choice of one document position per phrase slot, all DISTINCT,+/// whose adjusted positions (`p_i - i`) span at most `slop`?+///+/// Sweeps candidate window floors in ascending order. Every adjusted+/// position is a candidate floor, and the optimal window is floored at the+/// minimum adjusted position of its own assignment, so the sweep misses+/// nothing. For a floor `lo` each slot's admissible positions are the+/// contiguous run `L_i ∩ [lo + i, lo + slop + i]`, tracked with pointers+/// that only ever move forward as `lo` grows, so the pointer work over the+/// whole sweep is linear in the total number of positions.+///+/// Feasibility inside one window is a system-of-distinct-representatives+/// test. It only needs the first `n` admissible positions of each slot: a+/// slot with `n` or more candidates can always be re-pointed at a free one+/// (the other `n - 1` slots occupy at most `n - 1` positions), so truncating+/// there cannot turn a feasible window infeasible.+fn sloppy_phrase_match(all_positions: &[&[u32]], slop: u32) -> bool {+ let n = all_positions.len();+ let slop = i64::from(slop);+ let adj = |i: usize, k: usize| i64::from(all_positions[i][k]) - i as i64;++ // Cursors generating the candidate floors, plus the [lo_ptr, hi_ptr)+ // admissible run per slot. All three only move forward.+ let mut cursor = vec![0usize; n];+ let mut lo_ptr = vec![0usize; n];+ let mut hi_ptr = vec![0usize; n];+ // Scratch reused across windows so the sweep does not allocate per step.+ let mut cand: Vec<&[u32]> = Vec::with_capacity(n);+ let mut pool: Vec<u32> = Vec::with_capacity(n * n);+ let mut owner: Vec<usize> = Vec::with_capacity(n * n);+ let mut seen: Vec<bool> = Vec::with_capacity(n * n);++ loop {+ // Next candidate floor: the smallest adjusted position not yet used+ // as a floor. Exhausted cursors mean no window is left.+ let mut lo = i64::MAX;+ for i in 0..n {+ if cursor[i] < all_positions[i].len() {+ lo = lo.min(adj(i, cursor[i]));+ }+ }+ if lo == i64::MAX {+ return false;+ }+ for i in 0..n {+ while cursor[i] < all_positions[i].len() && adj(i, cursor[i]) <= lo {+ cursor[i] += 1;}}- if total_gaps <= slop {++ // Admissible run per slot for the window [lo, lo + slop].+ let hi = lo + slop;+ let mut window_full = true;+ for i in 0..n {+ let list = all_positions[i];+ let first = lo + i as i64;+ let last = hi + i as i64;+ while lo_ptr[i] < list.len() && i64::from(list[lo_ptr[i]]) < first {+ lo_ptr[i] += 1;+ }+ if hi_ptr[i] < lo_ptr[i] {+ hi_ptr[i] = lo_ptr[i];+ }+ while hi_ptr[i] < list.len() && i64::from(list[hi_ptr[i]]) <= last {+ hi_ptr[i] += 1;+ }+ if hi_ptr[i] == lo_ptr[i] {+ window_full = false;+ }+ }+ if !window_full {+ continue;+ }++ // Fast path — and the whole story whenever no term repeats: if the+ // earliest admissible position of every slot is a different document+ // position, that assignment is already a match.+ let earliest_are_distinct = (0..n).all(|i| {+ (i + 1..n).all(|j| all_positions[i][lo_ptr[i]] != all_positions[j][lo_ptr[j]])+ });+ if earliest_are_distinct {+ return true;+ }++ // Repeats collide here. Decide the window exactly instead of+ // guessing which pointer to advance: an SDR over the truncated+ // candidate runs.+ cand.clear();+ for i in 0..n {+ let end = hi_ptr[i].min(lo_ptr[i].saturating_add(n));+ cand.push(&all_positions[i][lo_ptr[i]..end]);+ }+ if window_admits_distinct_choice(&cand, &mut pool, &mut owner, &mut seen) {return true;}}+}++/// Bipartite matching (Kuhn's augmenting-path algorithm) of phrase slots to+/// document positions: `true` when every slot can be given a position from+/// its own candidate list with no position used twice. Both sides are+/// bounded by the phrase term count, so this is a handful of steps.+fn window_admits_distinct_choice(+ cand: &[&[u32]],+ pool: &mut Vec<u32>,+ owner: &mut Vec<usize>,+ seen: &mut Vec<bool>,+) -> bool {+ let n = cand.len();+ pool.clear();+ for c in cand {+ pool.extend_from_slice(c);+ }+ pool.sort_unstable();+ pool.dedup();+ if pool.len() < n {+ return false;+ }+ owner.clear();+ owner.resize(pool.len(), usize::MAX);+ for slot in 0..n {+ seen.clear();+ seen.resize(pool.len(), false);+ if !augment_slot(slot, cand, pool, seen, owner) {+ return false;+ }+ }+ true+}+/// One augmenting-path step for `window_admits_distinct_choice`: try to seat+/// `slot`, displacing an already-seated slot only if that slot can move.+fn augment_slot(+ slot: usize,+ cand: &[&[u32]],+ pool: &[u32],+ seen: &mut [bool],+ owner: &mut [usize],+) -> bool {+ for &position in cand[slot] {+ let Ok(p) = pool.binary_search(&position) else {+ continue;+ };+ if seen[p] {+ continue;+ }+ seen[p] = true;+ if owner[p] == usize::MAX || augment_slot(owner[p], cand, pool, seen, owner) {+ owner[p] = slot;+ return true;+ }+ }false}@@ -2023,6 +2211,300 @@ mod tests {);}+ /// #830: `slop` is Lucene `SloppyPhraseMatcher` move-distance, so a+ /// transposed pair matches at slop >= 2 — the old walk was in-order-only+ /// and never matched a reordering at ANY slop.+ #[test]+ fn sloppy_phrase_transposition_semantics() {+ // Doc reading "brown quick" (brown@0, quick@1), query "quick brown".+ let quick: &[u32] = &[1];+ let brown: &[u32] = &[0];+ assert!(!phrase_positions_match(&[quick, brown], 0));+ assert!(+ !phrase_positions_match(&[quick, brown], 1),+ "a transposition costs 2, not 1"+ );+ assert!(+ phrase_positions_match(&[quick, brown], 2),+ "a transposition must match at slop 2"+ );+ assert!(phrase_positions_match(&[quick, brown], 3));++ // In-order forward gap still costs its width ("quick lazy brown").+ let quick_gap: &[u32] = &[0];+ let brown_gap: &[u32] = &[2];+ assert!(!phrase_positions_match(&[quick_gap, brown_gap], 0));+ assert!(phrase_positions_match(&[quick_gap, brown_gap], 1));++ // Lucene's class-javadoc example doc "x a b a y" (a@[1,3], b@2):+ // "a b" matches exactly at slop 0, and "b a" (b@2 → a@3) at slop 0+ // too via the in-order pair; both trivially within slop 2.+ let a: &[u32] = &[1, 3];+ let b: &[u32] = &[2];+ assert!(phrase_positions_match(&[a, b], 0));+ // NB "b a" here is satisfied by the IN-ORDER pair b@2 -> a@3, so it+ // pins the javadoc example, not transposition cost — the transposed+ // pair is pinned by quick/brown above, where no in-order pick exists.+ assert!(phrase_positions_match(&[b, a], 0));+ assert!(phrase_positions_match(&[b, a], 2));++ // Adjacent transposed pair inside a 3-term phrase: doc "one three+ // two" for query "one two three" — one@0, two@2, three@1 → adjusted+ // span (2-1) - (1-2) = 2.+ let one: &[u32] = &[0];+ let two: &[u32] = &[2];+ let three: &[u32] = &[1];+ assert!(!phrase_positions_match(&[one, two, three], 1));+ assert!(phrase_positions_match(&[one, two, three], 2));+ }++ /// A repeated phrase term must land on DISTINCT document positions —+ /// one document token cannot fill two phrase slots, however large the+ /// slop (Lucene's repeat-collision rule).+ #[test]+ fn sloppy_phrase_repeat_needs_distinct_positions() {+ let a_once: &[u32] = &[3];+ assert!(+ !phrase_positions_match(&[a_once, a_once], 10),+ "\"a a\" must not match a doc holding a single 'a'"+ );+ let a_gap: &[u32] = &[3, 5];+ assert!(+ !phrase_positions_match(&[a_gap, a_gap], 0),+ "'a x a' is not an adjacent \"a a\""+ );+ assert!(phrase_positions_match(&[a_gap, a_gap], 1));+ let a_adj: &[u32] = &[3, 4];+ assert!(phrase_positions_match(&[a_adj, a_adj], 0));+ }++ /// Regression for the defect the #830 review caught: a phrase holding a+ /// REPEATED term must keep every match the pre-#830 in-order walk gave.+ ///+ /// The minimal-window sweep as first written advanced the pointer at the+ /// MINIMUM adjusted position whenever a window collided. That pointer is+ /// usually a non-colliding slot, so the sweep walked away from the very+ /// assignment that would separate the repeats and exhausted a+ /// single-occurrence slot instead: `"i said no no"~1` stopped matching+ /// `i said uh no no` at every slop. The window/SDR formulation decides+ /// each window instead of guessing which pointer to move.+ #[test]+ fn sloppy_phrase_repeated_term_keeps_in_order_matches() {+ // Doc "i said uh no no": i@0, said@1, no@[3,4]. Query "i said no no"+ // picks no@3 and no@4 — adjusted 0,0,1,1, span 1.+ let i: &[u32] = &[0];+ let said: &[u32] = &[1];+ let no: &[u32] = &[3, 4];+ for slop in [1u32, 2, 3, 5, 10] {+ assert!(+ phrase_positions_match(&[i, said, no, no], slop),+ "\"i said no no\"~{slop} must match 'i said uh no no'"+ );+ }++ // Doc "a x b b": a@0, b@[2,3]. "a b b" is in order with one gap.+ let a: &[u32] = &[0];+ let b: &[u32] = &[2, 3];+ for slop in [1u32, 2, 4] {+ assert!(+ phrase_positions_match(&[a, b, b], slop),+ "\"a b b\"~{slop} must match 'a x b b'"+ );+ }++ // Doc "no way no no": every phrase slot is the same term (no@[0,2,3]).+ let no3: &[u32] = &[0, 2, 3];+ for slop in [1u32, 2, 3] {+ assert!(+ phrase_positions_match(&[no3, no3, no3], slop),+ "\"no no no\"~{slop} must match 'no way no no'"+ );+ }++ // The distinctness rule still binds: three slots cannot be filled by+ // two document tokens, at any slop.+ let two_tokens: &[u32] = &[3, 4];+ assert!(+ !phrase_positions_match(&[two_tokens, two_tokens, two_tokens], 10),+ "three repeats of one term need three distinct positions"+ );+ }++ /// Exhaustive cross-check of the sloppy evaluator over every document of+ /// length <= 5 and every phrase of length <= 3 from a 3-symbol alphabet,+ /// at slop 0..=3, against two independent oracles:+ ///+ /// 1. brute force over all assignments (one position per slot, all+ /// distinct, adjusted span <= slop) — catches an INCOMPLETE search,+ /// which is exactly what the #830 review found; and+ /// 2. the pre-#830 in-order walk, which must never match a phrase this+ /// evaluator rejects. That is the "no previously-matching document+ /// stops matching" claim, checked rather than asserted.+ #[test]+ fn sloppy_phrase_agrees_with_brute_force_and_keeps_pre_830_hits() {+ /// The evaluator this replaced (main @ #830): greedy earliest+ /// in-order walk accumulating the gaps.+ fn pre_830_walk(all: &[&[u32]], slop: u32) -> bool {+ if all.iter().any(|p| p.is_empty()) {+ return false;+ }+ if all.len() == 1 {+ return true;+ }+ 'outer: for &start_pos in all[0] {+ let mut current = start_pos;+ let mut gaps: u32 = 0;+ for positions in &all[1..] {+ match positions.iter().copied().find(|&p| p > current) {+ Some(next) => {+ gaps += next - current - 1;+ if gaps > slop {+ continue 'outer;+ }+ current = next;+ }+ None => continue 'outer,+ }+ }+ return true;+ }+ false+ }++ /// Reference semantics, stated directly: some assignment of one+ /// distinct position per slot has adjusted span <= slop (adjacency+ /// for slop 0).+ fn brute(all: &[&[u32]], slop: u32) -> bool {+ fn rec(k: usize, all: &[&[u32]], pick: &mut Vec<u32>, slop: u32) -> bool {+ if k == all.len() {+ if slop == 0 {+ return pick+ .iter()+ .enumerate()+ .all(|(i, &p)| i64::from(p) == i64::from(pick[0]) + i as i64);+ }+ let adjusted = |i: usize, p: u32| i64::from(p) - i as i64;+ let lo = pick+ .iter()+ .enumerate()+ .map(|(i, &p)| adjusted(i, p))+ .min()+ .unwrap();+ let hi = pick+ .iter()+ .enumerate()+ .map(|(i, &p)| adjusted(i, p))+ .max()+ .unwrap();+ return hi - lo <= i64::from(slop);+ }+ for &p in all[k] {+ if pick.contains(&p) {+ continue;+ }+ pick.push(p);+ if rec(k + 1, all, pick, slop) {+ return true;+ }+ pick.pop();+ }+ false+ }+ if all.iter().any(|p| p.is_empty()) {+ return false;+ }+ if all.len() == 1 {+ return true;+ }+ rec(0, all, &mut Vec::new(), slop)+ }++ fn sequences(len: usize, alphabet: u32) -> Vec<Vec<u32>> {+ let mut out = vec![Vec::new()];+ for _ in 0..len {+ let mut next = Vec::new();+ for prefix in &out {+ for symbol in 0..alphabet {+ let mut s = prefix.clone();+ s.push(symbol);+ next.push(s);+ }+ }+ out = next;+ }+ out+ }++ let mut checked = 0u32;+ for doc_len in 1..=5usize {+ for doc in sequences(doc_len, 3) {+ for query_len in 1..=3usize {+ for query in sequences(query_len, 3) {+ let lists: Vec<Vec<u32>> = query+ .iter()+ .map(|t| {+ doc.iter()+ .enumerate()+ .filter(|(_, d)| *d == t)+ .map(|(p, _)| p as u32)+ .collect()+ })+ .collect();+ let all: Vec<&[u32]> = lists.iter().map(|v| v.as_slice()).collect();+ for slop in 0..=3u32 {+ checked += 1;+ let got = phrase_positions_match(&all, slop);+ assert_eq!(+ got,+ brute(&all, slop),+ "doc {doc:?} query {query:?} slop {slop}: \+ evaluator disagrees with brute force"+ );+ if pre_830_walk(&all, slop) {+ assert!(+ got,+ "doc {doc:?} query {query:?} slop {slop}: \+ matched before #830 and must still match"+ );+ }+ }+ }+ }+ }+ }+ assert!(checked > 50_000, "coverage collapsed to {checked} cases");+ }++ /// #830 segment arm end-to-end: a sloppy `PhraseQuery` admits the+ /// transposed pair at slop 2 and still rejects it at slop 0/1. Docs 0+ /// and 1 read "… quick brown …"; the query asks for "brown quick".+ #[test]+ fn phrase_query_slop_transposition() {+ let dir = TempDir::new().unwrap();+ let searcher = setup_searcher(dir.path());++ let mk = |slop: u32| {+ let mut q = PhraseQuery::new("body", vec!["brown".to_owned(), "quick".to_owned()]);+ q.slop = slop;+ Query::Phrase(q)+ };+ assert!(searcher.search(&mk(0), 10, false).unwrap().is_empty());+ assert!(+ searcher.search(&mk(1), 10, false).unwrap().is_empty(),+ "transposition costs 2 — slop 1 must not admit it"+ );+ let ids: Vec<u32> = searcher+ .search(&mk(2), 10, false)+ .unwrap()+ .iter()+ .map(|h| h.doc_id)+ .collect();+ assert!(+ ids.contains(&0) && ids.contains(&1),+ "docs with 'quick brown' must match the reversed query at slop 2, got {ids:?}"+ );+ }+/// Case-sensitive wildcard/fuzzy (text-field path): the whitespace analyzer/// preserves case, so an indexed term "Rust" is matched by "R*" but NOT by/// "r*" when `case_insensitive = false`.diff --git a/engine/crates/xerj-query/src/ast.rs b/engine/crates/xerj-query/src/ast.rsindex 067e8625c..a6662a238 100644--- a/engine/crates/xerj-query/src/ast.rs+++ b/engine/crates/xerj-query/src/ast.rs@@ -540,10 +540,10 @@ pub enum QueryNode {minimum_should_match: Option<MinShouldMatch>,},- /// Ordered phrase match with optional slop. `slop` admits forward gaps- /// between the in-order tokens ONLY — it does NOT admit transpositions- /// (reorderings), a known divergence from Lucene's SloppyPhraseMatcher- /// (see #830 and `index.rs::phrase_walk`).+ /// Ordered phrase match with optional slop. `slop` is Lucene+ /// SloppyPhraseMatcher move-distance: forward gaps cost their width and+ /// transpositions (reorderings) cost 2 — `"a b"~2` matches a doc reading+ /// `b a` (#830; evaluator: `xerj_fts::search::phrase_positions_match`).MatchPhrase {field: String,query: String,diff --git a/engine/crates/xerj-query/src/planner.rs b/engine/crates/xerj-query/src/planner.rsindex f0a6f241d..339b516bb 100644--- a/engine/crates/xerj-query/src/planner.rs+++ b/engine/crates/xerj-query/src/planner.rs@@ -62,8 +62,9 @@ pub enum ExecutionPlan {field: String,tokens: Vec<String>,require_all: bool,- /// Phrase mode — tokens must appear IN ORDER, with `slop` forward gaps- /// permitted between them (NOT transpositions/reorderings; see #830).+ /// Phrase mode — tokens must appear within `slop` under Lucene+ /// move-distance semantics: forward gaps cost their width,+ /// transpositions/reorderings cost 2 (#830).phrase: bool,slop: u32,cost: f64,