Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(fts): match_phrase slop admits transpositions at Lucene cost 2 (#830) (#882)

xerj-org66d5a2274a8e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 17ecffa01..063714f4e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **`match_phrase` `slop` now admits transposed terms at Lucene's cost of 2**
+ ([#830](https://github.com/xerj-org/xerj/issues/830)). The sloppy-phrase
+ walk was in-order only — each next term matched strictly after the
+ previous one — so a reordered pair never matched at ANY slop:
+ `{"match_phrase":{"t":{"query":"quick brown","slop":2}}}` returned zero
+ hits on a document reading `brown quick`, where Lucene/ES match it at
+ distance 2 (`SloppyPhraseMatcher`'s own javadoc example). The evaluator is
+ now the Lucene move-distance semantics — pick one document position per
+ phrase term; the distance is the span of the positions after subtracting
+ each term's query offset — implemented once
+ (`xerj_fts::search::phrase_positions_match`) and called by both the segment
+ positional clause and the engine's memtable/stored-scan walk
+ (`phrase_walk`), so slop is evaluated identically on either side of a
+ flush. `match_phrase_prefix` and `multi_match` phrase/phrase_prefix go
+ through the same evaluator and gain the same behavior. Documents that
+ matched before still match — an in-order pick has strictly increasing
+ positions, so its span telescopes to exactly the old summed-gaps value —
+ and that is checked rather than assumed, by an exhaustive test over every
+ document of length <= 5 and every phrase of length <= 3 from a 3-symbol
+ alphabet at slop 0..3, comparing against both the old walk and a
+ brute-force reference. A repeated phrase term still needs as many DISTINCT
+ document positions as it has slots (`"a a"` does not match a doc holding
+ one `a`).
+
- **Wrapping a query in a one-clause `bool` no longer changes its `_score` or
its ranking** ([#399](https://github.com/xerj-org/xerj/issues/399)).
`{"bool":{"must":[X]}}` and bare `X` are the same query — Lucene's
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 0814d574d..21a9f9d79 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -37476,20 +37476,15 @@ fn phrase_tokens(text: &str) -> Vec<String> {
.analyze_to_terms(text)
}
-/// True when `query_tokens` occur in `field_tokens` in order with at most
-/// `slop` intervening positions in total — the stored-scan twin of
-/// `xerj_fts::search`'s `phrase_positions_match`, which evaluates the same
+/// True when `query_tokens` occur in `field_tokens` within `slop` under
+/// Lucene sloppy-phrase semantics — the stored-scan twin of
+/// `xerj_fts::search::phrase_positions_match`, which evaluates the same
/// predicate over the segment's real term positions.
///
-/// `slop == 0` is exact adjacency (a window compare); `slop > 0` anchors on
-/// EVERY occurrence of the first term and walks each later term to its
-/// earliest position after the previous match, summing the gaps — the same
-/// greedy walk `phrase_positions_match` runs over segment positions.
-///
-/// Anchoring on every occurrence is load-bearing: the previous stored-scan
-/// walk anchored only on the FIRST occurrence of the leading term, so
-/// `[a, x, x, x, a, b]` rejected the slop-1 phrase `a b` that both ES and
-/// the segment path accept (anchor at the second `a`).
+/// `slop == 0` is exact adjacency; `slop > 0` is an edit distance in
+/// single-position term moves (transpositions cost 2 — `"a b"~2` matches a
+/// doc reading `b a`; see `phrase_positions_match`, which closed #830's
+/// in-order-only divergence for both arms at once).
fn phrase_positions_in_tokens(field_tokens: &[String], query_tokens: &[String], slop: u32) -> bool {
phrase_walk(field_tokens, query_tokens, slop, false)
}
@@ -37502,23 +37497,25 @@ fn phrase_positions_in_tokens(field_tokens: &[String], query_tokens: &[String],
/// query per expansion, and "some expansion term sits here" is the same
/// predicate as "this token starts with the prefix".
///
-/// Keeping one walk (rather than two look-alike ones) is deliberate: the
-/// two evaluators must not be able to drift apart the way the slop-0-only
-/// prefix walk had already drifted from the sloppy phrase walk.
+/// It materialises one token-index position list per query slot and hands
+/// them to `xerj_fts::search::phrase_positions_match` — the SAME function
+/// the segment positional clause runs over postings positions. Sharing the
+/// evaluator (rather than mirroring it) is deliberate, and stronger than the
+/// previous arrangement of two hand-synchronised walks: slop cannot be
+/// evaluated differently on the two sides (#218/#222/#230 regression class),
+/// and the #830 fix — Lucene `SloppyPhraseMatcher` distance semantics, where
+/// slop admits transpositions at cost 2 instead of the old in-order-only gap
+/// walk — lands in both arms at once.
///
-/// KNOWN DIVERGENCE FROM ES, not closed by #230: this walk is **in-order
-/// only**. Lucene's sloppy phrase admits a transposition at a distance
-/// cost — its own javadoc: «for query "a b"~2, a document "x a b a y" can
-/// be matched twice: once for "a b" (distance=0), and once for "b a"
-/// (distance=2)» (`lucene/core/.../search/SloppyPhraseMatcher.java`, class
-/// javadoc; Apache-2.0) — so ES answers `match_phrase {"query": "policy
-/// merge", "slop": 2}` on a document reading `merge policy`, and XERJ
-/// answers it with zero hits. That is the pre-existing semantics of
-/// `xerj_fts::search::phrase_positions_match`, which the segment side has
-/// always used and which this walk deliberately mirrors: matching ES here
-/// means changing BOTH evaluators together, and mirroring the existing one
-/// is what keeps the hit set flush-invariant today. Measured, not assumed
-/// (5-doc index, `"policy merge"` slop 2 and slop 3 → `[]` in both states).
+/// What is shared is the EVALUATOR, not the position model, so this is not
+/// flush-invariance "by construction": this arm re-analyses stored text with
+/// the standard analyzer and feeds token INDICES, while the segment arm
+/// feeds indexed postings positions. A custom analyzer (synonym
+/// co-location, `position_increment_gap`) can therefore still hand the two
+/// arms different position lists; multi-valued fields are handled per
+/// element (#332). The tests in
+/// `tests/match_phrase_slop_transposition.rs` pin the agreement for the
+/// standard-analyzer cases that this planner actually routes both ways.
fn phrase_walk(
field_tokens: &[String],
query_tokens: &[String],
@@ -37532,56 +37529,30 @@ fn phrase_walk(
return false;
}
let n = query_tokens.len();
- let matches_at = |idx: usize, qi: usize| -> bool {
- let ft = &field_tokens[idx];
- if last_is_prefix && qi == n - 1 {
- ft.starts_with(query_tokens[qi].as_str())
- } else {
- ft == &query_tokens[qi]
- }
- };
- // Exact-adjacency fast path for the whole-term case (the common one).
- if slop == 0 && !last_is_prefix {
- return field_tokens.windows(n).any(|w| w == query_tokens);
- }
- for start in 0..field_tokens.len() {
- if !matches_at(start, 0) {
- continue;
- }
- let mut current = start;
- let mut total_gaps: u32 = 0;
- let mut ok = true;
- for qi in 1..n {
- // Earliest match after `current` minimises the running gap sum,
- // so the greedy walk is optimal for an in-order phrase — the
- // same walk `xerj_fts::search::phrase_positions_match` runs over
- // segment positions.
- match (current + 1..field_tokens.len()).find(|&i| matches_at(i, qi)) {
- Some(next) => {
- total_gaps += (next - current - 1) as u32;
- if total_gaps > slop {
- ok = false;
- break;
- }
- current = next;
- }
- None => {
- ok = false;
- break;
- }
+ // Per-query-slot ascending token-index lists — the memtable analogue of
+ // the segment's per-term postings position lists.
+ let mut positions: Vec<Vec<u32>> = vec![Vec::new(); n];
+ for (idx, ft) in field_tokens.iter().enumerate() {
+ for (qi, qt) in query_tokens.iter().enumerate() {
+ let hit = if last_is_prefix && qi == n - 1 {
+ ft.starts_with(qt.as_str())
+ } else {
+ ft == qt
+ };
+ if hit {
+ positions[qi].push(idx as u32);
}
}
- if ok {
- return true;
- }
}
- false
+ let lists: Vec<&[u32]> = positions.iter().map(|v| v.as_slice()).collect();
+ xerj_fts::search::phrase_positions_match(&lists, slop)
}
-/// True when the leading `query_tokens[..n-1]` form an in-order phrase whose
+/// True when the leading `query_tokens[..n-1]` form a phrase whose
/// next position starts with `query_tokens[n-1]`, with at most `slop`
-/// intervening positions in total — ES `match_phrase_prefix` semantics over
-/// the token stream. A single token degrades to "any token starts with it".
+/// (Lucene move-distance semantics) — ES `match_phrase_prefix` semantics
+/// over the token stream. A single token degrades to "any token starts
+/// with it".
///
/// This is TERM-level, not substring-level: `merge poli` matches the token
/// stream `[merge, policy]` (so it matches raw text `merge, policy` too),
diff --git a/engine/crates/xerj-engine/tests/match_phrase_slop_transposition.rs b/engine/crates/xerj-engine/tests/match_phrase_slop_transposition.rs
new file mode 100644
index 000000000..a5ca841a3
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/match_phrase_slop_transposition.rs
@@ -0,0 +1,262 @@
+//! Regression tests for issue #830: `match_phrase` `slop` must follow
+//! Lucene `SloppyPhraseMatcher` move-distance semantics — a transposed
+//! (reordered) pair costs 2 — instead of the old in-order-only gap walk,
+//! under which a reordering never matched at ANY slop.
+//!
+//! Lucene's own class javadoc: for query `"a b"~2`, a document `x a b a y`
+//! matches once as `a b` (distance 0) and once as `b a` (distance 2), so
+//! ES answers `{"match_phrase": {"t": {"query": "quick brown", "slop": 2}}}`
+//! on a document reading `brown quick` — and pre-fix XERJ answered it with
+//! zero hits at slop 2, 3, 4, ….
+//!
+//! Every case is asserted BOTH pre-flush and post-flush, which pins
+//! flush-invariance (#218/#222/#230 regression class) as well as the
+//! semantics. Note what the post-flush half does and does not exercise:
+//! the planner keeps single-field `match_phrase`/`match_phrase_prefix` with
+//! `slop > 0` on the STORED SCAN ("slop>0 and non-standard analyzers keep
+//! the stored scan", index.rs), so those rows re-run `phrase_walk` after the
+//! flush rather than the segment positional clause. The segment arm is
+//! covered directly by the xerj-fts unit tests
+//! (`sloppy_phrase_*` in `search.rs`) and by the `multi_match` case here.
+//! Both arms call the one evaluator, `phrase_positions_match`.
+
+use std::collections::BTreeSet;
+
+use serde_json::{json, Value};
+use tempfile::TempDir;
+use xerj_common::config::Config;
+use xerj_common::types::Schema;
+use xerj_engine::{Engine, Index};
+use xerj_query::parse_request;
+
+fn make_engine(dir: &TempDir) -> Engine {
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ Engine::new(config).expect("engine::new")
+}
+
+fn req(q: Value) -> xerj_query::ast::SearchRequest {
+ parse_request(&json!({ "query": q, "size": 50 })).expect("parse_request")
+}
+
+async fn ids(idx: &Index, q: &Value) -> BTreeSet<String> {
+ idx.search(&req(q.clone()))
+ .await
+ .unwrap()
+ .hits
+ .iter()
+ .map(|h| h.id.clone())
+ .collect()
+}
+
+fn set(ids: &[&str]) -> BTreeSet<String> {
+ ids.iter().map(|s| s.to_string()).collect()
+}
+
+/// Run every case against the memtable, flush once, run every case again
+/// against the segment, and assert the hit set is the expected one in both
+/// states — semantics AND flush-invariance in one assertion.
+async fn assert_both_states(idx: &std::sync::Arc<Index>, cases: &[(Value, &[&str], &str)]) {
+ for (q, exp, label) in cases {
+ let pre = ids(idx, q).await;
+ assert_eq!(
+ pre,
+ set(exp),
+ "{label}: PRE-flush (memtable) hit set for {q}"
+ );
+ }
+ idx.flush().await.unwrap();
+ for (q, exp, label) in cases {
+ let post = ids(idx, q).await;
+ assert_eq!(
+ post,
+ set(exp),
+ "{label}: POST-flush (segment) hit set for {q}"
+ );
+ }
+}
+
+/// Doc "adj" holds the phrase in order and adjacent; "rev" holds it
+/// TRANSPOSED (the #830 case); "gap" holds it in order with one intervening
+/// token; "ctrl" must never match.
+async fn seed(engine: &Engine, name: &str) -> std::sync::Arc<Index> {
+ engine.create_index(name, Schema::empty()).unwrap();
+ let idx = engine.get_index(name).unwrap();
+ for (id, text) in [
+ ("adj", "quick brown fox"),
+ ("rev", "brown quick fox"),
+ ("gap", "quick lazy brown"),
+ ("ctrl", "the lazy dog"),
+ ] {
+ idx.index_document(Some(id.into()), json!({ "t": text }))
+ .await
+ .unwrap();
+ }
+ idx
+}
+
+/// `match_phrase` "quick brown": slop 0 → adjacency only; slop 1 → the
+/// forward gap joins; slop 2+ → the transposition joins (cost 2). The
+/// slop-1 rows double as the negative case: the transposition must NOT be
+/// admitted below 2, and `ctrl` never matches.
+#[tokio::test]
+async fn match_phrase_slop_admits_transposition_at_two() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ let idx = seed(&engine, "phrase_transposition").await;
+
+ let q =
+ |slop: u32| json!({ "match_phrase": { "t": { "query": "quick brown", "slop": slop } } });
+ let cases: Vec<(Value, &[&str], &str)> = vec![
+ (q(0), &["adj"], "slop 0: adjacency only"),
+ (
+ q(1),
+ &["adj", "gap"],
+ "slop 1: forward gap yes, transposition no",
+ ),
+ (
+ q(2),
+ &["adj", "gap", "rev"],
+ "slop 2: transposition matches",
+ ),
+ (
+ q(3),
+ &["adj", "gap", "rev"],
+ "slop 3: superset stays stable",
+ ),
+ ];
+ assert_both_states(&idx, &cases).await;
+}
+
+/// Same semantics through `match_phrase_prefix` (the `last_is_prefix` arm of
+/// the shared walk; the segment side evaluates it via per-expansion sloppy
+/// phrase queries): "quick bro" finds `bro*` transposed against "quick" at
+/// slop 2 and not below.
+#[tokio::test]
+async fn match_phrase_prefix_slop_admits_transposition_at_two() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ let idx = seed(&engine, "phrase_prefix_transposition").await;
+
+ let q = |slop: u32| json!({ "match_phrase_prefix": { "t": { "query": "quick bro", "slop": slop } } });
+ let cases: Vec<(Value, &[&str], &str)> = vec![
+ (q(0), &["adj"], "prefix slop 0: adjacency only"),
+ (
+ q(1),
+ &["adj", "gap"],
+ "prefix slop 1: forward gap yes, transposition no",
+ ),
+ (
+ q(2),
+ &["adj", "gap", "rev"],
+ "prefix slop 2: transposition matches",
+ ),
+ ];
+ assert_both_states(&idx, &cases).await;
+}
+
+/// The `multi_match` phrase type lowers to the same per-field phrase
+/// predicate — one case pins that arm too.
+#[tokio::test]
+async fn multi_match_phrase_slop_admits_transposition_at_two() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ let idx = seed(&engine, "multi_match_transposition").await;
+
+ let q = |slop: u32| json!({ "multi_match": { "query": "quick brown", "fields": ["t"], "type": "phrase", "slop": slop } });
+ let cases: Vec<(Value, &[&str], &str)> = vec![
+ (
+ q(1),
+ &["adj", "gap"],
+ "multi_match phrase slop 1: no transposition",
+ ),
+ (
+ q(2),
+ &["adj", "gap", "rev"],
+ "multi_match phrase slop 2: transposition matches",
+ ),
+ ];
+ assert_both_states(&idx, &cases).await;
+}
+
+/// Doc "twin" repeats a phrase term; "single" holds it only once. The
+/// review of #882 found that the first minimal-window sweep lost repeated
+/// terms: `"i said no no"~1` stopped matching `i said uh no no` at every
+/// slop, because on a collision it advanced the pointer at the minimum
+/// adjusted position (a non-colliding, single-occurrence slot) and exhausted
+/// it before ever trying the assignment that separates the two `no`s.
+async fn seed_repeats(engine: &Engine, name: &str) -> std::sync::Arc<Index> {
+ engine.create_index(name, Schema::empty()).unwrap();
+ let idx = engine.get_index(name).unwrap();
+ for (id, text) in [
+ ("twin", "i said uh no no"),
+ ("single", "i said no thanks"),
+ ("noway", "no way no no"),
+ ("ctrl", "the lazy dog"),
+ ] {
+ idx.index_document(Some(id.into()), json!({ "t": text }))
+ .await
+ .unwrap();
+ }
+ idx
+}
+
+/// A repeated phrase term must not cost a document its match. `"i said no
+/// no"` needs two DISTINCT `no` tokens: "twin" has them (adjusted 0,0,1,1 —
+/// span 1), "single" has one `no` and must never match at any slop.
+#[tokio::test]
+async fn match_phrase_slop_repeated_term_still_matches() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ let idx = seed_repeats(&engine, "phrase_repeats").await;
+
+ let q =
+ |slop: u32| json!({ "match_phrase": { "t": { "query": "i said no no", "slop": slop } } });
+ let cases: Vec<(Value, &[&str], &str)> = vec![
+ (
+ q(1),
+ &["twin"],
+ "slop 1: one intervening token, two distinct 'no' tokens",
+ ),
+ (q(2), &["twin"], "slop 2: still just the doc with two 'no's"),
+ (
+ q(3),
+ &["twin"],
+ "slop 3: 'single' has only one 'no' — never",
+ ),
+ (q(10), &["twin"], "slop 10: a repeat needs a second token"),
+ ];
+ assert_both_states(&idx, &cases).await;
+}
+
+/// Every slot the same term: `"no no no"~1` on "no way no no"
+/// (no@[0,2,3] — pick 0,2,3, adjusted 0,1,1, span 1).
+#[tokio::test]
+async fn match_phrase_slop_all_slots_repeated() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ let idx = seed_repeats(&engine, "phrase_repeats_all").await;
+
+ let q = |slop: u32| json!({ "match_phrase": { "t": { "query": "no no no", "slop": slop } } });
+ let cases: Vec<(Value, &[&str], &str)> = vec![
+ (q(1), &["noway"], "slop 1: three distinct 'no' positions"),
+ (q(3), &["noway"], "slop 3: superset stays stable"),
+ ];
+ assert_both_states(&idx, &cases).await;
+}
+
+/// The same defect through `multi_match` — the arm that DOES reach the
+/// segment positional clause after a flush.
+#[tokio::test]
+async fn multi_match_phrase_slop_repeated_term_still_matches() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ let idx = seed_repeats(&engine, "multi_match_repeats").await;
+
+ let q = |slop: u32| json!({ "multi_match": { "query": "i said no no", "fields": ["t"], "type": "phrase", "slop": slop } });
+ let cases: Vec<(Value, &[&str], &str)> = vec![
+ (q(1), &["twin"], "multi_match phrase slop 1: repeat matches"),
+ (q(3), &["twin"], "multi_match phrase slop 3: repeat matches"),
+ ];
+ assert_both_states(&idx, &cases).await;
+}
diff --git a/engine/crates/xerj-engine/tests/multi_match_phrase_positions.rs b/engine/crates/xerj-engine/tests/multi_match_phrase_positions.rs
index ba030e741..6b8711bad 100644
--- a/engine/crates/xerj-engine/tests/multi_match_phrase_positions.rs
+++ b/engine/crates/xerj-engine/tests/multi_match_phrase_positions.rs
@@ -285,26 +285,18 @@ async fn phrase_slop_is_honoured() {
.await;
}
-/// PINS A KNOWN DIVERGENCE FROM ES, so nothing in this PR can be read as
-/// claiming it away: XERJ's sloppy phrase is **in-order only**, at every
-/// entry point and in both states.
-///
-/// Lucene's is not. `SloppyPhraseMatcher`'s class javadoc: «for query "a
-/// b"~2, a document "x a b a y" can be matched twice: once for "a b"
-/// (distance=0), and once for "b a" (distance=2)» — so ES answers
-/// `{"query": "policy merge", "slop": 2}` with docs 1 and 2, and XERJ
-/// answers with none. That is the pre-existing behaviour of
-/// `xerj_fts::search::phrase_positions_match` (which single-field
-/// `match_phrase` has always used) and the stored-scan walk added by #230
-/// mirrors it deliberately, because the two evaluators disagreeing is the
-/// flush-variance class this PR exists to remove.
-///
-/// It is NOT a regression: before #230 the `multi_match` phrase arms tested
-/// raw-substring containment, which does not match a transposition either.
-/// When the walk learns transpositions, both evaluators change together and
-/// this test flips — that is the point of pinning it.
+/// The former in-order-only divergence pin, FLIPPED by the #830 fix — its
+/// own doc said «when the walk learns transpositions, both evaluators
+/// change together and this test flips». XERJ's sloppy phrase now follows
+/// Lucene `SloppyPhraseMatcher` move-distance: `SloppyPhraseMatcher`'s
+/// class javadoc — «for query "a b"~2, a document "x a b a y" can be
+/// matched twice: once for "a b" (distance=0), and once for "b a"
+/// (distance=2)» — so ES answers `{"query": "policy merge", "slop": 2}`
+/// with docs 1 and 2 (both read `merge policy`), and XERJ now gives the
+/// same answer, at every entry point and in both states. The dedicated
+/// slop-1 negative cases live in `match_phrase_slop_transposition.rs`.
#[tokio::test]
-async fn sloppy_phrase_is_in_order_only_unlike_lucene() {
+async fn sloppy_phrase_admits_transpositions_like_lucene() {
let dir = TempDir::new().unwrap();
let engine = make_engine(&dir);
let idx = seed(&engine, "mmpp_inorder").await;
@@ -315,16 +307,16 @@ async fn sloppy_phrase_is_in_order_only_unlike_lucene() {
(
json!({"multi_match": {"query": "policy merge", "fields": ["body"],
"type": "phrase", "slop": 2}}),
- &[],
- "transposed phrase, slop 2: ES matches, XERJ does not",
+ &["1", "2"],
+ "transposed phrase, slop 2: matches like ES (cost 2)",
),
(
json!({"match_phrase": {"body": {"query": "policy merge", "slop": 3}}}),
- &[],
+ &["1", "2"],
"transposed phrase, slop 3, single-field spelling: same answer",
),
- // The in-order control, so the case above cannot pass by
- // accident (e.g. if slop stopped being honoured at all).
+ // The in-order control at the same slop, so the cases above
+ // cannot pass by slop being ignored altogether.
(
json!({"multi_match": {"query": "log policy", "fields": ["body"],
"type": "phrase", "slop": 2}}),
diff --git a/engine/crates/xerj-fts/src/lib.rs b/engine/crates/xerj-fts/src/lib.rs
index f3d4cd533..f0c3d4560 100644
--- a/engine/crates/xerj-fts/src/lib.rs
+++ b/engine/crates/xerj-fts/src/lib.rs
@@ -84,5 +84,6 @@ pub use postings::{
};
pub use search::{
- search_segments, BoolQuery, FtsSearcher, PhraseQuery, PrefixQuery, Query, ScoredHit, TermQuery,
+ phrase_positions_match, search_segments, BoolQuery, FtsSearcher, PhraseQuery, PrefixQuery,
+ Query, ScoredHit, TermQuery,
};
diff --git a/engine/crates/xerj-fts/src/search.rs b/engine/crates/xerj-fts/src/search.rs
index 8012efcdd..1a2545ab1 100644
--- a/engine/crates/xerj-fts/src/search.rs
+++ b/engine/crates/xerj-fts/src/search.rs
@@ -169,13 +169,15 @@ impl TermQuery {
}
}
-/// An ordered phrase — terms must appear in the given order with adjacent positions.
+/// A phrase — terms must appear with adjacent positions in the given order
+/// (`slop == 0`, the default), or within `slop` under Lucene move-distance
+/// semantics (transpositions cost 2; see `phrase_positions_match`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PhraseQuery {
pub field: String,
/// Pre-analyzed phrase terms in order.
pub terms: Vec<String>,
- /// Allowed number of intervening positions (slop).
+ /// Allowed Lucene move-distance (slop).
#[serde(default)]
pub slop: u32,
#[serde(default = "default_boost")]
@@ -753,10 +755,10 @@ impl FtsSearcher {
'doc: for &doc_id in term_maps[min_idx].0.keys() {
// Gather each term's positions for this doc, in TERM ORDER (raw —
// no offset games; the matcher below handles adjacency + slop).
- let mut all_positions: Vec<&Vec<u32>> = Vec::with_capacity(pq.terms.len());
+ let mut all_positions: Vec<&[u32]> = Vec::with_capacity(pq.terms.len());
for (map, _) in &term_maps {
match map.get(&doc_id) {
- Some((_, positions)) => all_positions.push(positions),
+ Some((_, positions)) => all_positions.push(positions.as_slice()),
None => continue 'doc, // term absent from this doc
}
}
@@ -1479,18 +1481,71 @@ pub fn search_segments(
// ── Phrase matching helper ────────────────────────────────────────────────────
/// Returns `true` if the per-term **raw** position lists (in TERM ORDER)
-/// contain an in-order occurrence of the phrase within `slop`.
+/// contain an occurrence of the phrase within `slop` — Lucene sloppy-phrase
+/// semantics, transpositions included.
///
/// `all_positions[i]` is the ascending list of positions at which the i-th
/// phrase term occurs in one document.
///
/// - `slop == 0`: exact adjacency. There must be a start `p` such that term i
/// occupies exactly `p + i` for every i (start-position intersection).
-/// - `slop > 0`: greedy in-order walk — each next term is matched at the
-/// earliest position `> ` the previous match, and the summed gaps between
-/// adjacent matched terms must not exceed `slop` (the semantics the engine's
-/// stored-scan `MatchPhrase` uses).
-fn phrase_positions_match(all_positions: &[&Vec<u32>], slop: u32) -> bool {
+/// - `slop > 0`: Lucene `SloppyPhraseMatcher` semantics. Slop is an edit
+/// distance in single-position term MOVES: pick one document position per
+/// phrase term, and the match distance is the span of the ADJUSTED
+/// positions (document position minus the term's query offset), i.e.
+/// `max(p_i - i) - min(p_i - i)`; the phrase matches when some choice keeps
+/// that distance `<= slop`. An in-order pick has strictly increasing
+/// `p_i`, so `p_i - i` is non-decreasing and the span telescopes to
+/// `(p_last - p_first) - (n-1)` — exactly the summed-gaps value the
+/// pre-#830 walk measured — and its positions are automatically distinct.
+/// Every phrase that walk matched therefore still matches here, which is
+/// a property of the DECISION PROCEDURE being complete, not of the cost
+/// function alone: an incomplete search over assignments can lose an
+/// in-order match it never tries (it did, for repeated terms, before this
+/// evaluator got its SDR window test). Unlike that walk this also admits
+/// REORDERED terms — an out-of-order pair costs at least 2, so `"a b"~2`
+/// matches a document reading `b a` (Lucene's class javadoc example: for
+/// query `"a b"~2`, document `x a b a y` matches once at distance 0 and
+/// once, as `b a`, at distance 2). Issue #830.
+///
+/// Adapted (algorithm, not code) from Lucene's `SloppyPhraseMatcher`
+/// (lucene/core/src/java/org/apache/lucene/search/SloppyPhraseMatcher.java,
+/// Apache-2.0): `nextMatch` (:205-:237) pops the minimum `PhrasePositions`
+/// from a priority queue ordered by adjusted position (`pp.position` is set
+/// to `position - offset` in PhrasePositions.java `nextPosition`, :53-:59)
+/// and measures `matchLength = end - pp.position` — max minus min adjusted
+/// position. That plain "advance the minimum" walk is only complete once
+/// REPEATS have been resolved: a repeated query term needs as many DISTINCT
+/// document positions as it has slots, and Lucene therefore advances the
+/// colliding repeat rather than the current minimum (`advanceRpts`, :317).
+/// This implementation reaches the same completeness differently, because
+/// it has no term identity to group by (see the distinctness note below):
+/// it sweeps every candidate window floor `lo` — each adjusted position is
+/// one — and asks whether the window `[lo, lo + slop]` admits a choice of
+/// one position per slot with all positions DISTINCT, i.e. a system of
+/// distinct representatives, decided by bipartite matching. The sweep is
+/// complete: if any assignment has span `<= slop`, then the window floored
+/// at that assignment's own minimum adjusted position contains all of it.
+///
+/// Distinctness is enforced pairwise over ALL slots rather than only over
+/// repeated terms. Under the standard analyzer the two are the same thing
+/// (one token per position, so only a repeated term can collide). It is
+/// stricter than Lucene on a field whose analyzer co-locates DIFFERENT
+/// terms — XERJ's `SynonymFilter` emits synonyms at the same position —
+/// where Lucene would let two distinct query terms share one position and
+/// this does not. That is a deliberate, conservative divergence: it can
+/// only withhold a match, never invent one, and it is what the pre-#830
+/// walk did too (it required strictly increasing positions).
+///
+/// Both arms share this evaluator: the segment positional clause calls it
+/// with real postings positions, and the engine's stored-scan/memtable arm
+/// (`phrase_walk` in xerj-engine's `index.rs`) materialises token-index
+/// lists and calls it too, so slop is evaluated identically on both sides.
+/// What they do NOT share is the position model — the memtable arm
+/// re-analyses text with the standard analyzer while the segment arm reads
+/// indexed positions — so a custom analyzer (synonyms,
+/// `position_increment_gap`) can still hand the two arms different inputs.
+pub fn phrase_positions_match(all_positions: &[&[u32]], slop: u32) -> bool {
if all_positions.iter().any(|p| p.is_empty()) {
return false;
}
@@ -1517,28 +1572,161 @@ fn phrase_positions_match(all_positions: &[&Vec<u32>], slop: u32) -> bool {
return false;
}
- // Sloppy phrase: greedy earliest-match walk, accumulating the gaps.
- 'outer: for &start_pos in all_positions[0] {
- let mut current_pos = start_pos;
- let mut total_gaps: u32 = 0;
- for positions in &all_positions[1..] {
- // Earliest position strictly after the previous matched term.
- match positions.iter().copied().find(|&p| p > current_pos) {
- Some(next_pos) => {
- total_gaps += next_pos - current_pos - 1;
- if total_gaps > slop {
- continue 'outer;
- }
- current_pos = next_pos;
- }
- None => continue 'outer,
+ sloppy_phrase_match(all_positions, slop)
+}
+
+/// Is there a choice of one document position per phrase slot, all DISTINCT,
+/// whose adjusted positions (`p_i - i`) span at most `slop`?
+///
+/// Sweeps candidate window floors in ascending order. Every adjusted
+/// position is a candidate floor, and the optimal window is floored at the
+/// minimum adjusted position of its own assignment, so the sweep misses
+/// nothing. For a floor `lo` each slot's admissible positions are the
+/// contiguous run `L_i ∩ [lo + i, lo + slop + i]`, tracked with pointers
+/// that only ever move forward as `lo` grows, so the pointer work over the
+/// whole sweep is linear in the total number of positions.
+///
+/// Feasibility inside one window is a system-of-distinct-representatives
+/// test. It only needs the first `n` admissible positions of each slot: a
+/// slot with `n` or more candidates can always be re-pointed at a free one
+/// (the other `n - 1` slots occupy at most `n - 1` positions), so truncating
+/// there cannot turn a feasible window infeasible.
+fn sloppy_phrase_match(all_positions: &[&[u32]], slop: u32) -> bool {
+ let n = all_positions.len();
+ let slop = i64::from(slop);
+ let adj = |i: usize, k: usize| i64::from(all_positions[i][k]) - i as i64;
+
+ // Cursors generating the candidate floors, plus the [lo_ptr, hi_ptr)
+ // admissible run per slot. All three only move forward.
+ let mut cursor = vec![0usize; n];
+ let mut lo_ptr = vec![0usize; n];
+ let mut hi_ptr = vec![0usize; n];
+ // Scratch reused across windows so the sweep does not allocate per step.
+ let mut cand: Vec<&[u32]> = Vec::with_capacity(n);
+ let mut pool: Vec<u32> = Vec::with_capacity(n * n);
+ let mut owner: Vec<usize> = Vec::with_capacity(n * n);
+ let mut seen: Vec<bool> = Vec::with_capacity(n * n);
+
+ loop {
+ // Next candidate floor: the smallest adjusted position not yet used
+ // as a floor. Exhausted cursors mean no window is left.
+ let mut lo = i64::MAX;
+ for i in 0..n {
+ if cursor[i] < all_positions[i].len() {
+ lo = lo.min(adj(i, cursor[i]));
+ }
+ }
+ if lo == i64::MAX {
+ return false;
+ }
+ for i in 0..n {
+ while cursor[i] < all_positions[i].len() && adj(i, cursor[i]) <= lo {
+ cursor[i] += 1;
}
}
- if total_gaps <= slop {
+
+ // Admissible run per slot for the window [lo, lo + slop].
+ let hi = lo + slop;
+ let mut window_full = true;
+ for i in 0..n {
+ let list = all_positions[i];
+ let first = lo + i as i64;
+ let last = hi + i as i64;
+ while lo_ptr[i] < list.len() && i64::from(list[lo_ptr[i]]) < first {
+ lo_ptr[i] += 1;
+ }
+ if hi_ptr[i] < lo_ptr[i] {
+ hi_ptr[i] = lo_ptr[i];
+ }
+ while hi_ptr[i] < list.len() && i64::from(list[hi_ptr[i]]) <= last {
+ hi_ptr[i] += 1;
+ }
+ if hi_ptr[i] == lo_ptr[i] {
+ window_full = false;
+ }
+ }
+ if !window_full {
+ continue;
+ }
+
+ // Fast path — and the whole story whenever no term repeats: if the
+ // earliest admissible position of every slot is a different document
+ // position, that assignment is already a match.
+ let earliest_are_distinct = (0..n).all(|i| {
+ (i + 1..n).all(|j| all_positions[i][lo_ptr[i]] != all_positions[j][lo_ptr[j]])
+ });
+ if earliest_are_distinct {
+ return true;
+ }
+
+ // Repeats collide here. Decide the window exactly instead of
+ // guessing which pointer to advance: an SDR over the truncated
+ // candidate runs.
+ cand.clear();
+ for i in 0..n {
+ let end = hi_ptr[i].min(lo_ptr[i].saturating_add(n));
+ cand.push(&all_positions[i][lo_ptr[i]..end]);
+ }
+ if window_admits_distinct_choice(&cand, &mut pool, &mut owner, &mut seen) {
return true;
}
}
+}
+
+/// Bipartite matching (Kuhn's augmenting-path algorithm) of phrase slots to
+/// document positions: `true` when every slot can be given a position from
+/// its own candidate list with no position used twice. Both sides are
+/// bounded by the phrase term count, so this is a handful of steps.
+fn window_admits_distinct_choice(
+ cand: &[&[u32]],
+ pool: &mut Vec<u32>,
+ owner: &mut Vec<usize>,
+ seen: &mut Vec<bool>,
+) -> bool {
+ let n = cand.len();
+ pool.clear();
+ for c in cand {
+ pool.extend_from_slice(c);
+ }
+ pool.sort_unstable();
+ pool.dedup();
+ if pool.len() < n {
+ return false;
+ }
+ owner.clear();
+ owner.resize(pool.len(), usize::MAX);
+ for slot in 0..n {
+ seen.clear();
+ seen.resize(pool.len(), false);
+ if !augment_slot(slot, cand, pool, seen, owner) {
+ return false;
+ }
+ }
+ true
+}
+/// One augmenting-path step for `window_admits_distinct_choice`: try to seat
+/// `slot`, displacing an already-seated slot only if that slot can move.
+fn augment_slot(
+ slot: usize,
+ cand: &[&[u32]],
+ pool: &[u32],
+ seen: &mut [bool],
+ owner: &mut [usize],
+) -> bool {
+ for &position in cand[slot] {
+ let Ok(p) = pool.binary_search(&position) else {
+ continue;
+ };
+ if seen[p] {
+ continue;
+ }
+ seen[p] = true;
+ if owner[p] == usize::MAX || augment_slot(owner[p], cand, pool, seen, owner) {
+ owner[p] = slot;
+ return true;
+ }
+ }
false
}
@@ -2023,6 +2211,300 @@ mod tests {
);
}
+ /// #830: `slop` is Lucene `SloppyPhraseMatcher` move-distance, so a
+ /// transposed pair matches at slop >= 2 — the old walk was in-order-only
+ /// and never matched a reordering at ANY slop.
+ #[test]
+ fn sloppy_phrase_transposition_semantics() {
+ // Doc reading "brown quick" (brown@0, quick@1), query "quick brown".
+ let quick: &[u32] = &[1];
+ let brown: &[u32] = &[0];
+ assert!(!phrase_positions_match(&[quick, brown], 0));
+ assert!(
+ !phrase_positions_match(&[quick, brown], 1),
+ "a transposition costs 2, not 1"
+ );
+ assert!(
+ phrase_positions_match(&[quick, brown], 2),
+ "a transposition must match at slop 2"
+ );
+ assert!(phrase_positions_match(&[quick, brown], 3));
+
+ // In-order forward gap still costs its width ("quick lazy brown").
+ let quick_gap: &[u32] = &[0];
+ let brown_gap: &[u32] = &[2];
+ assert!(!phrase_positions_match(&[quick_gap, brown_gap], 0));
+ assert!(phrase_positions_match(&[quick_gap, brown_gap], 1));
+
+ // Lucene's class-javadoc example doc "x a b a y" (a@[1,3], b@2):
+ // "a b" matches exactly at slop 0, and "b a" (b@2 → a@3) at slop 0
+ // too via the in-order pair; both trivially within slop 2.
+ let a: &[u32] = &[1, 3];
+ let b: &[u32] = &[2];
+ assert!(phrase_positions_match(&[a, b], 0));
+ // NB "b a" here is satisfied by the IN-ORDER pair b@2 -> a@3, so it
+ // pins the javadoc example, not transposition cost — the transposed
+ // pair is pinned by quick/brown above, where no in-order pick exists.
+ assert!(phrase_positions_match(&[b, a], 0));
+ assert!(phrase_positions_match(&[b, a], 2));
+
+ // Adjacent transposed pair inside a 3-term phrase: doc "one three
+ // two" for query "one two three" — one@0, two@2, three@1 → adjusted
+ // span (2-1) - (1-2) = 2.
+ let one: &[u32] = &[0];
+ let two: &[u32] = &[2];
+ let three: &[u32] = &[1];
+ assert!(!phrase_positions_match(&[one, two, three], 1));
+ assert!(phrase_positions_match(&[one, two, three], 2));
+ }
+
+ /// A repeated phrase term must land on DISTINCT document positions —
+ /// one document token cannot fill two phrase slots, however large the
+ /// slop (Lucene's repeat-collision rule).
+ #[test]
+ fn sloppy_phrase_repeat_needs_distinct_positions() {
+ let a_once: &[u32] = &[3];
+ assert!(
+ !phrase_positions_match(&[a_once, a_once], 10),
+ "\"a a\" must not match a doc holding a single 'a'"
+ );
+ let a_gap: &[u32] = &[3, 5];
+ assert!(
+ !phrase_positions_match(&[a_gap, a_gap], 0),
+ "'a x a' is not an adjacent \"a a\""
+ );
+ assert!(phrase_positions_match(&[a_gap, a_gap], 1));
+ let a_adj: &[u32] = &[3, 4];
+ assert!(phrase_positions_match(&[a_adj, a_adj], 0));
+ }
+
+ /// Regression for the defect the #830 review caught: a phrase holding a
+ /// REPEATED term must keep every match the pre-#830 in-order walk gave.
+ ///
+ /// The minimal-window sweep as first written advanced the pointer at the
+ /// MINIMUM adjusted position whenever a window collided. That pointer is
+ /// usually a non-colliding slot, so the sweep walked away from the very
+ /// assignment that would separate the repeats and exhausted a
+ /// single-occurrence slot instead: `"i said no no"~1` stopped matching
+ /// `i said uh no no` at every slop. The window/SDR formulation decides
+ /// each window instead of guessing which pointer to move.
+ #[test]
+ fn sloppy_phrase_repeated_term_keeps_in_order_matches() {
+ // Doc "i said uh no no": i@0, said@1, no@[3,4]. Query "i said no no"
+ // picks no@3 and no@4 — adjusted 0,0,1,1, span 1.
+ let i: &[u32] = &[0];
+ let said: &[u32] = &[1];
+ let no: &[u32] = &[3, 4];
+ for slop in [1u32, 2, 3, 5, 10] {
+ assert!(
+ phrase_positions_match(&[i, said, no, no], slop),
+ "\"i said no no\"~{slop} must match 'i said uh no no'"
+ );
+ }
+
+ // Doc "a x b b": a@0, b@[2,3]. "a b b" is in order with one gap.
+ let a: &[u32] = &[0];
+ let b: &[u32] = &[2, 3];
+ for slop in [1u32, 2, 4] {
+ assert!(
+ phrase_positions_match(&[a, b, b], slop),
+ "\"a b b\"~{slop} must match 'a x b b'"
+ );
+ }
+
+ // Doc "no way no no": every phrase slot is the same term (no@[0,2,3]).
+ let no3: &[u32] = &[0, 2, 3];
+ for slop in [1u32, 2, 3] {
+ assert!(
+ phrase_positions_match(&[no3, no3, no3], slop),
+ "\"no no no\"~{slop} must match 'no way no no'"
+ );
+ }
+
+ // The distinctness rule still binds: three slots cannot be filled by
+ // two document tokens, at any slop.
+ let two_tokens: &[u32] = &[3, 4];
+ assert!(
+ !phrase_positions_match(&[two_tokens, two_tokens, two_tokens], 10),
+ "three repeats of one term need three distinct positions"
+ );
+ }
+
+ /// Exhaustive cross-check of the sloppy evaluator over every document of
+ /// length <= 5 and every phrase of length <= 3 from a 3-symbol alphabet,
+ /// at slop 0..=3, against two independent oracles:
+ ///
+ /// 1. brute force over all assignments (one position per slot, all
+ /// distinct, adjusted span <= slop) — catches an INCOMPLETE search,
+ /// which is exactly what the #830 review found; and
+ /// 2. the pre-#830 in-order walk, which must never match a phrase this
+ /// evaluator rejects. That is the "no previously-matching document
+ /// stops matching" claim, checked rather than asserted.
+ #[test]
+ fn sloppy_phrase_agrees_with_brute_force_and_keeps_pre_830_hits() {
+ /// The evaluator this replaced (main @ #830): greedy earliest
+ /// in-order walk accumulating the gaps.
+ fn pre_830_walk(all: &[&[u32]], slop: u32) -> bool {
+ if all.iter().any(|p| p.is_empty()) {
+ return false;
+ }
+ if all.len() == 1 {
+ return true;
+ }
+ 'outer: for &start_pos in all[0] {
+ let mut current = start_pos;
+ let mut gaps: u32 = 0;
+ for positions in &all[1..] {
+ match positions.iter().copied().find(|&p| p > current) {
+ Some(next) => {
+ gaps += next - current - 1;
+ if gaps > slop {
+ continue 'outer;
+ }
+ current = next;
+ }
+ None => continue 'outer,
+ }
+ }
+ return true;
+ }
+ false
+ }
+
+ /// Reference semantics, stated directly: some assignment of one
+ /// distinct position per slot has adjusted span <= slop (adjacency
+ /// for slop 0).
+ fn brute(all: &[&[u32]], slop: u32) -> bool {
+ fn rec(k: usize, all: &[&[u32]], pick: &mut Vec<u32>, slop: u32) -> bool {
+ if k == all.len() {
+ if slop == 0 {
+ return pick
+ .iter()
+ .enumerate()
+ .all(|(i, &p)| i64::from(p) == i64::from(pick[0]) + i as i64);
+ }
+ let adjusted = |i: usize, p: u32| i64::from(p) - i as i64;
+ let lo = pick
+ .iter()
+ .enumerate()
+ .map(|(i, &p)| adjusted(i, p))
+ .min()
+ .unwrap();
+ let hi = pick
+ .iter()
+ .enumerate()
+ .map(|(i, &p)| adjusted(i, p))
+ .max()
+ .unwrap();
+ return hi - lo <= i64::from(slop);
+ }
+ for &p in all[k] {
+ if pick.contains(&p) {
+ continue;
+ }
+ pick.push(p);
+ if rec(k + 1, all, pick, slop) {
+ return true;
+ }
+ pick.pop();
+ }
+ false
+ }
+ if all.iter().any(|p| p.is_empty()) {
+ return false;
+ }
+ if all.len() == 1 {
+ return true;
+ }
+ rec(0, all, &mut Vec::new(), slop)
+ }
+
+ fn sequences(len: usize, alphabet: u32) -> Vec<Vec<u32>> {
+ let mut out = vec![Vec::new()];
+ for _ in 0..len {
+ let mut next = Vec::new();
+ for prefix in &out {
+ for symbol in 0..alphabet {
+ let mut s = prefix.clone();
+ s.push(symbol);
+ next.push(s);
+ }
+ }
+ out = next;
+ }
+ out
+ }
+
+ let mut checked = 0u32;
+ for doc_len in 1..=5usize {
+ for doc in sequences(doc_len, 3) {
+ for query_len in 1..=3usize {
+ for query in sequences(query_len, 3) {
+ let lists: Vec<Vec<u32>> = query
+ .iter()
+ .map(|t| {
+ doc.iter()
+ .enumerate()
+ .filter(|(_, d)| *d == t)
+ .map(|(p, _)| p as u32)
+ .collect()
+ })
+ .collect();
+ let all: Vec<&[u32]> = lists.iter().map(|v| v.as_slice()).collect();
+ for slop in 0..=3u32 {
+ checked += 1;
+ let got = phrase_positions_match(&all, slop);
+ assert_eq!(
+ got,
+ brute(&all, slop),
+ "doc {doc:?} query {query:?} slop {slop}: \
+ evaluator disagrees with brute force"
+ );
+ if pre_830_walk(&all, slop) {
+ assert!(
+ got,
+ "doc {doc:?} query {query:?} slop {slop}: \
+ matched before #830 and must still match"
+ );
+ }
+ }
+ }
+ }
+ }
+ }
+ assert!(checked > 50_000, "coverage collapsed to {checked} cases");
+ }
+
+ /// #830 segment arm end-to-end: a sloppy `PhraseQuery` admits the
+ /// transposed pair at slop 2 and still rejects it at slop 0/1. Docs 0
+ /// and 1 read "… quick brown …"; the query asks for "brown quick".
+ #[test]
+ fn phrase_query_slop_transposition() {
+ let dir = TempDir::new().unwrap();
+ let searcher = setup_searcher(dir.path());
+
+ let mk = |slop: u32| {
+ let mut q = PhraseQuery::new("body", vec!["brown".to_owned(), "quick".to_owned()]);
+ q.slop = slop;
+ Query::Phrase(q)
+ };
+ assert!(searcher.search(&mk(0), 10, false).unwrap().is_empty());
+ assert!(
+ searcher.search(&mk(1), 10, false).unwrap().is_empty(),
+ "transposition costs 2 — slop 1 must not admit it"
+ );
+ let ids: Vec<u32> = searcher
+ .search(&mk(2), 10, false)
+ .unwrap()
+ .iter()
+ .map(|h| h.doc_id)
+ .collect();
+ assert!(
+ ids.contains(&0) && ids.contains(&1),
+ "docs with 'quick brown' must match the reversed query at slop 2, got {ids:?}"
+ );
+ }
+
/// Case-sensitive wildcard/fuzzy (text-field path): the whitespace analyzer
/// preserves case, so an indexed term "Rust" is matched by "R*" but NOT by
/// "r*" when `case_insensitive = false`.
diff --git a/engine/crates/xerj-query/src/ast.rs b/engine/crates/xerj-query/src/ast.rs
index 067e8625c..a6662a238 100644
--- a/engine/crates/xerj-query/src/ast.rs
+++ b/engine/crates/xerj-query/src/ast.rs
@@ -540,10 +540,10 @@ pub enum QueryNode {
minimum_should_match: Option<MinShouldMatch>,
},
- /// Ordered phrase match with optional slop. `slop` admits forward gaps
- /// between the in-order tokens ONLY — it does NOT admit transpositions
- /// (reorderings), a known divergence from Lucene's SloppyPhraseMatcher
- /// (see #830 and `index.rs::phrase_walk`).
+ /// Ordered phrase match with optional slop. `slop` is Lucene
+ /// SloppyPhraseMatcher move-distance: forward gaps cost their width and
+ /// transpositions (reorderings) cost 2 — `"a b"~2` matches a doc reading
+ /// `b a` (#830; evaluator: `xerj_fts::search::phrase_positions_match`).
MatchPhrase {
field: String,
query: String,
diff --git a/engine/crates/xerj-query/src/planner.rs b/engine/crates/xerj-query/src/planner.rs
index f0a6f241d..339b516bb 100644
--- a/engine/crates/xerj-query/src/planner.rs
+++ b/engine/crates/xerj-query/src/planner.rs
@@ -62,8 +62,9 @@ pub enum ExecutionPlan {
field: String,
tokens: Vec<String>,
require_all: bool,
- /// Phrase mode — tokens must appear IN ORDER, with `slop` forward gaps
- /// permitted between them (NOT transpositions/reorderings; see #830).
+ /// Phrase mode — tokens must appear within `slop` under Lucene
+ /// move-distance semantics: forward gaps cost their width,
+ /// transpositions/reorderings cost 2 (#830).
phrase: bool,
slop: u32,
cost: f64,