AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): gate the third sort prefilter on meta-fields too (#401)
xerj-org78a1294f0e32
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex aa6b599e9..1ad8c4c40 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -33,8 +33,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0Real sort values are only half of it: the memtable's bounded sort-candidatepath narrows the heap's input using a doc-values column keyed by the sort- field, and no such column exists for a meta-field, so it classified every- buffered document as "missing the field" and returned an arbitrary+ field, which for a meta-field is derived from `_source` rather than from the+ version map the heap now ranks on. With no such key present it classified+ every buffered document as "missing the field" and returned an arbitrary`materialisation_limit`-sized prefix. A correctly ranked page over a wrongcandidate set is still a silently wrong page, so that path — and thesegment-side sort shadow — now decline meta-fields and take the full walk.@@ -44,6 +45,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`["d1995","d1989","d1981","d1973","d1961"]` instead of`["d1999","d1998","d1997","d1996","d1995"]`.+ There is a third such prefilter, and it is the one the two gates above route+ meta-sorts *onto*: the memtable scan's pre-clone rejection+ (`memtable_primary_key_rejects`) skips a document whose primary sort key,+ read from `_source`, already loses to the full heap. Once the heap ranks on+ version-map metadata, a document carrying a `_source` key of the same name+ is compared against the wrong frontier and dropped before the heap ever sees+ it. Measured over ES-compat HTTP on 2 000 documents where only `d0000`+ carried `"_seq_no": 10000000`: `{"size":5,"sort":[{"_seq_no":"asc"}]}`+ returned `[d0001, d0002, d0003, d0004, d0005]`, omitting the true first+ document, while `"size": 3000` returned it — a silent, size-dependent+ missing document. That path now declines meta-fields too.++ **Behaviour change:** `sort` on `_seq_no` / `_version` / `_primary_term` /+ `_index` now ignores a `_source` key of the same name and always ranks on+ engine metadata. Previously it ranked on the `_source` value where one+ existed and on `null` everywhere else. This is observable because xerj —+ unlike Elasticsearch, which rejects a metadata-named key inside a document+ with a `document_parsing_exception` — accepts `{"_seq_no": 1}` in a document+ body and echoes it back in `_source`. Such a key is now inert for sorting;+ it is still stored, still returned in `_source`, and still usable for+ ordinary queries. Rejecting metadata-named `_source` keys at the API edge is+ deliberately **not** part of this change — xerj writes `_routing` into+ `_source` itself, so the edge rule needs its own design.+- **`xerj autoindex` aborted a whole run when one file declared two or more SQLtables** ([#360](https://github.com/xerj-org/xerj/issues/360)). One file canfeed several datasets — a SQL dump is one file and N tables — but the sealeddiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex ef4728ead..de5e7b310 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -14743,14 +14743,19 @@ impl Index {if sf.is_score()|| sf.is_doc_order()|| sf.field == "_id"- // An ES meta-field has no memtable dv column, so- // `sort_candidates_numeric` would classify EVERY- // buffered doc as "missing this field" and hand- // back an arbitrary `cap`-sized prefix as the- // candidate set. With #401's real `_seq_no`- // values that prefix is no longer a superset of- // the true top-cap — it would rank the wrong- // documents correctly. Take the full walk.+ // `sort_candidates_numeric` cuts the candidate set+ // by reading the sort field out of each buffered+ // doc's `_source`, but #401 resolves a meta-field+ // from the version map — so the cut and the heap+ // would rank on different data. Usually `_source`+ // has no such key at all and EVERY doc classifies+ // as "missing the field", yielding an arbitrary+ // `cap`-sized prefix; where a doc DOES carry a+ // literal `_seq_no` (xerj accepts one, see+ // `memtable_primary_key_rejects`) the cut ranks on+ // the shadowing value instead. Both give a page+ // of the wrong documents, correctly ranked. Take+ // the full walk.|| is_meta_sort_field(&sf.field)|| sf.mode != SortMode::default()|| sf.missing != SortMissing::default()@@ -23062,13 +23067,25 @@ impl Index {field: &str,seg_doc_count: u64,) -> Option<Resident<Option<Arc<Vec<(i64, u32)>>>>> {- // An ES meta-field (`_seq_no`, `_version`, …) is engine metadata, not- // a `_source` key, so no segment ever writes a dv column under that- // name. Bail BEFORE the registry insert so the publish-time warm- // doesn't burn one of its 16 slots probing a column that cannot- // exist, and so a future column that happened to share the name- // could never silently drive the candidate cut for a key the- // engine resolves from the version map instead (#401).+ // An ES meta-field (`_seq_no`, `_version`, …) is resolved from the+ // version map, never from the segment, so a dv column keyed by that+ // name must not drive the candidate cut (#401).+ //+ // Today no such column exists — but NOT because the name is+ // impossible: xerj accepts a literal `_seq_no` inside a document+ // (real ES rejects it), so it can reach `_source`. It is the dv+ // sidecar builder that drops it, skipping every `_`-prefixed source+ // key as envelope bookkeeping (the `starts_with('_')` skip in+ // `build_doc_value_columns`). That invariant lives elsewhere and+ // is not this path's to assume, so the guard is kept as the local+ // enforcement: relax the skip and the cut would silently rank on a+ // user-supplied value while the heap ranks on real metadata — the+ // memtable's identical shape, which IS reachable today, is what+ // `memtable_primary_key_rejects` had to be gated for.+ //+ // Bailing BEFORE the registry insert also stops the publish-time+ // warm burning one of its 16 slots probing a column that is not+ // written.if is_meta_sort_field(field) {return None;}@@ -37007,6 +37024,26 @@ fn memtable_primary_key_rejects(topk: &SortTopK, source: &Value) -> bool {if sf.is_score() || sf.is_doc_order() {return false;}+ // An ES meta-field is resolved by `compute_sort_values` from the version+ // map, NOT from `_source` — so deriving it here from `_source` would+ // compare the prefilter's frontier against a heap ranked on different+ // data. xerj (unlike ES, which rejects the key outright) accepts a+ // literal `_seq_no` / `_version` / `_primary_term` / `_index` inside a+ // document, so that disagreement is wire-reachable: the shadowing value+ // rejects a doc whose REAL metadata belongs on the page, and it is+ // dropped before `offer` ever sees it — silently, and only at small+ // `size`. Meta-sorts take the unaccelerated full walk (#401).+ //+ // `_id` is deliberately absent from `is_meta_sort_field` and needs no+ // gate here only because `compute_sort_values` emits it as a STRING:+ // `primary_f64_rejects` reads the heap frontier through `as_f64()`, gets+ // `None` for a string, and can never reject. Verified over the wire with+ // a dense numeric `_source._id` — the `_id desc` page is identical at+ // size 5 and size 3000. Give `_id` a numeric sort value and it becomes+ // the same defect; add it here if that ever changes.+ if is_meta_sort_field(&sf.field) {+ return false;+ }// Root-level literal key only — mirrors `get_field_value`'s fast path.// A dotted/nested primary field misses here and is admitted (correct,// just not accelerated).diff --git a/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs b/engine/crates/xerj-engine/tests/sort_on_meta_fields.rsindex 8cf4a6212..ff5e299bb 100644--- a/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs+++ b/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs@@ -18,21 +18,34 @@//! (`lucene/core/src/java/org/apache/lucene/search/SortedNumericSortField.java:49-59`)//! and `NumericComparator` compares those column values directly//! (`lucene/core/src/java/org/apache/lucene/search/comparators/NumericComparator.java:47-63`).-//! xerj has no column for a meta-field, so `Index::meta_sort_value` resolves-//! it through the same version-map accessors that populate the response-//! meta-fields (`lookup_seq_no` / `lookup_version`).+//! xerj indexes no column for a meta-field, so `Index::meta_sort_value`+//! resolves it through the same version-map accessors that populate the+//! response meta-fields (`lookup_seq_no` / `lookup_version`).//!-//! Two collectors had to agree for that to be enough, and the second one is-//! the trap: the memtable's bounded sort-candidate path-//! (`sort_candidates_numeric`) cuts the candidate set with a dv column keyed-//! by the sort field. There is no `_seq_no` column, so it classified every-//! buffered doc as "missing the field" and handed back an arbitrary-//! `materialisation_limit`-sized prefix. Real sort values over a wrong-//! candidate set is still a silently wrong page — the same defect wearing-//! the other hat — so `is_meta_sort_field` also gates that path. The same-//! gate is what fixes `_id` at the cap: `_id` already resolved to a real-//! sort value, and `sort: [{"_id":"desc"}]` over a >cap memtable STILL+//! Real sort values are NOT enough on their own, and that is what the rest of+//! this file is about. Three prefilters narrow the heap's input by deriving+//! the primary sort key from `_source`, and once the heap ranks a meta-field+//! on version-map metadata, any of them ranking on `_source` is comparing two+//! different frontiers. A correctly ranked page over a wrong candidate set is+//! still a silently wrong page — the same defect wearing the other hat — so+//! all three now decline meta-fields via `is_meta_sort_field`:+//!+//! 1. `sort_candidates_numeric` (memtable bounded candidate cut),+//! 2. `sorted_shadow_for` (segment sort shadow),+//! 3. `memtable_primary_key_rejects` (memtable pre-clone rejection) — the+//! one gates 1 and 2 route meta-sorts ONTO, missed in the first pass.+//!+//! Gate 1 is also what fixes `_id` at the cap: `_id` already resolved to a+//! real sort value, and `sort: [{"_id":"desc"}]` over a >cap memtable STILL//! returned the wrong five documents on main.+//!+//! Note the premise these gates do NOT rest on: "a meta-field name can never+//! be a `_source` key". It can — xerj accepts `{"_seq_no": 1}` in a document+//! body where ES rejects it — and the shadowing tests at the end of this file+//! exercise exactly that. Segments happen to be safe today only because+//! `build_doc_value_columns` skips every `_`-prefixed source key; the memtable+//! has no such skip, which is why gates 1 and 3 are load-bearing rather than+//! defensive.use serde_json::{json, Value};use tempfile::TempDir;@@ -303,3 +316,180 @@ async fn seq_no_sort_is_exact_after_flush() {);}}++/// A `_source` key that SHADOWS the meta-field being sorted on.+///+/// xerj — unlike ES, which rejects `_seq_no` inside a document with a+/// `document_parsing_exception` — accepts one and echoes it back in+/// `_source`, so this is wire-reachable, not a synthetic shape.+///+/// The trap is that #401 moved meta-field RESOLUTION to the version map+/// while three prefilters still derived the sort key from `_source`. Two+/// were gated with `is_meta_sort_field`; `memtable_primary_key_rejects` —+/// the pre-clone rejection on the very full-walk path the other two gates+/// FORCE meta-sorts onto — was not. A shadowing key then made the+/// prefilter's frontier and the heap's frontier disagree, and the doc was+/// dropped before `offer` ever saw it: silently, and only when `size` is+/// small enough for the heap to fill.+///+/// Before the gate, with only d0000 carrying `"_seq_no": 10000000`,+/// `sort _seq_no asc size 5` returned [d0001..d0005] — the true first+/// document missing — while `size 3000` returned it, which is what makes+/// the loss silent.+#[tokio::test]+async fn source_key_shadowing_a_meta_sort_field_drops_no_document() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ engine.create_index("shadow_sort", Schema::empty()).unwrap();+ let idx = engine.get_index("shadow_sort").unwrap();++ // > materialisation_limit (256), memtable-resident: the only path where+ // a source-derived prefilter still runs for a meta sort.+ const N: u64 = 2000;+ for i in 0..N {+ // SPARSE shadow: one doc claims a `_seq_no` far past the corpus, so+ // the prefilter ranks it last while its real `_seq_no` (0) ranks it+ // first. Every other doc is underivable and admitted, which is why+ // the page looks plausible instead of empty.+ let body = if i == 0 {+ json!({ "n": i, "_seq_no": 10_000_000 })+ } else {+ json!({ "n": i })+ };+ idx.index_document(Some(format!("d{i:04}")), body)+ .await+ .unwrap();+ }++ let res = idx.search(&sorted_req("_seq_no", "asc", 5)).await.unwrap();+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();+ let oldest: Vec<String> = (0..5).map(|i| format!("d{i:04}")).collect();+ assert_eq!(+ ids, oldest,+ "a shadowing `_source._seq_no` must not drop the true first document"+ );+ // The sort value must be the METADATA, not the shadowing source value.+ assert_eq!(+ res.hits+ .first()+ .and_then(|h| h.sort.first())+ .and_then(Value::as_u64),+ idx.lookup_seq_no("d0000"),+ "d0000 must rank and report on its real _seq_no, not `_source._seq_no`"+ );++ // Size-independence is the actual contract: the page must not change+ // when the heap is large enough that no doc is ever rejected early.+ let big = idx+ .search(&sorted_req("_seq_no", "asc", 3000))+ .await+ .unwrap();+ let big_ids: Vec<String> = big.hits.iter().take(5).map(|h| h.id.clone()).collect();+ assert_eq!(+ ids, big_ids,+ "the size-5 page must equal the first 5 of the size-3000 page"+ );+}++/// DENSE shadow: EVERY doc carries a `_source._seq_no` counting down, so the+/// prefilter's frontier is the exact inverse of the heap's — no doc is+/// "underivable" and the rejection fires on every comparison. Before the+/// gate this returned [d1995, d1989, d1981, d1973, d1961]: the true maximum+/// dropped, and not even a contiguous run.+#[tokio::test]+async fn dense_shadowing_source_key_does_not_displace_the_meta_sort_page() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ engine+ .create_index("dense_shadow", Schema::empty())+ .unwrap();+ let idx = engine.get_index("dense_shadow").unwrap();++ const N: u64 = 2000;+ for i in 0..N {+ idx.index_document(+ Some(format!("d{i:04}")),+ json!({ "n": i, "_seq_no": N - i }),+ )+ .await+ .unwrap();+ }++ let res = idx.search(&sorted_req("_seq_no", "desc", 5)).await.unwrap();+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();+ let newest: Vec<String> = (N - 5..N).rev().map(|i| format!("d{i:04}")).collect();+ assert_eq!(+ ids, newest,+ "a dense shadowing `_source._seq_no` must not displace the true top 5"+ );+ for hit in &res.hits {+ assert_eq!(+ hit.sort.first().and_then(Value::as_u64),+ idx.lookup_seq_no(&hit.id),+ "hit {} must rank on real metadata, not `_source._seq_no`",+ hit.id+ );+ }++ // The same corpus flushed to a segment. This passes with or without the+ // segment-side guard, because `build_doc_value_columns` skips every+ // `_`-prefixed source key so no `_seq_no` dv column is written — pinning+ // that here so relaxing the skip trips a test instead of a user.+ idx.flush().await.unwrap();+ let res = idx.search(&sorted_req("_seq_no", "desc", 5)).await.unwrap();+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();+ assert_eq!(+ ids, newest,+ "flushed: dense shadowing key must not displace the page"+ );+}++/// `_version` shadow. Worth its own case because it is the one where the+/// pre-#401 engine happened to return the RIGHT document (incidentally — its+/// sort values were `[0]` for the shadowed doc and `[null]` for the rest), so+/// an ungated prefilter turns #401 into an observable regression rather than+/// merely leaving a defect unfixed.+#[tokio::test]+async fn shadowing_source_key_does_not_regress_version_sort() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ engine.create_index("ver_shadow", Schema::empty()).unwrap();+ let idx = engine.get_index("ver_shadow").unwrap();++ const N: u64 = 2000;+ for i in 0..N {+ let body = if i == 7 {+ json!({ "n": i, "_version": 0 })+ } else {+ json!({ "n": i })+ };+ idx.index_document(Some(format!("d{i:04}")), body)+ .await+ .unwrap();+ }+ // Second write => d0007 has real `_version` 2, the unique maximum, while+ // its `_source._version` still claims 0, the unique minimum.+ idx.index_document(Some("d0007".to_string()), json!({ "n": 7, "_version": 0 }))+ .await+ .unwrap();++ let res = idx+ .search(&sorted_req("_version", "desc", 5))+ .await+ .unwrap();+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();+ assert_eq!(+ ids.first().map(String::as_str),+ Some("d0007"),+ "the unique maximum _version must lead the page, not be rejected by \+ its own shadowing `_source._version`"+ );+ assert_eq!(+ res.hits+ .first()+ .and_then(|h| h.sort.first())+ .and_then(Value::as_u64),+ Some(2),+ "d0007 must rank on its real _version (2), not `_source._version` (0)"+ );+}