Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): gate the third sort prefilter on meta-fields too (#401)

xerj-org78a1294f0e32
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aa6b599e9..1ad8c4c40 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -33,8 +33,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Real sort values are only half of it: the memtable's bounded sort-candidate
path narrows the heap's input using a doc-values column keyed by the sort
- field, and no such column exists for a meta-field, so it classified every
- buffered document as "missing the field" and returned an arbitrary
+ field, which for a meta-field is derived from `_source` rather than from the
+ version map the heap now ranks on. With no such key present it classified
+ every buffered document as "missing the field" and returned an arbitrary
`materialisation_limit`-sized prefix. A correctly ranked page over a wrong
candidate set is still a silently wrong page, so that path — and the
segment-side sort shadow — now decline meta-fields and take the full walk.
@@ -44,6 +45,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`["d1995","d1989","d1981","d1973","d1961"]` instead of
`["d1999","d1998","d1997","d1996","d1995"]`.
+ There is a third such prefilter, and it is the one the two gates above route
+ meta-sorts *onto*: the memtable scan's pre-clone rejection
+ (`memtable_primary_key_rejects`) skips a document whose primary sort key,
+ read from `_source`, already loses to the full heap. Once the heap ranks on
+ version-map metadata, a document carrying a `_source` key of the same name
+ is compared against the wrong frontier and dropped before the heap ever sees
+ it. Measured over ES-compat HTTP on 2 000 documents where only `d0000`
+ carried `"_seq_no": 10000000`: `{"size":5,"sort":[{"_seq_no":"asc"}]}`
+ returned `[d0001, d0002, d0003, d0004, d0005]`, omitting the true first
+ document, while `"size": 3000` returned it — a silent, size-dependent
+ missing document. That path now declines meta-fields too.
+
+ **Behaviour change:** `sort` on `_seq_no` / `_version` / `_primary_term` /
+ `_index` now ignores a `_source` key of the same name and always ranks on
+ engine metadata. Previously it ranked on the `_source` value where one
+ existed and on `null` everywhere else. This is observable because xerj —
+ unlike Elasticsearch, which rejects a metadata-named key inside a document
+ with a `document_parsing_exception` — accepts `{"_seq_no": 1}` in a document
+ body and echoes it back in `_source`. Such a key is now inert for sorting;
+ it is still stored, still returned in `_source`, and still usable for
+ ordinary queries. Rejecting metadata-named `_source` keys at the API edge is
+ deliberately **not** part of this change — xerj writes `_routing` into
+ `_source` itself, so the edge rule needs its own design.
+
- **`xerj autoindex` aborted a whole run when one file declared two or more SQL
tables** ([#360](https://github.com/xerj-org/xerj/issues/360)). One file can
feed several datasets — a SQL dump is one file and N tables — but the sealed
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index ef4728ead..de5e7b310 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -14743,14 +14743,19 @@ impl Index {
if sf.is_score()
|| sf.is_doc_order()
|| sf.field == "_id"
- // An ES meta-field has no memtable dv column, so
- // `sort_candidates_numeric` would classify EVERY
- // buffered doc as "missing this field" and hand
- // back an arbitrary `cap`-sized prefix as the
- // candidate set. With #401's real `_seq_no`
- // values that prefix is no longer a superset of
- // the true top-cap — it would rank the wrong
- // documents correctly. Take the full walk.
+ // `sort_candidates_numeric` cuts the candidate set
+ // by reading the sort field out of each buffered
+ // doc's `_source`, but #401 resolves a meta-field
+ // from the version map — so the cut and the heap
+ // would rank on different data. Usually `_source`
+ // has no such key at all and EVERY doc classifies
+ // as "missing the field", yielding an arbitrary
+ // `cap`-sized prefix; where a doc DOES carry a
+ // literal `_seq_no` (xerj accepts one, see
+ // `memtable_primary_key_rejects`) the cut ranks on
+ // the shadowing value instead. Both give a page
+ // of the wrong documents, correctly ranked. Take
+ // the full walk.
|| is_meta_sort_field(&sf.field)
|| sf.mode != SortMode::default()
|| sf.missing != SortMissing::default()
@@ -23062,13 +23067,25 @@ impl Index {
field: &str,
seg_doc_count: u64,
) -> Option<Resident<Option<Arc<Vec<(i64, u32)>>>>> {
- // An ES meta-field (`_seq_no`, `_version`, …) is engine metadata, not
- // a `_source` key, so no segment ever writes a dv column under that
- // name. Bail BEFORE the registry insert so the publish-time warm
- // doesn't burn one of its 16 slots probing a column that cannot
- // exist, and so a future column that happened to share the name
- // could never silently drive the candidate cut for a key the
- // engine resolves from the version map instead (#401).
+ // An ES meta-field (`_seq_no`, `_version`, …) is resolved from the
+ // version map, never from the segment, so a dv column keyed by that
+ // name must not drive the candidate cut (#401).
+ //
+ // Today no such column exists — but NOT because the name is
+ // impossible: xerj accepts a literal `_seq_no` inside a document
+ // (real ES rejects it), so it can reach `_source`. It is the dv
+ // sidecar builder that drops it, skipping every `_`-prefixed source
+ // key as envelope bookkeeping (the `starts_with('_')` skip in
+ // `build_doc_value_columns`). That invariant lives elsewhere and
+ // is not this path's to assume, so the guard is kept as the local
+ // enforcement: relax the skip and the cut would silently rank on a
+ // user-supplied value while the heap ranks on real metadata — the
+ // memtable's identical shape, which IS reachable today, is what
+ // `memtable_primary_key_rejects` had to be gated for.
+ //
+ // Bailing BEFORE the registry insert also stops the publish-time
+ // warm burning one of its 16 slots probing a column that is not
+ // written.
if is_meta_sort_field(field) {
return None;
}
@@ -37007,6 +37024,26 @@ fn memtable_primary_key_rejects(topk: &SortTopK, source: &Value) -> bool {
if sf.is_score() || sf.is_doc_order() {
return false;
}
+ // An ES meta-field is resolved by `compute_sort_values` from the version
+ // map, NOT from `_source` — so deriving it here from `_source` would
+ // compare the prefilter's frontier against a heap ranked on different
+ // data. xerj (unlike ES, which rejects the key outright) accepts a
+ // literal `_seq_no` / `_version` / `_primary_term` / `_index` inside a
+ // document, so that disagreement is wire-reachable: the shadowing value
+ // rejects a doc whose REAL metadata belongs on the page, and it is
+ // dropped before `offer` ever sees it — silently, and only at small
+ // `size`. Meta-sorts take the unaccelerated full walk (#401).
+ //
+ // `_id` is deliberately absent from `is_meta_sort_field` and needs no
+ // gate here only because `compute_sort_values` emits it as a STRING:
+ // `primary_f64_rejects` reads the heap frontier through `as_f64()`, gets
+ // `None` for a string, and can never reject. Verified over the wire with
+ // a dense numeric `_source._id` — the `_id desc` page is identical at
+ // size 5 and size 3000. Give `_id` a numeric sort value and it becomes
+ // the same defect; add it here if that ever changes.
+ if is_meta_sort_field(&sf.field) {
+ return false;
+ }
// Root-level literal key only — mirrors `get_field_value`'s fast path.
// A dotted/nested primary field misses here and is admitted (correct,
// just not accelerated).
diff --git a/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs b/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs
index 8cf4a6212..ff5e299bb 100644
--- a/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs
+++ b/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs
@@ -18,21 +18,34 @@
//! (`lucene/core/src/java/org/apache/lucene/search/SortedNumericSortField.java:49-59`)
//! and `NumericComparator` compares those column values directly
//! (`lucene/core/src/java/org/apache/lucene/search/comparators/NumericComparator.java:47-63`).
-//! xerj has no column for a meta-field, so `Index::meta_sort_value` resolves
-//! it through the same version-map accessors that populate the response
-//! meta-fields (`lookup_seq_no` / `lookup_version`).
+//! xerj indexes no column for a meta-field, so `Index::meta_sort_value`
+//! resolves it through the same version-map accessors that populate the
+//! response meta-fields (`lookup_seq_no` / `lookup_version`).
//!
-//! Two collectors had to agree for that to be enough, and the second one is
-//! the trap: the memtable's bounded sort-candidate path
-//! (`sort_candidates_numeric`) cuts the candidate set with a dv column keyed
-//! by the sort field. There is no `_seq_no` column, so it classified every
-//! buffered doc as "missing the field" and handed back an arbitrary
-//! `materialisation_limit`-sized prefix. Real sort values over a wrong
-//! candidate set is still a silently wrong page — the same defect wearing
-//! the other hat — so `is_meta_sort_field` also gates that path. The same
-//! gate is what fixes `_id` at the cap: `_id` already resolved to a real
-//! sort value, and `sort: [{"_id":"desc"}]` over a >cap memtable STILL
+//! Real sort values are NOT enough on their own, and that is what the rest of
+//! this file is about. Three prefilters narrow the heap's input by deriving
+//! the primary sort key from `_source`, and once the heap ranks a meta-field
+//! on version-map metadata, any of them ranking on `_source` is comparing two
+//! different frontiers. A correctly ranked page over a wrong candidate set is
+//! still a silently wrong page — the same defect wearing the other hat — so
+//! all three now decline meta-fields via `is_meta_sort_field`:
+//!
+//! 1. `sort_candidates_numeric` (memtable bounded candidate cut),
+//! 2. `sorted_shadow_for` (segment sort shadow),
+//! 3. `memtable_primary_key_rejects` (memtable pre-clone rejection) — the
+//! one gates 1 and 2 route meta-sorts ONTO, missed in the first pass.
+//!
+//! Gate 1 is also what fixes `_id` at the cap: `_id` already resolved to a
+//! real sort value, and `sort: [{"_id":"desc"}]` over a >cap memtable STILL
//! returned the wrong five documents on main.
+//!
+//! Note the premise these gates do NOT rest on: "a meta-field name can never
+//! be a `_source` key". It can — xerj accepts `{"_seq_no": 1}` in a document
+//! body where ES rejects it — and the shadowing tests at the end of this file
+//! exercise exactly that. Segments happen to be safe today only because
+//! `build_doc_value_columns` skips every `_`-prefixed source key; the memtable
+//! has no such skip, which is why gates 1 and 3 are load-bearing rather than
+//! defensive.
use serde_json::{json, Value};
use tempfile::TempDir;
@@ -303,3 +316,180 @@ async fn seq_no_sort_is_exact_after_flush() {
);
}
}
+
+/// A `_source` key that SHADOWS the meta-field being sorted on.
+///
+/// xerj — unlike ES, which rejects `_seq_no` inside a document with a
+/// `document_parsing_exception` — accepts one and echoes it back in
+/// `_source`, so this is wire-reachable, not a synthetic shape.
+///
+/// The trap is that #401 moved meta-field RESOLUTION to the version map
+/// while three prefilters still derived the sort key from `_source`. Two
+/// were gated with `is_meta_sort_field`; `memtable_primary_key_rejects` —
+/// the pre-clone rejection on the very full-walk path the other two gates
+/// FORCE meta-sorts onto — was not. A shadowing key then made the
+/// prefilter's frontier and the heap's frontier disagree, and the doc was
+/// dropped before `offer` ever saw it: silently, and only when `size` is
+/// small enough for the heap to fill.
+///
+/// Before the gate, with only d0000 carrying `"_seq_no": 10000000`,
+/// `sort _seq_no asc size 5` returned [d0001..d0005] — the true first
+/// document missing — while `size 3000` returned it, which is what makes
+/// the loss silent.
+#[tokio::test]
+async fn source_key_shadowing_a_meta_sort_field_drops_no_document() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ engine.create_index("shadow_sort", Schema::empty()).unwrap();
+ let idx = engine.get_index("shadow_sort").unwrap();
+
+ // > materialisation_limit (256), memtable-resident: the only path where
+ // a source-derived prefilter still runs for a meta sort.
+ const N: u64 = 2000;
+ for i in 0..N {
+ // SPARSE shadow: one doc claims a `_seq_no` far past the corpus, so
+ // the prefilter ranks it last while its real `_seq_no` (0) ranks it
+ // first. Every other doc is underivable and admitted, which is why
+ // the page looks plausible instead of empty.
+ let body = if i == 0 {
+ json!({ "n": i, "_seq_no": 10_000_000 })
+ } else {
+ json!({ "n": i })
+ };
+ idx.index_document(Some(format!("d{i:04}")), body)
+ .await
+ .unwrap();
+ }
+
+ let res = idx.search(&sorted_req("_seq_no", "asc", 5)).await.unwrap();
+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();
+ let oldest: Vec<String> = (0..5).map(|i| format!("d{i:04}")).collect();
+ assert_eq!(
+ ids, oldest,
+ "a shadowing `_source._seq_no` must not drop the true first document"
+ );
+ // The sort value must be the METADATA, not the shadowing source value.
+ assert_eq!(
+ res.hits
+ .first()
+ .and_then(|h| h.sort.first())
+ .and_then(Value::as_u64),
+ idx.lookup_seq_no("d0000"),
+ "d0000 must rank and report on its real _seq_no, not `_source._seq_no`"
+ );
+
+ // Size-independence is the actual contract: the page must not change
+ // when the heap is large enough that no doc is ever rejected early.
+ let big = idx
+ .search(&sorted_req("_seq_no", "asc", 3000))
+ .await
+ .unwrap();
+ let big_ids: Vec<String> = big.hits.iter().take(5).map(|h| h.id.clone()).collect();
+ assert_eq!(
+ ids, big_ids,
+ "the size-5 page must equal the first 5 of the size-3000 page"
+ );
+}
+
+/// DENSE shadow: EVERY doc carries a `_source._seq_no` counting down, so the
+/// prefilter's frontier is the exact inverse of the heap's — no doc is
+/// "underivable" and the rejection fires on every comparison. Before the
+/// gate this returned [d1995, d1989, d1981, d1973, d1961]: the true maximum
+/// dropped, and not even a contiguous run.
+#[tokio::test]
+async fn dense_shadowing_source_key_does_not_displace_the_meta_sort_page() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ engine
+ .create_index("dense_shadow", Schema::empty())
+ .unwrap();
+ let idx = engine.get_index("dense_shadow").unwrap();
+
+ const N: u64 = 2000;
+ for i in 0..N {
+ idx.index_document(
+ Some(format!("d{i:04}")),
+ json!({ "n": i, "_seq_no": N - i }),
+ )
+ .await
+ .unwrap();
+ }
+
+ let res = idx.search(&sorted_req("_seq_no", "desc", 5)).await.unwrap();
+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();
+ let newest: Vec<String> = (N - 5..N).rev().map(|i| format!("d{i:04}")).collect();
+ assert_eq!(
+ ids, newest,
+ "a dense shadowing `_source._seq_no` must not displace the true top 5"
+ );
+ for hit in &res.hits {
+ assert_eq!(
+ hit.sort.first().and_then(Value::as_u64),
+ idx.lookup_seq_no(&hit.id),
+ "hit {} must rank on real metadata, not `_source._seq_no`",
+ hit.id
+ );
+ }
+
+ // The same corpus flushed to a segment. This passes with or without the
+ // segment-side guard, because `build_doc_value_columns` skips every
+ // `_`-prefixed source key so no `_seq_no` dv column is written — pinning
+ // that here so relaxing the skip trips a test instead of a user.
+ idx.flush().await.unwrap();
+ let res = idx.search(&sorted_req("_seq_no", "desc", 5)).await.unwrap();
+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();
+ assert_eq!(
+ ids, newest,
+ "flushed: dense shadowing key must not displace the page"
+ );
+}
+
+/// `_version` shadow. Worth its own case because it is the one where the
+/// pre-#401 engine happened to return the RIGHT document (incidentally — its
+/// sort values were `[0]` for the shadowed doc and `[null]` for the rest), so
+/// an ungated prefilter turns #401 into an observable regression rather than
+/// merely leaving a defect unfixed.
+#[tokio::test]
+async fn shadowing_source_key_does_not_regress_version_sort() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ engine.create_index("ver_shadow", Schema::empty()).unwrap();
+ let idx = engine.get_index("ver_shadow").unwrap();
+
+ const N: u64 = 2000;
+ for i in 0..N {
+ let body = if i == 7 {
+ json!({ "n": i, "_version": 0 })
+ } else {
+ json!({ "n": i })
+ };
+ idx.index_document(Some(format!("d{i:04}")), body)
+ .await
+ .unwrap();
+ }
+ // Second write => d0007 has real `_version` 2, the unique maximum, while
+ // its `_source._version` still claims 0, the unique minimum.
+ idx.index_document(Some("d0007".to_string()), json!({ "n": 7, "_version": 0 }))
+ .await
+ .unwrap();
+
+ let res = idx
+ .search(&sorted_req("_version", "desc", 5))
+ .await
+ .unwrap();
+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();
+ assert_eq!(
+ ids.first().map(String::as_str),
+ Some("d0007"),
+ "the unique maximum _version must lead the page, not be rejected by \
+ its own shadowing `_source._version`"
+ );
+ assert_eq!(
+ res.hits
+ .first()
+ .and_then(|h| h.sort.first())
+ .and_then(Value::as_u64),
+ Some(2),
+ "d0007 must rank on its real _version (2), not `_source._version` (0)"
+ );
+}