AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): resolve ES meta-field sort keys instead of nulling them (#401)
xerj-orgbf8eca801e1a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c3711e8f8..aa6b599e9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **`sort` on an ES meta-field other than `_score` / `_doc` / `_id` resolved to+ `null` on every hit** ([#401](https://github.com/xerj-org/xerj/issues/401)).+ `compute_sort_values` special-cased exactly those three keys and sent+ everything else to `get_field_value(source, field)` — a lookup *inside*+ `_source`. `_seq_no`, `_version`, `_primary_term` and `_index` are engine+ metadata and are never literal `_source` keys, so the lookup missed on every+ document and the whole result set tied on `[null]`. The damage lands on+ keyset pagination: `search_after: [null]` is never strictly greater than the+ next hit's `[null]`, so `sort: [{"_seq_no":"asc"}]` + `search_after` — the+ consistent full-scan pattern used for migrations — either re-reads page one+ forever or stops after it, with no error either way. Measured on a 30-document+ index before the fix: 4 ids collected out of 30, page two empty.++ These fields now resolve through the same version-map accessors that populate+ the response meta-fields (`lookup_seq_no` / `lookup_version`), so a hit's+ `sort` value and its `_seq_no` / `_version` agree. `_primary_term` is the+ constant `1` the hit meta-field already emits and `_index` is the index name;+ both are constant within one index, so they tie like any other constant sort+ key and a client paginating on them still needs a tie-breaker. Sorting on+ `_seq_no` under `index.disable_sequence_numbers` is unchanged — es_compat+ rejects it at the API edge before the request reaches the engine.++ Real sort values are only half of it: the memtable's bounded sort-candidate+ path narrows the heap's input using a doc-values column keyed by the sort+ field, and no such column exists for a meta-field, so it classified every+ buffered document as "missing the field" and returned an arbitrary+ `materialisation_limit`-sized prefix. A correctly ranked page over a wrong+ candidate set is still a silently wrong page, so that path — and the+ segment-side sort shadow — now decline meta-fields and take the full walk.+ The same gate fixes an independent instance of the identical defect on `_id`,+ which already resolved to a real sort value: over a 2 000-document memtable+ `sort: [{"_id":"desc"}]` returned+ `["d1995","d1989","d1981","d1973","d1961"]` instead of+ `["d1999","d1998","d1997","d1996","d1995"]`.+- **`xerj autoindex` aborted a whole run when one file declared two or more SQLtables** ([#360](https://github.com/xerj-org/xerj/issues/360)). One file canfeed several datasets — a SQL dump is one file and N tables — but the sealeddiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex c1403c1ce..ef4728ead 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -12683,6 +12683,61 @@ impl Index {)}+ /// Resolve an ES meta-field named as a SORT key (#401).+ ///+ /// These fields are engine metadata: they are never literal keys in+ /// `_source`, so `compute_sort_values`' generic `get_field_value` path+ /// resolved every one of them to `Null` on every document — a sort that+ /// silently ties the whole result set, and a `search_after` cursor that+ /// can never advance past page one (`[null]` is never strictly greater+ /// than `[null]`). Lucene has no such failure mode because a sort key+ /// is read from the field's DOC VALUES column rather than from the+ /// stored document — `SortedNumericSortField` selects the document's+ /// representative sort value out of `SortedNumericDocValues`+ /// (`lucene/core/src/java/org/apache/lucene/search/SortedNumericSortField.java:49-59`)+ /// and `NumericComparator` compares those column values directly+ /// (`lucene/core/src/java/org/apache/lucene/search/comparators/NumericComparator.java:47-63`).+ /// xerj has no column for these fields, so it resolves them through the+ /// SAME accessors that populate the response meta-fields — a hit's+ /// `sort` value and its `_seq_no` / `_version` therefore agree.+ ///+ /// Returns `None` for every other field name, so the caller falls+ /// through to the ordinary `_source` lookup.+ fn meta_sort_value(&self, field: &str, id: &str) -> Option<Value> {+ // Hot path: this runs per candidate doc per sort field inside the+ // bounded collectors, and an ordinary field name is the overwhelming+ // case — settle it with one byte compare instead of four `str` eq.+ if !field.starts_with('_') {+ return None;+ }+ match field {+ // `_seq_no`: the ES-sortable one (real ES gives it doc values+ // and rejects the sort only under+ // `index.disable_sequence_numbers`, which es_compat.rs already+ // rejects at the API edge before the request reaches here).+ // Unresolvable (unknown / tombstoned) stays `Null` so the+ // request's `missing` policy still applies.+ "_seq_no" => Some(+ self.lookup_seq_no(id)+ .map_or(Value::Null, |s| Value::Number(s.into())),+ ),+ "_version" => Some(+ self.lookup_version(id)+ .map_or(Value::Null, |v| Value::Number(v.into())),+ ),+ // xerj is single-shard with `primary_term` fixed at 1 — the same+ // constant the hit meta-field emits (es_compat.rs:13250). Constant+ // for every doc, so it ties like any other constant sort key;+ // callers paginating on it still need a tie-breaker field.+ "_primary_term" => Some(Value::Number(1.into())),+ // Constant within one index; it discriminates only in the+ // cross-index merge, which re-sorts on these same `sort` arrays+ // (es_compat.rs:10855).+ "_index" => Some(Value::String(self.name.as_str().to_string())),+ _ => None,+ }+ }+/// Retrieve a document by its string ID.////// Checks the memtable first, then searches on-disk segments.@@ -14431,7 +14486,7 @@ impl Index {}seen_ids.insert(hit.id.clone());let seq = self.lookup_seq_no(&hit.id).unwrap_or(u64::MAX);- topk.offer(hit, seq);+ topk.offer(hit, seq, self);}} else if all_hits.len() < materialisation_limit && !seen_ids.contains(&hit.id){@@ -14480,6 +14535,7 @@ impl Index {passage: None,},seq,+ self,);}}@@ -14686,6 +14742,16 @@ impl Index {let sf = &request.sort[0];if sf.is_score()|| sf.is_doc_order()+ || sf.field == "_id"+ // An ES meta-field has no memtable dv column, so+ // `sort_candidates_numeric` would classify EVERY+ // buffered doc as "missing this field" and hand+ // back an arbitrary `cap`-sized prefix as the+ // candidate set. With #401's real `_seq_no`+ // values that prefix is no longer a superset of+ // the true top-cap — it would rank the wrong+ // documents correctly. Take the full walk.+ || is_meta_sort_field(&sf.field)|| sf.mode != SortMode::default()|| sf.missing != SortMissing::default(){@@ -16302,7 +16368,7 @@ impl Index {if let Some(topk) = sort_topk.as_mut() {let seq =self.lookup_seq_no(&hit.id).unwrap_or(u64::MAX);- topk.offer(hit, seq);+ topk.offer(hit, seq, self);} else {all_hits.push(hit);// #179 — hold the collector at@@ -17096,7 +17162,7 @@ impl Index {// ordering agree on the exact total order (incl. date-string// normalisation).for hit in &mut final_hits {- hit.sort = compute_sort_values(&hit.source, hit.score, &hit.id, sort_fields);+ hit.sort = compute_sort_values(&hit.source, hit.score, &hit.id, sort_fields, self);}// Sort by the populated sort keys. Key ties break by seq_no// ASC (insertion order == ES internal doc-id order post-B1),@@ -22996,6 +23062,16 @@ impl Index {field: &str,seg_doc_count: u64,) -> Option<Resident<Option<Arc<Vec<(i64, u32)>>>>> {+ // An ES meta-field (`_seq_no`, `_version`, …) is engine metadata, not+ // a `_source` key, so no segment ever writes a dv column under that+ // name. Bail BEFORE the registry insert so the publish-time warm+ // doesn't burn one of its 16 slots probing a column that cannot+ // exist, and so a future column that happened to share the name+ // could never silently drive the candidate cut for a key the+ // engine resolves from the version map instead (#401).+ if is_meta_sort_field(field) {+ return None;+ }// Register the field so the publish-time warm pre-builds this// shadow for every FUTURE segment (bounded registry).if self.sort_shadow_fields.len() < 16 && !self.sort_shadow_fields.contains_key(field) {@@ -23496,7 +23572,7 @@ impl Index {} else {scorer(source_ref, id_ref)};- let key = compute_sort_values(source_ref, score, id_ref, topk.fields.as_slice());+ let key = compute_sort_values(source_ref, score, id_ref, topk.fields.as_slice(), self);if !topk.would_admit(&key) {continue;}@@ -24192,7 +24268,8 @@ impl Index {} else {scorer(source_ref, id_ref)};- let key = compute_sort_values(source_ref, score, id_ref, topk.fields.as_slice());+ let key =+ compute_sort_values(source_ref, score, id_ref, topk.fields.as_slice(), self);if !topk.would_admit(&key) {continue;}@@ -35908,9 +35985,10 @@ impl SortTopK {}/// Offer a fully-sourced hit. Computes its sort key from `_source`,/// then delegates to `offer_keyed`. `seq` is the hit's insertion-order- /// tie-break (see `SortHeapEntry::seq`).- fn offer(&mut self, mut hit: Hit, seq: u64) {- hit.sort = compute_sort_values(&hit.source, hit.score, &hit.id, &self.fields);+ /// tie-break (see `SortHeapEntry::seq`). `idx` resolves the ES+ /// meta-fields that live outside `_source` (see `meta_sort_value`).+ fn offer(&mut self, mut hit: Hit, seq: u64, idx: &Index) {+ hit.sort = compute_sort_values(&hit.source, hit.score, &hit.id, &self.fields, idx);self.offer_keyed(hit, seq);}/// Offer a hit whose `sort` key is already computed. Applies the@@ -36969,11 +37047,27 @@ fn sort_epoch_memo(s: &str) -> Option<f64> {v}+/// ES meta-fields whose sort value comes from engine metadata rather than+/// from `_source` or from the hit itself (#401). `_score` / `_doc` / `_id`+/// are deliberately NOT here: `compute_sort_values` resolves those from the+/// hit it is already holding, with no index lookup.+///+/// Also the gate for the columnar sort-candidate prefilters: those narrow the+/// candidate set with a doc-values column keyed by the sort field name, and+/// no such column exists for a meta-field — a memtable/segment column probe+/// would report "every doc is missing this field" and cut the candidate set+/// to an arbitrary `cap` docs, which is the same silently-wrong page the+/// null sort values produced.+fn is_meta_sort_field(field: &str) -> bool {+ matches!(field, "_seq_no" | "_version" | "_primary_term" | "_index")+}+fn compute_sort_values(source: &Value,score: f32,id: &str,sort_fields: &[xerj_query::sort::SortField],+ idx: &Index,) -> Vec<Value> {let mut sort_vals: Vec<Value> = Vec::with_capacity(sort_fields.len());for sf in sort_fields {@@ -36991,6 +37085,12 @@ fn compute_sort_values(// ordering and `search_after: [last_id]` keyset paging (used by// reindex) compare correctly. Matches ES, which echoes the id.Value::String(id.to_string())+ } else if let Some(meta) = idx.meta_sort_value(&sf.field, id) {+ // The rest of the ES meta-fields (`_seq_no`, `_version`,+ // `_primary_term`, `_index`) — engine metadata, resolved from+ // the version map instead of from `_source` (#401). Everything+ // else falls through to the `_source` lookup below.+ meta} else {// `.keyword` is ES's conventional multi-field sub-type on// dynamic/keyword fields: sorting on `foo.keyword` reads thediff --git a/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs b/engine/crates/xerj-engine/tests/sort_on_meta_fields.rsnew file mode 100644index 000000000..8cf4a6212--- /dev/null+++ b/engine/crates/xerj-engine/tests/sort_on_meta_fields.rs@@ -0,0 +1,305 @@+//! Regression test for #401 — `sort` on an ES meta-field other than+//! `_score` / `_doc` / `_id` resolved to `null` on EVERY hit.+//!+//! `compute_sort_values` special-cased exactly `_score`, `_doc` and `_id`;+//! everything else went to `get_field_value(source, field)`, a lookup INSIDE+//! `_source`. `_seq_no` / `_version` / `_primary_term` / `_index` are engine+//! metadata and are never literal `_source` keys, so the lookup missed on+//! every document and the whole result set tied on `[null]`. The visible+//! damage is keyset pagination: `search_after: [null]` is never strictly+//! greater than the next hit's `[null]`, so a client paging on+//! `sort: [{"_seq_no":"asc"}]` — the consistent-full-scan pattern ES+//! documents, and the one this repo's own `wal_tap` shape uses — either+//! re-reads page one forever or stops after it, with no error either way.+//!+//! Lucene cannot express this bug: a sort key is read from the field's DOC+//! VALUES column, not from the stored document. `SortedNumericSortField`+//! selects the document's representative value out of `SortedNumericDocValues`+//! (`lucene/core/src/java/org/apache/lucene/search/SortedNumericSortField.java:49-59`)+//! and `NumericComparator` compares those column values directly+//! (`lucene/core/src/java/org/apache/lucene/search/comparators/NumericComparator.java:47-63`).+//! xerj has no column for a meta-field, so `Index::meta_sort_value` resolves+//! it through the same version-map accessors that populate the response+//! meta-fields (`lookup_seq_no` / `lookup_version`).+//!+//! Two collectors had to agree for that to be enough, and the second one is+//! the trap: the memtable's bounded sort-candidate path+//! (`sort_candidates_numeric`) cuts the candidate set with a dv column keyed+//! by the sort field. There is no `_seq_no` column, so it classified every+//! buffered doc as "missing the field" and handed back an arbitrary+//! `materialisation_limit`-sized prefix. Real sort values over a wrong+//! candidate set is still a silently wrong page — the same defect wearing+//! the other hat — so `is_meta_sort_field` also gates that path. The same+//! gate is what fixes `_id` at the cap: `_id` already resolved to a real+//! sort value, and `sort: [{"_id":"desc"}]` over a >cap memtable STILL+//! returned the wrong five documents on main.++use serde_json::{json, Value};+use tempfile::TempDir;+use xerj_common::config::Config;+use xerj_common::types::Schema;+use xerj_engine::Engine;+use xerj_query::parse_request;++fn make_engine(dir: &TempDir) -> Engine {+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ Engine::new(config).expect("engine::new")+}++fn sorted_req(field: &str, order: &str, size: usize) -> xerj_query::ast::SearchRequest {+ parse_request(&json!({+ "query": {"match_all": {}},+ "size": size,+ "sort": [{field: order}],+ }))+ .expect("parse_request")+}++/// Every hit's `sort` value for a meta-field must be the document's REAL+/// metadata, not `null` — and `_seq_no` must agree with `lookup_seq_no`,+/// the accessor the `_seq_no` response field is built from.+#[tokio::test]+async fn meta_field_sort_values_are_real_not_null() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ engine.create_index("meta_sort", Schema::empty()).unwrap();+ let idx = engine.get_index("meta_sort").unwrap();++ for i in 0..8u64 {+ idx.index_document(Some(format!("d{i}")), json!({ "n": i }))+ .await+ .unwrap();+ }+ // A second write on one doc so `_version` is not constant across the+ // corpus: a fix that resolved `_version` to a hard-coded 1 would pass a+ // "not null" assertion but still not be reading per-doc metadata.+ idx.index_document(Some("d3".to_string()), json!({ "n": 3, "touched": true }))+ .await+ .unwrap();++ let res = idx.search(&sorted_req("_seq_no", "asc", 50)).await.unwrap();+ assert_eq!(res.hits.len(), 8, "corpus size");+ for hit in &res.hits {+ let expected = idx+ .lookup_seq_no(&hit.id)+ .unwrap_or_else(|| panic!("{} has no seq_no", hit.id));+ assert_eq!(+ hit.sort.first(),+ Some(&Value::from(expected)),+ "hit {} sort value must be its real _seq_no, got {:?}",+ hit.id,+ hit.sort+ );+ }+ // …and the page must actually be ordered by it.+ let seqs: Vec<u64> = res+ .hits+ .iter()+ .map(|h| h.sort[0].as_u64().expect("numeric _seq_no sort value"))+ .collect();+ let mut ascending = seqs.clone();+ ascending.sort_unstable();+ assert_eq!(seqs, ascending, "_seq_no asc page must be sorted by seq_no");+ assert_eq!(+ seqs.iter().collect::<std::collections::HashSet<_>>().len(),+ seqs.len(),+ "_seq_no is unique per live doc, so no two hits may tie"+ );++ // `_version`: real per-doc write counter (d3 was written twice).+ let res = idx+ .search(&sorted_req("_version", "desc", 50))+ .await+ .unwrap();+ assert_eq!(res.hits.len(), 8, "corpus size under _version sort");+ for hit in &res.hits {+ let expected = idx.lookup_version(&hit.id).expect("version");+ assert_eq!(+ hit.sort.first(),+ Some(&Value::from(expected)),+ "hit {} sort value must be its real _version",+ hit.id+ );+ }+ assert_eq!(+ res.hits[0].sort.first().and_then(Value::as_u64),+ Some(2),+ "the twice-written doc's _version sort value must be 2, not a constant"+ );+ assert_eq!(+ res.hits[0].id,+ "d3",+ "_version desc must put the twice-written doc first, got {:?}",+ res.hits.iter().map(|h| &h.id).collect::<Vec<_>>()+ );++ // `_primary_term` / `_index` are constant in a single-shard, single-index+ // engine, but they must be the CONSTANT the response meta-fields emit,+ // never `null` — a null key silently ties the whole result set.+ for (field, expected) in [+ ("_primary_term", Value::from(1u64)),+ ("_index", Value::from("meta_sort")),+ ] {+ let res = idx.search(&sorted_req(field, "asc", 50)).await.unwrap();+ // Non-vacuity: an empty page would satisfy the loop below.+ assert_eq!(res.hits.len(), 8, "sort on {field} must return the corpus");+ for hit in &res.hits {+ assert_eq!(+ hit.sort.first(),+ Some(&expected),+ "sort on {field} must resolve, got {:?} for {}",+ hit.sort,+ hit.id+ );+ }+ }+}++/// The reported impact: keyset pagination on `_seq_no`. With `[null]` sort+/// values the cursor never advances, so this loop either re-reads page one+/// forever (caught by the iteration budget) or returns fewer than the whole+/// corpus.+#[tokio::test]+async fn search_after_on_seq_no_pages_the_whole_index() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ engine.create_index("sa_seq", Schema::empty()).unwrap();+ let idx = engine.get_index("sa_seq").unwrap();++ const N: usize = 25;+ for i in 0..N {+ idx.index_document(Some(format!("d{i:03}")), json!({ "n": i }))+ .await+ .unwrap();+ }+ // Flush these to a segment, then buffer five more, so the paged walk+ // spans both the segment and the memtable source.+ idx.flush().await.unwrap();+ for i in N..(N + 5) {+ idx.index_document(Some(format!("d{i:03}")), json!({ "n": i }))+ .await+ .unwrap();+ }+ let total = N + 5;++ let mut collected: Vec<String> = Vec::new();+ let mut after: Option<Vec<Value>> = None;+ // Budget: enough pages to drain the corpus, few enough that a stuck+ // cursor trips it instead of hanging the suite.+ for _ in 0..(total + 5) {+ let mut body = json!({+ "query": {"match_all": {}},+ "size": 4,+ "sort": [{"_seq_no": "asc"}],+ });+ if let Some(ref a) = after {+ body["search_after"] = Value::Array(a.clone());+ }+ let res = idx.search(&parse_request(&body).unwrap()).await.unwrap();+ if res.hits.is_empty() {+ break;+ }+ after = res.hits.last().map(|h| h.sort.clone());+ collected.extend(res.hits.iter().map(|h| h.id.clone()));+ assert!(+ collected.len() <= total,+ "cursor is not advancing: {} ids collected from a {}-doc index \+ (first repeat page starts at {:?})",+ collected.len(),+ total,+ &collected[total.min(collected.len().saturating_sub(1))]+ );+ }++ let mut unique = collected.clone();+ unique.sort();+ unique.dedup();+ assert_eq!(+ unique.len(),+ total,+ "_seq_no keyset pagination must visit every doc exactly once; \+ collected {} ids, {} unique",+ collected.len(),+ unique.len()+ );+ assert_eq!(collected.len(), total, "no doc may be returned twice");+}++/// The candidate-cut half. `materialisation_limit` is+/// `max(from + size + 100, 256)`, so a memtable with more docs than that+/// exercises the bounded candidate path — which has no dv column to cut on+/// for a meta-field (nor for `_id`) and must therefore not be used.+#[tokio::test]+async fn meta_field_sort_is_exact_past_the_memtable_candidate_cap() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ engine.create_index("cap_sort", Schema::empty()).unwrap();+ let idx = engine.get_index("cap_sort").unwrap();++ // > materialisation_limit (256) buffered docs, none flushed.+ const N: u64 = 2000;+ for i in 0..N {+ idx.index_document(Some(format!("d{i:04}")), json!({ "n": i }))+ .await+ .unwrap();+ }++ // Ids are lexicographically ordered with insertion order, so the+ // expected page is the same for `_seq_no` and `_id`.+ let newest: Vec<String> = (N - 5..N).rev().map(|i| format!("d{i:04}")).collect();+ let oldest: Vec<String> = (0..5).map(|i| format!("d{i:04}")).collect();++ for field in ["_seq_no", "_id"] {+ let res = idx.search(&sorted_req(field, "desc", 5)).await.unwrap();+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();+ assert_eq!(+ ids, newest,+ "sort {field} desc over a {N}-doc memtable must return the true \+ top 5, not an arbitrary prefix of the candidate cut"+ );++ let res = idx.search(&sorted_req(field, "asc", 5)).await.unwrap();+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();+ assert_eq!(ids, oldest, "sort {field} asc over a {N}-doc memtable");+ }++ // Control: a real `_source` field DOES have a memtable dv column, so the+ // bounded candidate path stays enabled for it and must stay correct.+ let res = idx.search(&sorted_req("n", "desc", 5)).await.unwrap();+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();+ assert_eq!(ids, newest, "control: plain numeric field sort");+}++/// Same guarantee once the docs live in a segment rather than the memtable —+/// the segment side resolves `_seq_no` through the version map too.+#[tokio::test]+async fn seq_no_sort_is_exact_after_flush() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ engine+ .create_index("flushed_sort", Schema::empty())+ .unwrap();+ let idx = engine.get_index("flushed_sort").unwrap();++ const N: u64 = 600;+ for i in 0..N {+ idx.index_document(Some(format!("d{i:04}")), json!({ "n": i }))+ .await+ .unwrap();+ }+ idx.flush().await.unwrap();++ let res = idx.search(&sorted_req("_seq_no", "desc", 5)).await.unwrap();+ let ids: Vec<String> = res.hits.iter().map(|h| h.id.clone()).collect();+ let newest: Vec<String> = (N - 5..N).rev().map(|i| format!("d{i:04}")).collect();+ assert_eq!(ids, newest, "flushed _seq_no desc page");+ for hit in &res.hits {+ assert_eq!(+ hit.sort.first().and_then(Value::as_u64),+ idx.lookup_seq_no(&hit.id),+ "flushed hit {} must carry its real _seq_no",+ hit.id+ );+ }+}