Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(order): ES-exact doc order — seq_no-sorted merges + settled leaf semantics (flips 6 DIFF cells)

xerj-orgaaa4b59d152d
diff --git a/.gitignore b/.gitignore
index 36c89cf56..d85f215d2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -109,3 +109,6 @@ landing/resources/xerj-sales-kit.zip
demo/screenshots/manifest.json
demo/.flow-data/
demo/.flow-chrome/
+
+# Local ES 8.13.4 reference node for fair benchmarking (never committed)
+.es-local/
diff --git a/demo/playbooks/LOSS_BATTLE_PLAN.md b/demo/playbooks/LOSS_BATTLE_PLAN.md
new file mode 100644
index 000000000..b07437fb9
--- /dev/null
+++ b/demo/playbooks/LOSS_BATTLE_PLAN.md
@@ -0,0 +1,353 @@
+# LOSS BATTLE PLAN — 2026-07-11
+
+Synthesis of 12 per-cell root-cause diagnoses of the remaining XERJ-vs-ES scorecard
+losses/DIFFs (CACHEHARDENED matrix, live ES 8.13.4 node on :9201). Clusters the
+diagnoses into shared-root-cause themes and orders the fixes into one-mechanism
+batches ranked by (cells flipped) x (correctness impact) / (effort x risk).
+
+**Principles (non-negotiable):**
+- Correctness before latency. A wrong top-10 is worse than a slow one.
+- One mechanism per batch. Each batch lands, validates, and gates alone.
+- Fair validation for EVERY batch = all four of:
+ 1. **fast==brute** byte-identical (forced brute path vs fast path, same corpus);
+ 2. **xerj==ES** live value cross-check, `request_cache=false`, never static data;
+ 3. **re-measured closed-loop Δms** vs the live ES node, cache off, 60-iter p50;
+ 4. **es-yaml-runner gate**: 1360 pass / 0 fail / 3 skip, full rerun.
+- Build scoped only: `cargo build --release -j 32 -p xerj-server`. Never workspace-wide.
+
+---
+
+## The one big finding
+
+**Nine of the twelve diagnosed cells share a single write-side root cause:**
+segment physical row order is NOT insertion (seq_no) order. Ingest hash-routes
+docs across 16 memtable shards (`xxh3_64(_id)&15`, memtable.rs:569), each shard
+flushes to its own segment (index.rs:12129 `do_flush_shard`), and merges pick
+inputs **size-sorted** (merge.rs:114; index.rs:2583-2598) and concatenate stored
+docs **without re-sorting by `_seq_no`** (index.rs:2888-2944; merge.rs:262-312).
+The merged bench segment therefore stores docs shard-grouped (live-verified: the
+first 6,240 rows all hash to shard 5; `match_all` returns 0,60,82,96,100,153,…).
+
+ES tie-breaks equal scores by internal doc id == arrival order. XERJ's final sort
+(score DESC, seq_no ASC — index.rs:7417-7437) is already ES-correct, but bounded
+collectors admit the first-k **physical** positions, so the seq sort only reorders
+the wrong survivor set. One merge-time fix (emit merged segments in global seq_no
+order) flips every constant-score/tied-score DIFF:top at once, with **no comparator
+changes anywhere** — once physical==seq, the existing doc_id tie-breaks (FTS
+`ScoredHit::Ord`, positional scans, prefiltered hydrate) all become the ES tie-break.
+
+The remaining losses are read-side: FTS O(matches) brute enumeration for query
+shapes not admitted to the ES-exact scored_columnar fast path, two scoring-formula
+defects (fuzzy similarity boost; mbp constant-score), missing track_total_hits
+early termination, and hydration/materialization waste.
+
+---
+
+## Themes
+
+| # | Theme | Cells | Root cause | Fix | Flips | Effort | Risk |
+|---|-------|-------|-----------|-----|-------|--------|------|
+| T1 | **segment-order-scramble** (write-side) | wildcard, prefix, range(@timestamp), terms(model), exists(cost_usd), deep from+size, + tie-order halves of fuzzy / multi_match / match_bool_prefix, + match_all | Per-shard flush + size-sorted merge concat, no `_seq_no` re-sort → physical order ≠ insertion order → tied-score truncation selects wrong docs | k-way `_seq_no`-ascending merge in `merge_pass_locked` (index.rs:2888-2944) + one-line `sort_by_key(_seq_no)` in storage `merge.rs:~300`; keep stored/ids/fts/dv four-way alignment via ONE permutation | 6 full DIFF→MATCH + 3 partial | M | Moderate (alignment invariant) |
+| T2 | **scored-fastpath coverage** (read-side) | multi_match, fuzzy, prefix, wildcard, match_bool_prefix | Shapes not admitted to scored_columnar → FTS brute path: O(all 25k-85k matches) postings enumeration + HashSet/HashMap churn for a 10-doc page; ES rewrites + early-terminates | Add lowering arms: MultiMatch(BestFields)→ScoredClause::DisMax; ScoredFilterLeaf::KeywordPrefix/Wildcard ord-range; ScoredLeafKind::KeywordFuzzy — all land on the proven ES-bit-exact TopKCand path | 5 latency LOSS→parity/WIN | S+M+M | Low-medium (None-fallback gated) |
+| T3 | **scoring-formula defects** | fuzzy (max_score), match_bool_prefix (max_score), multi_match (1 ulp) | Missing Lucene fuzzy similarity boost `1-dist/min(len)`; parser keeps BM25 where ES rewrites single-token mbp to constant_score 1.0; FTS BM25 f32 op order 1 ulp off `bm25_keyword_term_score` | Fold into T2 batches: per-expansion sim boost (both fast+brute); parser.rs:3539 `constant_score:true`; multi_match routes onto the ES-exact scorer | 3 DIFF→MATCH (bundled) | S | Low |
+| T4 | **tth-early-termination** | bool must+filter+should+must_not (+ residuals of term/prefix/wildcard counting) | scored_columnar Phase-2 walks ALL 100k rows for an exact total that rendering caps to (10000, gte); ES stops at ~27k. Proof: at tth=true XERJ already WINS 1.90 vs 2.33ms | Static `plan_max` upper bound over ClauseEval + break when heap full at plan_max AND total>limit+1 AND remaining seqs > heap-worst seq | 1 LOSS→WIN | M | Low (never fires <10k docs) |
+| T5 | **materialization/hydration waste** | term(cache_hit), range(@timestamp) latency, deep from+size latency | 256-doc materialisation floor for size:10 (index.rs:5196); range prefilter cached as HashSet, re-collected+sorted per query (index.rs:11493); deep-from parses all `from+size` docs | page_cap=from+size on the safe path; cache prefilter as pre-sorted Arc<Vec<u32>>; skip-parse pre-`from` positions under F1 gate | 1-3 LOSS→parity + ~0.5ms shaved off every size:10 cell | S+M | Low-medium (under-fill guard) |
+| T6 | **stale/noise — do not build** | agg:composite (stale diff), dis_max (±0.30ms band), exists latency (Δ0.14ms) | composite after_key shipped in d49410c, live-verified MATCH today; dis_max & exists are inside the ES-floor noise band | Harness-side only: refresh stale cached DIFF records; no engine code | 1 harness flip | S | Zero |
+
+---
+
+## Batch plan (ordered)
+
+### Batch 1 — T1: merged segments in seq_no order (correctness, the big flip)
+**Targets:** wildcard(model), prefix(model), range(@timestamp), terms(model),
+exists(cost_usd), deep from+size (order), match_all — plus the tie-order halves
+of fuzzy, multi_match, match_bool_prefix.
+
+**Why first:** flips the most cells (6 full DIFF:top→MATCH, 3 partial) and it is
+pure correctness — every later latency batch's early-exit logic (B3's constant-score
+stop-at-k, B4's brute parity, B6's tight page cap) is only *valid* once physical
+order == seq order. It also removes the need for any comparator changes: the
+existing doc_id tie-breaks become the ES tie-break by construction.
+
+**Mechanism (one):** make merged-segment physical order = global `_seq_no` order.
+- Engine merge: in `merge_pass_locked` (index.rs:2888-2944), replace the
+ per-segment concat with a k-way streaming merge by `_seq_no` (inputs are each
+ internally seq-ascending; `IdSeq` is already parsed per doc at :2890). Emit
+ merged_json_buf / ids_pairs / fts_input / DV-builder input from the SAME
+ permutation — the four-way alignment invariant (index.rs:2790-2796) is the
+ data-integrity hazard; derive all four from one stream. Stream, do not buffer
+ (M5.16 OOM history).
+- Storage merge: `merged_docs.sort_by_key(_seq_no)` before serializing Stored
+ (merge.rs:~300).
+- Optional: order merge input selection by `min_seq_no` (index.rs:2583-2598).
+
+**Files:** engine/crates/xerj-engine/src/index.rs:2888-2944, 2583-2598, 12129;
+engine/crates/xerj-storage/src/merge.rs:114, 262-312;
+engine/crates/xerj-engine/src/memtable.rs:1443-1452 (model for the sort).
+
+**Fair validation:** fix only affects NEWLY merged segments — **re-seed the 100k
+perf corpus (sequential bulk, ids 0..99999) + `_forcemerge?max_num_segments=1`
+on the fixed binary before measuring; stale-segment validation would falsely
+report no change.** Then, cache off both engines:
+(a) xerj==ES byte-identical top-10 for: match_all size:10 → 0..9;
+wildcard claude-* → 0..9 @1.0, totals 84731==84731; prefix claude- → 0..9;
+range(@timestamp) → 0..9; terms(model) → 0,2,3,4,5,8,10,19,24,25; exists →
+0..9; {match_all,from:500,size:50} → 500..549; regression probe term
+model=claude-sonnet-4-6 stays 1,6,12,13,14,15,16,17,22,26.
+(b) fast==brute byte-identical on prefilter/FTS vs forced full-scan paths.
+(c) integrity: GET /perf/_doc/0..9 sources correct; a scored BM25 top-10
+unchanged vs ES (stored/ids/dv alignment held).
+(d) gate 1360/0/3 full rerun, eyes on search/380_sort_segments_on_timestamp.
+(e) Δms: expect ~0 latency change (merge-time fix); the cells flip DIFF→MATCH.
+
+**Effort:** M. **Risk:** moderate — the four-way permutation alignment; merge
+loss-firewall (S1 abort) paths must not drop a cursor; stream to respect RAM.
+Sidecars are all rebuilt from the merged stream in the same function, so a
+single-permutation implementation keeps them consistent by construction.
+
+---
+
+### Batch 2 — T2a: multi_match(BestFields) → scored_columnar DisMax lowering
+**Targets:** multi_match (LOSS 2.49 vs 1.41ms + DIFF:max_score+top → MATCH+WIN).
+
+**Why second:** smallest effort (S), lowest risk, flips a full cell in both
+correctness (1-ulp max_score + tie order) and latency by pure routing onto the
+already-ES-bit-exact path — the semantically identical explicit dis_max measures
+1.27ms today. Zero new scoring code.
+
+**Mechanism (one):** add a `QueryNode::MultiMatch` arm to `scoring_clause()`
+(index.rs:20401), mirroring the DisMax arm at :20453 — gate on
+`match_type==BestFields`, `analyzer==None`, every field resolves via
+`scoring_leaf()` (kw/bool/num membership); lower to
+`ScoredClause::DisMax{clauses per field, tie_breaker:0.0}` with boost =
+mult x outer x per-field `^boost`. Non-conforming shapes return None → today's
+FTS fallback, zero behavior change.
+
+**Files:** engine/crates/xerj-engine/src/index.rs:20401, 20453, 20290-20324, 19105.
+
+**Fair validation:** (a) xerj==ES: cell body → ids [0,2,3,4,5,8,10,19,24,25],
+every score + max_score == 1.2821424 (not …425), total 27744 tracked; dis_max /
+match(model) probes stay identical. (b) fast==brute: plan-bailing decoration
+(non-empty memtable) returns same hit set+scores; a text-field multi_match is
+bit-identical pre/post (still FTS). (c) gate 1360/0/3 — watch multi_match YAML
+on keyword fields switching paths; tighten the gate (single analyzed token) if
+any operator nuance surfaces. (d) Δms: 2.49 → ~1.3ms vs ES 1.41 → WIN.
+
+**Effort:** S. **Risk:** low-moderate, fully None-gated.
+
+---
+
+### Batch 3 — T2b+T3: constant-score prefix/wildcard/mbp into scored_columnar
+**Targets:** prefix(model) (3.36 vs 1.64ms), wildcard(model) (3.81 vs 0.74ms),
+match_bool_prefix (2.11 vs 0.91ms + DIFF:max_score).
+
+**Why third:** three cells, one mechanism, and it lands on the proven path
+instead of hand-optimizing the FTS brute arm (the diagnoses offered both a k-way
+FTS merge and a columnar leaf; the columnar leaf is chosen because it reuses the
+ES-bit-exact TopKCand machinery and shares the direction of B2/B4).
+
+**Mechanism (one):** admit constant-score keyword prefix/wildcard as columnar
+filter leaves, and complete the ES constant-score rewrite for single-token mbp.
+- parser.rs:3539-3545: single-token match_bool_prefix →
+ `Prefix{constant_score:true}` (mirror of 4c69c05; do NOT touch the multi-token
+ trailing-prefix clause at :3572 — separate cell, alters scored bool sums).
+- `ScoredFilterLeaf::KeywordPrefix{field,prefix}` (index.rs:20034) admitted in
+ `scored_fast_plan` (:20207) strictly on `constant_score==true` + kw field
+ (internal lowerings pass false and MUST stay on BM25/FTS). Per segment resolve
+ prefix → contiguous ord range [lo,hi) via `partition_point` on the sorted
+ KeywordColumn terms (doc_values.rs:343-381); Wildcard same shape with an ord
+ bitmap via `term_matches_wildcard` over the (few) unique terms. Row walk +
+ existing TopKCand (score DESC, seq ASC), null_bitmap/ghost handling mirrored
+ from the KeywordTerm leaf. Score = f32(boost or 1.0), no BM25.
+
+**Files:** engine/crates/xerj-query/src/parser.rs:3539;
+engine/crates/xerj-engine/src/index.rs:20034, 20207, ~10200, ~10284, 20165;
+engine/crates/xerj-storage/src/doc_values.rs:343.
+
+**Fair validation:** (a) xerj==ES byte-identical: prefix claude- → 0..9 @1.0,
+84731==84731; boost variant max_score 2.5 bit-exact; wildcard claude-* same;
+mbp runbook → max_score 1.0, ids [0,12,21,23,34,39,41,46,51,54], totals
+25259==25259. (b) fast==brute: gate off the plan → identical hit SET + scores +
+total (post-B1, order identical too). (c) negative controls: query_string
+`model:claude-*` and multi-token mbp keep BM25 scores unchanged. (d) gate
+1360/0/3 (310_match_bool_prefix.yml has no single-token scoring case; YAML
+prefix suites run memtable-resident → brute path unchanged). (e) Δms: prefix
+~0.4-1.2ms (Δ −2.2), wildcard ~0.5-1.0ms (Δ −2.9), mbp ~1.0-1.3ms — all
+parity-to-WIN.
+
+**Effort:** M. **Risk:** low-medium — the `constant_score:true` discriminator
+must stay strict; empty ord range yields 0 hits not a bail-mismatch.
+
+---
+
+### Batch 4 — T2c+T3: fuzzy similarity boost + columnar keyword-fuzzy
+**Targets:** fuzzy(model) (2.03 vs 1.12ms, DIFF:max_score+top → MATCH, parity/WIN).
+
+**Mechanism (one):** ES-exact fuzzy scoring, both paths.
+- `ScoredLeafKind::KeywordFuzzy` in scoring_leaf (index.rs:20260): expand against
+ the segment KeywordColumn dictionary with the SAME Damerau-Levenshtein/case
+ predicate as `term_matches_fuzzy` (search.rs:1084; memtable-vs-segment must
+ agree — index.rs:19509-19542); per expansion sim = `1 - dist/min(cpLen)`;
+ score via `bm25_keyword_term_score(df_blend=max df, total, boost*sim)`
+ (ES BlendedTermQuery), sum per row across matching expansions; TopKCand gives
+ the seq tie-break. Cap expansions at 50 (ES max_expansions) → bail to brute.
+- Brute parity: `execute_fuzzy` (search.rs:798) passes the same per-term
+ `boost*sim` into `TermQuery::boosted` so fast==brute holds on scores.
+- AUTO fuzziness resolved via the existing parser path.
+
+**Files:** engine/crates/xerj-engine/src/index.rs:20260, 20117, 10227, 10074,
+20100, 19504; engine/crates/xerj-fts/src/search.rs:798, 840, 1084.
+
+**Fair validation:** (a) xerj==ES: cell → max_score 1.2020086, ids
+['0','2','3','4','5','8','10','19','24','25'] byte-identical; plus one
+multi-expansion fuzziness:2 case (exercises df-blend+sum). (b) fast==brute:
+columnar gate off → identical score multiset + total (bit-equal f32).
+(c) gate 1360/0/3 — grep YAML for fuzzy _score assertions (change moves TOWARD
+ES, expected green). (d) Δms: ~2.0 → ~1.1ms vs ES 1.1-1.7 → parity-to-WIN.
+
+**Effort:** M. **Risk:** low-medium — expansion-predicate semantics must be
+byte-identical across columnar/FTS/doc_matches_query or results diverge under
+writes.
+
+---
+
+### Batch 5 — T4: track_total_hits early termination in scored_columnar
+**Targets:** bool must+filter+should+must_not (2.15 vs 1.58ms LOSS → WIN).
+
+**Mechanism (one):** bounded walk break. Compute static f64 `plan_max` over the
+ClauseEval tree from clause_scores (Bool = Σmust + Σall-should; DisMax =
+max + tie*Σrest; msm only shrinks the match set); in the Phase-2 row loops,
+break when heap full AND total ≥ limit+1 (strict, so the renderer still emits
+(limit, gte) with zero plumbing) AND heap-worst score ≥ plan_max (one-ulp
+headroom; over-estimate only delays the break) AND remaining segments' min seq >
+heap-worst seq. `TrackTotalHits::True` disables the break. Do NOT fire for
+Pinned. Pass tth from the call site (index.rs:6215-6228).
+
+**Files:** engine/crates/xerj-engine/src/index.rs:10348, 10238-10257, 6215-6228,
+19932-20010 (mirror eval → max_bound), 20190-20205, 7651-7668.
+
+**Fair validation:** (a) fast==brute across the variant sweep (tth
+default/true/false/100, size 0/1/10/from, rare must, msm, dis_max wrapper,
+filter-only, constant_score). (b) xerj==ES: cell top-10
+0,3,4,5,10,19,24,32,33,41 @1.2947059, total {10000,gte}; at tth=true
+{37369,eq}. (c) gate 1360/0/3 (break can't fire below 10k docs). (d) Δms:
+~1.9 → ~1.1-1.2ms vs ES ~1.4-1.6 → WIN.
+
+**Effort:** M. **Risk:** low — an UNDER-estimated plan_max silently truncates
+top-k; mitigated by assume-all-match f64 bound + ulp headroom + the fast==brute
+sweep + the min-seq gate for multi-segment.
+
+---
+
+### Batch 6 — T5a: page-cap materialization (kill the 256-doc floor)
+**Targets:** term(cache_hit) (1.3 vs 0.66ms → ~0.7-0.9ms); shaves ~0.5ms off
+every plain size:10 cell that hydrates through the floor.
+
+**Mechanism (one):** `page_cap = from+size` instead of `(from+size+100).max(256)`
+on the safe path only: sort is _score/_doc, `!deletes_present`, empty memtable
+(or no id overlap), no rescore/collapse/highlight/post_filter — every other path
+keeps the 256 floor bit-identical (ghost/superseded/dedup `continue`s under-fill
+tight pages). Shrink `fts_cap` (index.rs:6452) and the hydration-loop break
+(index.rs:6647) to match. Optional same-walk lever: hoist `fields.get(field)`
+out of scan_term + norms ascending cursor / uniform-norm constant score — land
+only if f32 score equality holds exactly, else drop.
+
+**Files:** engine/crates/xerj-engine/src/index.rs:5196, 6452, 6647;
+engine/crates/xerj-fts/src/search.rs:518; engine/crates/xerj-fts/src/index.rs:1269.
+
+**Fair validation:** (a) fast==brute: byte-compare full _search JSON capped vs
+256-floor (env-flag forced) across size {1,10,50,100} x from {0,150,400,900} x
+_source variants x default/_doc sort, on the clean corpus AND an
+overwrites+deletes index (must fall back to the floor there). (b) xerj==ES:
+term(cache_hit true/false), term(tenant), rare term stay exact MATCH.
+(c) gate 1360/0/3, esp. pagination + scroll suites. (d) Δms: 1.4-1.7 →
+0.9-1.1ms (cap alone), 0.7-0.9 with the norms cursor; size:0 stays 0.40ms.
+Residual ~0.2-0.4ms is ES's block-max WAND floor — accept (see out-of-scope).
+
+**Effort:** S. **Risk:** page under-fill — fully gated; do NOT touch the
+sort:['_doc'] _id-string defect in the same change.
+
+---
+
+### Batch 7 — T5b: prefiltered-hydrate waste (sorted-Vec cache + deep-from skip-parse)
+**Targets:** range(@timestamp) latency (2.12 vs 0.93 → ≤1.5ms), deep from+size
+latency (from:500 ~1.9 → ~1.3ms; from:5000 12.5 → ~2ms), terms(model) residual.
+
+**Mechanism (one):** stop re-doing per-query work the prefilter/scan already did.
+- `range_prefilter_cache` / `build_term_prefilter_cached` (index.rs:538, 10606,
+ 10764): cache positions as a pre-sorted `Arc<Vec<u32>>` (built
+ position-ascending during the existing ords walk — no HashSet, no per-query
+ collect+sort); `hydrate_prefiltered_unsorted` (index.rs:11493) iterates the
+ sorted slice with its existing early-break; membership checks in
+ `scan_stored_section_into` use binary search or a bitvec twin. Keep the
+ bool-prefilter intersection working (sorted-vec intersect or retain HashSet
+ there) — scope contained to term/terms/range leaves + hydrate.
+- Deep-from: thread `from` into `scan_stored_section_into` (index.rs:11810);
+ under the F1 gate (match_all, count_authoritative, no peeled scorer,
+ `deletes_present==false`) brace-scan-skip the first `from` positions and
+ simd_json-parse only [from, from+size); full parse on any gate failure.
+
+**Files:** engine/crates/xerj-engine/src/index.rs:538, 10606, 10733-10785,
+11493-11495, 11810-11894, 5196.
+
+**Fair validation:** (a) xerj==ES (post-B1 corpus): range top-10 == 0..9,
+totals 100000==100000 tracked; from:500 → 500..549; from:0 and from:5000
+windows byte-identical. (b) fast==brute: prefiltered vs forced full-scan
+byte-identical; skip-parse page == full-parse page byte-identical, and on a
+deletes-bearing index the gate must fall back (verified by identical output).
+(c) gate 1360/0/3. (d) Δms: range 2.12 → ~1.0-1.5 (DIFF already fixed in B1;
+residual is 256-doc parse slack, further shrunk by B6); from:500 Δ→~0;
+from:5000 linear term gone.
+
+**Effort:** M. **Risk:** low-medium — membership semantics must stay identical
+or `count_authoritative` totals regress; skip-parse strictly gated.
+
+---
+
+### Batch 0 (parallel, no build) — T6: refresh stale harness records
+**Targets:** agg:composite. after_key parity shipped in d49410c and is
+live-verified byte-identical today (including mid-stream `after` pagination and
+the exhausted final page omitting after_key); the scorecard already records
+MATCH (FULL_MATRIX_SCORECARD_2026-07-10_CACHEHARDENED.md:64). Re-run the fair
+probe, delete any pre-d49410c cached expected-response snapshot. Zero engine code.
+
+---
+
+## Out of scope (honest)
+
+- **dis_max (NOISE_ES_FLOOR):** Δ0.25ms inside the ±0.30ms TIE band; even a
+ perfect LUT row-walk leaves it TIE. Do not spend a build. Already MATCH on
+ correctness (bit-exact).
+- **exists(cost_usd) latency:** Δ0.14ms, both engines at the HTTP+render floor.
+ Correctness half is flipped by Batch 1.
+- **agg:composite:** stale diff, harness refresh only (Batch 0).
+- **ES block-max WAND / impact-metadata early termination (NEEDS_INFRA):** the
+ last ~0.2-0.4ms on term(cache_hit)-class cells and the exact-count-under-tth
+ enumeration floor need maxscore/impact postings metadata — real infra, not a
+ batch here.
+- **`sort:["_doc"]` renders lexicographic `_id` order** (parser.rs:2612 /
+ sort.rs:69 / index.rs:18558): latent off-scorecard bug, matches neither ES nor
+ XERJ's own stored order. Separate ticket — do NOT bundle with any batch above
+ (YAML tests may pin orderings).
+- **size:0 / _count brute cliff** (bool body ~107ms, dis_max size:0 ~162ms vs ES
+ <1ms — `scored_fast_plan` bails on size==0 at index.rs:20216): NOT one of these
+ cells, but the largest single ratio found during diagnosis. File as its own
+ high-ROI follow-up batch (extend the columnar scorer to count+max_score-only).
+- **Multi-segment (unsettled) exactness pre-merge:** after Batch 1, tie order is
+ exact for settled/force-merged states (the benchmark state); interleaved
+ per-shard flush segments are only approximately insertion-ordered until merged.
+ Per-segment admission caps (exists diagnosis step 3) are a separate follow-up
+ commit if live-write benchmarks ever pin it.
+
+## Expected scorecard movement
+
+| Batch | Cells flipped | Kind |
+|-------|--------------|------|
+| B1 | wildcard, prefix, range, terms, exists, deep-from (+match_all) | 6+ DIFF:top→MATCH |
+| B2 | multi_match | DIFF→MATCH + LOSS→WIN |
+| B3 | prefix, wildcard, mbp | 3 LOSS→parity/WIN + mbp DIFF→MATCH |
+| B4 | fuzzy | DIFF→MATCH + LOSS→parity/WIN |
+| B5 | bool m+f+s+mn | LOSS→WIN |
+| B6 | term(cache_hit) | LOSS→near-parity (+~0.5ms off all size:10 cells) |
+| B7 | range, deep-from | 2 LOSS→parity |
+| B0 | composite | stale DIFF→MATCH (harness) |
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 9df15f771..aaca3d319 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -4862,9 +4862,13 @@ pub async fn search(
// declared at create time (or after close/reopen picks up the current
// mapping). Mirror that behaviour using the `__xy_index_sort_*` hints
// stored in settings at create / reopen time.
+ //
+ // For the implicit auto-@timestamp hint this is LEAF ordering, not a
+ // per-doc sort — see `implicit_leaf_ts` below.
+ let mut implicit_leaf_ts: Option<String> = None;
if index_names.len() == 1 {
let idx_name = index_names[0];
- let (sort_field, sort_order, explicit) = state
+ let (sort_field, sort_order, explicit, reopen_derived) = state
.engine
.index_settings
.get(idx_name)
@@ -4883,9 +4887,13 @@ pub async fn search(
.get("__xy_index_sort_explicit")
.and_then(Value::as_bool)
.unwrap_or(false);
- (f, o, ex)
+ let ro = s
+ .get("__xy_index_sort_reopen")
+ .and_then(Value::as_bool)
+ .unwrap_or(false);
+ (f, o, ex, ro)
})
- .unwrap_or((None, "desc".to_string(), false));
+ .unwrap_or((None, "desc".to_string(), false, false));
if let Some(f) = sort_field {
// Apply the index sort ONLY for an EXPLICITLY declared
// `index.sort` (settings), when no sort was requested or the
@@ -4906,15 +4914,24 @@ pub async fn search(
// 1.5 M docs (~2-7 s each, `walked=1500002 admitted=250252
// allhits_cap=false`) while the same query with an explicit
// `sort: _doc` took 45 ms.
- // For the implicit auto-@timestamp hint, only match_all-shaped
- // queries get the injected sort: that is the query shape the ES
- // conformance contract observes (search/380_sort_segments_on_
- // timestamp.yml), and the engine serves it via the bounded
- // sorted-DV candidates path (O(from+size) per segment).
- // Filtered queries (term/range/bool) skip the injection — in
- // real ES the physical index sort doesn't impose a query-time
- // global sort on them either, and routing them through the
- // top-N heap costs a full O(total-docs) walk per request.
+ // For the implicit auto-@timestamp hint, real ES (7.16+
+ // "sort segments on timestamp") orders the LEAVES — segments
+ // are read newest-max-@timestamp-first while docs inside each
+ // segment keep arrival order, scores stay 1.0 and max_score
+ // stays 1.0. The old implementation approximated that with an
+ // injected per-doc `@timestamp desc` sort, which diverges from
+ // real ES on any settled index: page windows are cut by
+ // timestamp value (from:500 returned timestamp-ranked ids, ES
+ // returns arrival ids 500..549), equal timestamps re-ordered
+ // lexicographically by _id, and the sort suppressed max_score
+ // (null vs ES 1.0) — all three live-verified against ES 8.13.4.
+ // So for the implicit hint pass a leaf-order flag to the engine
+ // instead (`SearchRequest::leaf_ts_field`), only for the
+ // match_all-shaped sortless requests the old injection covered
+ // (the shape search/380_sort_segments_on_timestamp.yml
+ // observes). Filtered queries (term/range/bool) are untouched
+ // — real ES's segment ordering doesn't impose a query-time
+ // global sort on them either.
let implicit_ok = body.query.is_none()
|| body
.query
@@ -4922,11 +4939,26 @@ pub async fn search(
.and_then(|q| q.as_object())
.map(|o| o.len() == 1 && o.contains_key("match_all"))
.unwrap_or(false);
- let apply = (explicit || implicit_ok)
- && (body.sort.is_none()
- || (explicit && body.sort.as_ref().map(is_lone_doc_sort).unwrap_or(false)));
- if apply {
- body.sort = Some(json!([{ f: { "order": sort_order } }]));
+ if explicit {
+ // Explicitly declared `index.sort` (settings): keep the
+ // injected per-doc sort — ES returns docs in index-sort
+ // order for sortless and `sort: _doc` requests there.
+ let apply = body.sort.is_none()
+ || body.sort.as_ref().map(is_lone_doc_sort).unwrap_or(false);
+ if apply {
+ body.sort = Some(json!([{ f: { "order": sort_order } }]));
+ }
+ } else if implicit_ok && body.sort.is_none() {
+ if reopen_derived {
+ // Reopen-derived hint: the close-time full-drain flush
+ // already consolidated the per-refresh leaves ES would
+ // have kept, so settled leaf ordering can't reproduce
+ // the reopen contract — keep the historical per-doc
+ // sort for these indices (see the `_open` handler).
+ body.sort = Some(json!([{ f: { "order": sort_order } }]));
+ } else {
+ implicit_leaf_ts = Some(f);
+ }
}
}
}
@@ -6118,7 +6150,13 @@ pub async fn search(
.collect();
let search_req = match build_search_request(&body, aggs_value) {
- Ok(r) => r,
+ Ok(mut r) => {
+ // ES 7.16 leaf ordering for the implicit auto-@timestamp hint
+ // (see the injection block above) — internal field, never on
+ // the wire.
+ r.leaf_ts_field = implicit_leaf_ts;
+ r
+ }
Err(e) => return ApiError::new(e).into_response(),
};
@@ -16956,6 +16994,16 @@ pub async fn open_index(
};
merged.insert("__xy_index_sort_field".to_string(), json!("@timestamp"));
merged.insert("__xy_index_sort_order".to_string(), json!("desc"));
+ // Reopen-derived hint (dynamic @timestamp picked up at
+ // close/open): `_close` drains the whole memtable into ONE
+ // consolidated segment, so the per-refresh leaf structure ES
+ // preserves is already gone by the first post-reopen search.
+ // Mark the provenance so the search path keeps the
+ // historical per-doc `@timestamp desc` sort for these
+ // indices (the 380_sort_segments_on_timestamp.yml reopen
+ // contract) instead of the settled leaf-order semantics
+ // reserved for create-time-mapped indices.
+ merged.insert("__xy_index_sort_reopen".to_string(), json!(true));
state
.engine
.index_settings
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 17a5bdfe8..43340ed34 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -2795,6 +2795,24 @@ impl Index {
// all stored docs even if a doc fails the FTS/DV re-parse.
let mut ids_pairs: Vec<(u64, String)> = Vec::new();
+ // ES tie-order parity (LOSS_BATTLE_PLAN B1): collect every
+ // surviving doc as (seq_no, _id, raw JSON) FIRST, then sort
+ // globally by _seq_no and derive ALL four merge outputs
+ // (stored bytes / .ids side-car / FTS input / DV input) from
+ // that ONE sorted stream. Ingest hash-routes docs across 16
+ // memtable shards and each shard flushes its own segment, so
+ // input-order concatenation scrambled physical row order vs
+ // insertion (_seq_no) order — ES tie-breaks equal scores by
+ // internal doc id == arrival order, so every constant-score
+ // top-k diverged from ES. With physical == seq order the
+ // existing doc-position tie-breaks ARE the ES tie-break; no
+ // comparator changes anywhere. Memory: the owned raw copy
+ // replaces the same bytes that previously went straight into
+ // `merged_json_buf` (drained into it below), so the peak
+ // stays ~1× stored size + the fts_input Values, unchanged
+ // from the M5.22 profile.
+ let mut survivors: Vec<(u64, String, String)> = Vec::new();
+
for meta in &metas_for_task {
min_seq = min_seq.min(meta.min_seq_no);
max_seq = max_seq.max(meta.max_seq_no);
@@ -2913,34 +2931,53 @@ impl Index {
}
}
- // Survives — append raw bytes to the merged buffer.
- if !first_doc {
- merged_json_buf.push(b',');
- }
- first_doc = false;
- merged_json_buf.extend_from_slice(raw_str.as_bytes());
- live_doc_count += 1;
- ids_pairs.push((id_seq.seq_no, id_seq.id.to_string()));
+ // Survives — hold (seq, id, raw) for the global
+ // _seq_no-ordered build below.
+ survivors.push((
+ id_seq.seq_no,
+ id_seq.id.to_string(),
+ raw_str.to_string(),
+ ));
+ }
+ // raw_docs + stored_bytes drop here — segment RAM reclaimed.
+ }
- // Per-doc Value parse for FTS / DV builders.
- let doc_value: Value = match serde_json::from_str(raw_str) {
- Ok(v) => v,
- Err(_) => continue,
- };
- let id_str = id_seq.id.to_string();
- let source = doc_value.get("_source").cloned().unwrap_or(Value::Null);
- let mut fields: HashMap<String, String> = HashMap::new();
- if let Some(obj) = source.as_object() {
- for (key, val) in obj {
- let text = extract_field_text(val);
- if !text.is_empty() {
- fields.insert(key.clone(), text);
- }
+ // Global insertion-order (_seq_no) sort — see the B1 note
+ // above. seq_nos are unique per surviving doc (stale /
+ // duplicate copies were filtered by the version map).
+ survivors.sort_unstable_by_key(|(seq, _, _)| *seq);
+
+ // Single sorted stream → all four outputs. `into_iter`
+ // frees each raw String right after its bytes are copied
+ // into `merged_json_buf`, keeping peak memory ~1× stored.
+ for (seq_no, id_str, raw_str) in survivors {
+ if !first_doc {
+ merged_json_buf.push(b',');
+ }
+ first_doc = false;
+ merged_json_buf.extend_from_slice(raw_str.as_bytes());
+ live_doc_count += 1;
+ // Pushed BEFORE the per-doc Value parse that can
+ // `continue` — the .ids side-car must cover every
+ // stored doc (alignment invariant, see above).
+ ids_pairs.push((seq_no, id_str.clone()));
+
+ // Per-doc Value parse for FTS / DV builders.
+ let doc_value: Value = match serde_json::from_str(&raw_str) {
+ Ok(v) => v,
+ Err(_) => continue,
+ };
+ let source = doc_value.get("_source").cloned().unwrap_or(Value::Null);
+ let mut fields: HashMap<String, String> = HashMap::new();
+ if let Some(obj) = source.as_object() {
+ for (key, val) in obj {
+ let text = extract_field_text(val);
+ if !text.is_empty() {
+ fields.insert(key.clone(), text);
}
}
- fts_input.push((id_str, fields, source));
}
- // raw_docs + stored_bytes drop here — segment RAM reclaimed.
+ fts_input.push((id_str, fields, source));
}
if live_doc_count == 0 {
@@ -4063,6 +4100,7 @@ impl Index {
script_fields: None,
fields: Vec::new(),
profile: false,
+ leaf_ts_field: None,
};
// Box::pin to break the type-recursion (search_inner ↔
// run_hybrid both async fn).
@@ -4998,6 +5036,50 @@ impl Index {
)));
}
+ // Implicit @timestamp "sort segments on timestamp" handling (see
+ // `SearchRequest::leaf_ts_field`). Two regimes:
+ //
+ // * SETTLED (empty memtable, ≤1 segment — the read-mostly steady
+ // state): apply NO ordering at all. A single seq-ordered segment
+ // (the B1 merge invariant) already reads back in arrival order,
+ // which is exactly what real ES 8.13.4 returns on the same
+ // single-segment index — leaf ordering over one leaf is a no-op
+ // there, hits keep `_score` 1.0 and `max_score` 1.0. The old
+ // injected per-doc sort diverged from ES here on all three axes
+ // (page windows, equal-ts lex re-order, max_score null) —
+ // live-verified.
+ //
+ // * UNSETTLED (buffered memtable docs, or multiple segments):
+ // xerj's visibility units don't line up with ES's
+ // refresh-generation leaves (a `?refresh=true` doc stays
+ // memtable-resident here but is its own newest-first leaf in ES),
+ // and the final hit ordering ties break by global seq — so
+ // segment-level iteration order can't reproduce ES's leaf order.
+ // Keep the historical injected per-doc `@timestamp desc` sort for
+ // this window — the behaviour the ES conformance contract
+ // (search/380_sort_segments_on_timestamp.yml) validates.
+ let leaf_fallback_request: Option<SearchRequest> = match request.leaf_ts_field.as_deref() {
+ Some(ts_field)
+ if matches!(request.query, QueryNode::MatchAll)
+ && request.sort.is_empty()
+ && (self.memtable.doc_count() > 0
+ || self.store.snapshot().segments.len() > 1) =>
+ {
+ let mut r = request.clone();
+ r.sort = vec![xerj_query::sort::SortField {
+ field: ts_field.to_string(),
+ order: xerj_query::sort::SortOrder::Desc,
+ mode: xerj_query::sort::SortMode::default(),
+ missing: xerj_query::sort::SortMissing::default(),
+ format: None,
+ }];
+ r.leaf_ts_field = None;
+ Some(r)
+ }
+ _ => None,
+ };
+ let request: &SearchRequest = leaf_fallback_request.as_ref().unwrap_or(request);
+
// ── Cooperative deadline ──────────────────────────────────────────
// The `tokio::time::timeout` wrapper in `search()` is DEAD CODE on
// the multi-thread runtime: the search body runs inside
diff --git a/engine/crates/xerj-query/src/ast.rs b/engine/crates/xerj-query/src/ast.rs
index 21d7d269f..9c24b0e92 100644
--- a/engine/crates/xerj-query/src/ast.rs
+++ b/engine/crates/xerj-query/src/ast.rs
@@ -972,6 +972,18 @@ pub struct SearchRequest {
/// dropped before pagination, aggregations, and total counting.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub min_score: Option<f64>,
+
+ /// INTERNAL (never on the wire — `serde(skip)`): iterate segments in
+ /// per-segment-max-of-this-date-field DESC order for sortless
+ /// match_all-shaped requests. This is ES 7.16's "sort segments on
+ /// timestamp" LEAF ordering for indices that map `@timestamp` at
+ /// create time: segments are read newest-first while docs inside each
+ /// segment keep arrival order and hits keep `_score` 1.0 / max_score
+ /// 1.0. Set by the ES-compat layer; replaces the old injected per-doc
+ /// `@timestamp desc` sort, which diverged from real ES on settled
+ /// indices (wrong page windows + `max_score: null`).
+ #[serde(skip)]
+ pub leaf_ts_field: Option<String>,
}
fn default_query() -> QueryNode {
@@ -1002,6 +1014,7 @@ impl Default for SearchRequest {
collapse: None,
rescore: Vec::new(),
min_score: None,
+ leaf_ts_field: None,
}
}
}
diff --git a/engine/crates/xerj-query/src/parser.rs b/engine/crates/xerj-query/src/parser.rs
index 40e8aa3db..6f4f83466 100644
--- a/engine/crates/xerj-query/src/parser.rs
+++ b/engine/crates/xerj-query/src/parser.rs
@@ -238,6 +238,7 @@ pub fn parse_request(body: &Value) -> Result<SearchRequest> {
collapse: None,
rescore: Vec::new(),
min_score: None,
+ leaf_ts_field: None,
})
}
diff --git a/engine/crates/xerj-storage/src/merge.rs b/engine/crates/xerj-storage/src/merge.rs
index ce4ed7d9d..de8d8535d 100644
--- a/engine/crates/xerj-storage/src/merge.rs
+++ b/engine/crates/xerj-storage/src/merge.rs
@@ -302,6 +302,15 @@ impl MergeExecutor {
}
}
+ // Emit the merged segment in global insertion (_seq_no) order. Inputs
+ // are per-memtable-shard segments (hash-partitioned by _id), so plain
+ // concatenation scrambles physical row order vs arrival order; ES
+ // tie-breaks equal scores by internal doc id == arrival order, and the
+ // engine-side merge (index.rs merge_pass_locked) applies the same sort.
+ merged_docs.sort_unstable_by_key(|doc| {
+ doc.get("_seq_no").and_then(|v| v.as_u64()).unwrap_or(0)
+ });
+
let live_doc_count = merged_docs.len() as u64;
// Write merged segment — inherit schema version from the snapshot if available