AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(order): ES-exact doc order — seq_no-sorted merges + settled leaf semantics (flips 6 DIFF cells)
xerj-orgaaa4b59d152d
diff --git a/.gitignore b/.gitignoreindex 36c89cf56..d85f215d2 100644--- a/.gitignore+++ b/.gitignore@@ -109,3 +109,6 @@ landing/resources/xerj-sales-kit.zipdemo/screenshots/manifest.jsondemo/.flow-data/demo/.flow-chrome/++# Local ES 8.13.4 reference node for fair benchmarking (never committed)+.es-local/diff --git a/demo/playbooks/LOSS_BATTLE_PLAN.md b/demo/playbooks/LOSS_BATTLE_PLAN.mdnew file mode 100644index 000000000..b07437fb9--- /dev/null+++ b/demo/playbooks/LOSS_BATTLE_PLAN.md@@ -0,0 +1,353 @@+# LOSS BATTLE PLAN — 2026-07-11++Synthesis of 12 per-cell root-cause diagnoses of the remaining XERJ-vs-ES scorecard+losses/DIFFs (CACHEHARDENED matrix, live ES 8.13.4 node on :9201). Clusters the+diagnoses into shared-root-cause themes and orders the fixes into one-mechanism+batches ranked by (cells flipped) x (correctness impact) / (effort x risk).++**Principles (non-negotiable):**+- Correctness before latency. A wrong top-10 is worse than a slow one.+- One mechanism per batch. Each batch lands, validates, and gates alone.+- Fair validation for EVERY batch = all four of:+ 1. **fast==brute** byte-identical (forced brute path vs fast path, same corpus);+ 2. **xerj==ES** live value cross-check, `request_cache=false`, never static data;+ 3. **re-measured closed-loop Δms** vs the live ES node, cache off, 60-iter p50;+ 4. **es-yaml-runner gate**: 1360 pass / 0 fail / 3 skip, full rerun.+- Build scoped only: `cargo build --release -j 32 -p xerj-server`. Never workspace-wide.++---++## The one big finding++**Nine of the twelve diagnosed cells share a single write-side root cause:**+segment physical row order is NOT insertion (seq_no) order. Ingest hash-routes+docs across 16 memtable shards (`xxh3_64(_id)&15`, memtable.rs:569), each shard+flushes to its own segment (index.rs:12129 `do_flush_shard`), and merges pick+inputs **size-sorted** (merge.rs:114; index.rs:2583-2598) and concatenate stored+docs **without re-sorting by `_seq_no`** (index.rs:2888-2944; merge.rs:262-312).+The merged bench segment therefore stores docs shard-grouped (live-verified: the+first 6,240 rows all hash to shard 5; `match_all` returns 0,60,82,96,100,153,…).++ES tie-breaks equal scores by internal doc id == arrival order. XERJ's final sort+(score DESC, seq_no ASC — index.rs:7417-7437) is already ES-correct, but bounded+collectors admit the first-k **physical** positions, so the seq sort only reorders+the wrong survivor set. One merge-time fix (emit merged segments in global seq_no+order) flips every constant-score/tied-score DIFF:top at once, with **no comparator+changes anywhere** — once physical==seq, the existing doc_id tie-breaks (FTS+`ScoredHit::Ord`, positional scans, prefiltered hydrate) all become the ES tie-break.++The remaining losses are read-side: FTS O(matches) brute enumeration for query+shapes not admitted to the ES-exact scored_columnar fast path, two scoring-formula+defects (fuzzy similarity boost; mbp constant-score), missing track_total_hits+early termination, and hydration/materialization waste.++---++## Themes++| # | Theme | Cells | Root cause | Fix | Flips | Effort | Risk |+|---|-------|-------|-----------|-----|-------|--------|------|+| T1 | **segment-order-scramble** (write-side) | wildcard, prefix, range(@timestamp), terms(model), exists(cost_usd), deep from+size, + tie-order halves of fuzzy / multi_match / match_bool_prefix, + match_all | Per-shard flush + size-sorted merge concat, no `_seq_no` re-sort → physical order ≠ insertion order → tied-score truncation selects wrong docs | k-way `_seq_no`-ascending merge in `merge_pass_locked` (index.rs:2888-2944) + one-line `sort_by_key(_seq_no)` in storage `merge.rs:~300`; keep stored/ids/fts/dv four-way alignment via ONE permutation | 6 full DIFF→MATCH + 3 partial | M | Moderate (alignment invariant) |+| T2 | **scored-fastpath coverage** (read-side) | multi_match, fuzzy, prefix, wildcard, match_bool_prefix | Shapes not admitted to scored_columnar → FTS brute path: O(all 25k-85k matches) postings enumeration + HashSet/HashMap churn for a 10-doc page; ES rewrites + early-terminates | Add lowering arms: MultiMatch(BestFields)→ScoredClause::DisMax; ScoredFilterLeaf::KeywordPrefix/Wildcard ord-range; ScoredLeafKind::KeywordFuzzy — all land on the proven ES-bit-exact TopKCand path | 5 latency LOSS→parity/WIN | S+M+M | Low-medium (None-fallback gated) |+| T3 | **scoring-formula defects** | fuzzy (max_score), match_bool_prefix (max_score), multi_match (1 ulp) | Missing Lucene fuzzy similarity boost `1-dist/min(len)`; parser keeps BM25 where ES rewrites single-token mbp to constant_score 1.0; FTS BM25 f32 op order 1 ulp off `bm25_keyword_term_score` | Fold into T2 batches: per-expansion sim boost (both fast+brute); parser.rs:3539 `constant_score:true`; multi_match routes onto the ES-exact scorer | 3 DIFF→MATCH (bundled) | S | Low |+| T4 | **tth-early-termination** | bool must+filter+should+must_not (+ residuals of term/prefix/wildcard counting) | scored_columnar Phase-2 walks ALL 100k rows for an exact total that rendering caps to (10000, gte); ES stops at ~27k. Proof: at tth=true XERJ already WINS 1.90 vs 2.33ms | Static `plan_max` upper bound over ClauseEval + break when heap full at plan_max AND total>limit+1 AND remaining seqs > heap-worst seq | 1 LOSS→WIN | M | Low (never fires <10k docs) |+| T5 | **materialization/hydration waste** | term(cache_hit), range(@timestamp) latency, deep from+size latency | 256-doc materialisation floor for size:10 (index.rs:5196); range prefilter cached as HashSet, re-collected+sorted per query (index.rs:11493); deep-from parses all `from+size` docs | page_cap=from+size on the safe path; cache prefilter as pre-sorted Arc<Vec<u32>>; skip-parse pre-`from` positions under F1 gate | 1-3 LOSS→parity + ~0.5ms shaved off every size:10 cell | S+M | Low-medium (under-fill guard) |+| T6 | **stale/noise — do not build** | agg:composite (stale diff), dis_max (±0.30ms band), exists latency (Δ0.14ms) | composite after_key shipped in d49410c, live-verified MATCH today; dis_max & exists are inside the ES-floor noise band | Harness-side only: refresh stale cached DIFF records; no engine code | 1 harness flip | S | Zero |++---++## Batch plan (ordered)++### Batch 1 — T1: merged segments in seq_no order (correctness, the big flip)+**Targets:** wildcard(model), prefix(model), range(@timestamp), terms(model),+exists(cost_usd), deep from+size (order), match_all — plus the tie-order halves+of fuzzy, multi_match, match_bool_prefix.++**Why first:** flips the most cells (6 full DIFF:top→MATCH, 3 partial) and it is+pure correctness — every later latency batch's early-exit logic (B3's constant-score+stop-at-k, B4's brute parity, B6's tight page cap) is only *valid* once physical+order == seq order. It also removes the need for any comparator changes: the+existing doc_id tie-breaks become the ES tie-break by construction.++**Mechanism (one):** make merged-segment physical order = global `_seq_no` order.+- Engine merge: in `merge_pass_locked` (index.rs:2888-2944), replace the+ per-segment concat with a k-way streaming merge by `_seq_no` (inputs are each+ internally seq-ascending; `IdSeq` is already parsed per doc at :2890). Emit+ merged_json_buf / ids_pairs / fts_input / DV-builder input from the SAME+ permutation — the four-way alignment invariant (index.rs:2790-2796) is the+ data-integrity hazard; derive all four from one stream. Stream, do not buffer+ (M5.16 OOM history).+- Storage merge: `merged_docs.sort_by_key(_seq_no)` before serializing Stored+ (merge.rs:~300).+- Optional: order merge input selection by `min_seq_no` (index.rs:2583-2598).++**Files:** engine/crates/xerj-engine/src/index.rs:2888-2944, 2583-2598, 12129;+engine/crates/xerj-storage/src/merge.rs:114, 262-312;+engine/crates/xerj-engine/src/memtable.rs:1443-1452 (model for the sort).++**Fair validation:** fix only affects NEWLY merged segments — **re-seed the 100k+perf corpus (sequential bulk, ids 0..99999) + `_forcemerge?max_num_segments=1`+on the fixed binary before measuring; stale-segment validation would falsely+report no change.** Then, cache off both engines:+(a) xerj==ES byte-identical top-10 for: match_all size:10 → 0..9;+wildcard claude-* → 0..9 @1.0, totals 84731==84731; prefix claude- → 0..9;+range(@timestamp) → 0..9; terms(model) → 0,2,3,4,5,8,10,19,24,25; exists →+0..9; {match_all,from:500,size:50} → 500..549; regression probe term+model=claude-sonnet-4-6 stays 1,6,12,13,14,15,16,17,22,26.+(b) fast==brute byte-identical on prefilter/FTS vs forced full-scan paths.+(c) integrity: GET /perf/_doc/0..9 sources correct; a scored BM25 top-10+unchanged vs ES (stored/ids/dv alignment held).+(d) gate 1360/0/3 full rerun, eyes on search/380_sort_segments_on_timestamp.+(e) Δms: expect ~0 latency change (merge-time fix); the cells flip DIFF→MATCH.++**Effort:** M. **Risk:** moderate — the four-way permutation alignment; merge+loss-firewall (S1 abort) paths must not drop a cursor; stream to respect RAM.+Sidecars are all rebuilt from the merged stream in the same function, so a+single-permutation implementation keeps them consistent by construction.++---++### Batch 2 — T2a: multi_match(BestFields) → scored_columnar DisMax lowering+**Targets:** multi_match (LOSS 2.49 vs 1.41ms + DIFF:max_score+top → MATCH+WIN).++**Why second:** smallest effort (S), lowest risk, flips a full cell in both+correctness (1-ulp max_score + tie order) and latency by pure routing onto the+already-ES-bit-exact path — the semantically identical explicit dis_max measures+1.27ms today. Zero new scoring code.++**Mechanism (one):** add a `QueryNode::MultiMatch` arm to `scoring_clause()`+(index.rs:20401), mirroring the DisMax arm at :20453 — gate on+`match_type==BestFields`, `analyzer==None`, every field resolves via+`scoring_leaf()` (kw/bool/num membership); lower to+`ScoredClause::DisMax{clauses per field, tie_breaker:0.0}` with boost =+mult x outer x per-field `^boost`. Non-conforming shapes return None → today's+FTS fallback, zero behavior change.++**Files:** engine/crates/xerj-engine/src/index.rs:20401, 20453, 20290-20324, 19105.++**Fair validation:** (a) xerj==ES: cell body → ids [0,2,3,4,5,8,10,19,24,25],+every score + max_score == 1.2821424 (not …425), total 27744 tracked; dis_max /+match(model) probes stay identical. (b) fast==brute: plan-bailing decoration+(non-empty memtable) returns same hit set+scores; a text-field multi_match is+bit-identical pre/post (still FTS). (c) gate 1360/0/3 — watch multi_match YAML+on keyword fields switching paths; tighten the gate (single analyzed token) if+any operator nuance surfaces. (d) Δms: 2.49 → ~1.3ms vs ES 1.41 → WIN.++**Effort:** S. **Risk:** low-moderate, fully None-gated.++---++### Batch 3 — T2b+T3: constant-score prefix/wildcard/mbp into scored_columnar+**Targets:** prefix(model) (3.36 vs 1.64ms), wildcard(model) (3.81 vs 0.74ms),+match_bool_prefix (2.11 vs 0.91ms + DIFF:max_score).++**Why third:** three cells, one mechanism, and it lands on the proven path+instead of hand-optimizing the FTS brute arm (the diagnoses offered both a k-way+FTS merge and a columnar leaf; the columnar leaf is chosen because it reuses the+ES-bit-exact TopKCand machinery and shares the direction of B2/B4).++**Mechanism (one):** admit constant-score keyword prefix/wildcard as columnar+filter leaves, and complete the ES constant-score rewrite for single-token mbp.+- parser.rs:3539-3545: single-token match_bool_prefix →+ `Prefix{constant_score:true}` (mirror of 4c69c05; do NOT touch the multi-token+ trailing-prefix clause at :3572 — separate cell, alters scored bool sums).+- `ScoredFilterLeaf::KeywordPrefix{field,prefix}` (index.rs:20034) admitted in+ `scored_fast_plan` (:20207) strictly on `constant_score==true` + kw field+ (internal lowerings pass false and MUST stay on BM25/FTS). Per segment resolve+ prefix → contiguous ord range [lo,hi) via `partition_point` on the sorted+ KeywordColumn terms (doc_values.rs:343-381); Wildcard same shape with an ord+ bitmap via `term_matches_wildcard` over the (few) unique terms. Row walk ++ existing TopKCand (score DESC, seq ASC), null_bitmap/ghost handling mirrored+ from the KeywordTerm leaf. Score = f32(boost or 1.0), no BM25.++**Files:** engine/crates/xerj-query/src/parser.rs:3539;+engine/crates/xerj-engine/src/index.rs:20034, 20207, ~10200, ~10284, 20165;+engine/crates/xerj-storage/src/doc_values.rs:343.++**Fair validation:** (a) xerj==ES byte-identical: prefix claude- → 0..9 @1.0,+84731==84731; boost variant max_score 2.5 bit-exact; wildcard claude-* same;+mbp runbook → max_score 1.0, ids [0,12,21,23,34,39,41,46,51,54], totals+25259==25259. (b) fast==brute: gate off the plan → identical hit SET + scores ++total (post-B1, order identical too). (c) negative controls: query_string+`model:claude-*` and multi-token mbp keep BM25 scores unchanged. (d) gate+1360/0/3 (310_match_bool_prefix.yml has no single-token scoring case; YAML+prefix suites run memtable-resident → brute path unchanged). (e) Δms: prefix+~0.4-1.2ms (Δ −2.2), wildcard ~0.5-1.0ms (Δ −2.9), mbp ~1.0-1.3ms — all+parity-to-WIN.++**Effort:** M. **Risk:** low-medium — the `constant_score:true` discriminator+must stay strict; empty ord range yields 0 hits not a bail-mismatch.++---++### Batch 4 — T2c+T3: fuzzy similarity boost + columnar keyword-fuzzy+**Targets:** fuzzy(model) (2.03 vs 1.12ms, DIFF:max_score+top → MATCH, parity/WIN).++**Mechanism (one):** ES-exact fuzzy scoring, both paths.+- `ScoredLeafKind::KeywordFuzzy` in scoring_leaf (index.rs:20260): expand against+ the segment KeywordColumn dictionary with the SAME Damerau-Levenshtein/case+ predicate as `term_matches_fuzzy` (search.rs:1084; memtable-vs-segment must+ agree — index.rs:19509-19542); per expansion sim = `1 - dist/min(cpLen)`;+ score via `bm25_keyword_term_score(df_blend=max df, total, boost*sim)`+ (ES BlendedTermQuery), sum per row across matching expansions; TopKCand gives+ the seq tie-break. Cap expansions at 50 (ES max_expansions) → bail to brute.+- Brute parity: `execute_fuzzy` (search.rs:798) passes the same per-term+ `boost*sim` into `TermQuery::boosted` so fast==brute holds on scores.+- AUTO fuzziness resolved via the existing parser path.++**Files:** engine/crates/xerj-engine/src/index.rs:20260, 20117, 10227, 10074,+20100, 19504; engine/crates/xerj-fts/src/search.rs:798, 840, 1084.++**Fair validation:** (a) xerj==ES: cell → max_score 1.2020086, ids+['0','2','3','4','5','8','10','19','24','25'] byte-identical; plus one+multi-expansion fuzziness:2 case (exercises df-blend+sum). (b) fast==brute:+columnar gate off → identical score multiset + total (bit-equal f32).+(c) gate 1360/0/3 — grep YAML for fuzzy _score assertions (change moves TOWARD+ES, expected green). (d) Δms: ~2.0 → ~1.1ms vs ES 1.1-1.7 → parity-to-WIN.++**Effort:** M. **Risk:** low-medium — expansion-predicate semantics must be+byte-identical across columnar/FTS/doc_matches_query or results diverge under+writes.++---++### Batch 5 — T4: track_total_hits early termination in scored_columnar+**Targets:** bool must+filter+should+must_not (2.15 vs 1.58ms LOSS → WIN).++**Mechanism (one):** bounded walk break. Compute static f64 `plan_max` over the+ClauseEval tree from clause_scores (Bool = Σmust + Σall-should; DisMax =+max + tie*Σrest; msm only shrinks the match set); in the Phase-2 row loops,+break when heap full AND total ≥ limit+1 (strict, so the renderer still emits+(limit, gte) with zero plumbing) AND heap-worst score ≥ plan_max (one-ulp+headroom; over-estimate only delays the break) AND remaining segments' min seq >+heap-worst seq. `TrackTotalHits::True` disables the break. Do NOT fire for+Pinned. Pass tth from the call site (index.rs:6215-6228).++**Files:** engine/crates/xerj-engine/src/index.rs:10348, 10238-10257, 6215-6228,+19932-20010 (mirror eval → max_bound), 20190-20205, 7651-7668.++**Fair validation:** (a) fast==brute across the variant sweep (tth+default/true/false/100, size 0/1/10/from, rare must, msm, dis_max wrapper,+filter-only, constant_score). (b) xerj==ES: cell top-10+0,3,4,5,10,19,24,32,33,41 @1.2947059, total {10000,gte}; at tth=true+{37369,eq}. (c) gate 1360/0/3 (break can't fire below 10k docs). (d) Δms:+~1.9 → ~1.1-1.2ms vs ES ~1.4-1.6 → WIN.++**Effort:** M. **Risk:** low — an UNDER-estimated plan_max silently truncates+top-k; mitigated by assume-all-match f64 bound + ulp headroom + the fast==brute+sweep + the min-seq gate for multi-segment.++---++### Batch 6 — T5a: page-cap materialization (kill the 256-doc floor)+**Targets:** term(cache_hit) (1.3 vs 0.66ms → ~0.7-0.9ms); shaves ~0.5ms off+every plain size:10 cell that hydrates through the floor.++**Mechanism (one):** `page_cap = from+size` instead of `(from+size+100).max(256)`+on the safe path only: sort is _score/_doc, `!deletes_present`, empty memtable+(or no id overlap), no rescore/collapse/highlight/post_filter — every other path+keeps the 256 floor bit-identical (ghost/superseded/dedup `continue`s under-fill+tight pages). Shrink `fts_cap` (index.rs:6452) and the hydration-loop break+(index.rs:6647) to match. Optional same-walk lever: hoist `fields.get(field)`+out of scan_term + norms ascending cursor / uniform-norm constant score — land+only if f32 score equality holds exactly, else drop.++**Files:** engine/crates/xerj-engine/src/index.rs:5196, 6452, 6647;+engine/crates/xerj-fts/src/search.rs:518; engine/crates/xerj-fts/src/index.rs:1269.++**Fair validation:** (a) fast==brute: byte-compare full _search JSON capped vs+256-floor (env-flag forced) across size {1,10,50,100} x from {0,150,400,900} x+_source variants x default/_doc sort, on the clean corpus AND an+overwrites+deletes index (must fall back to the floor there). (b) xerj==ES:+term(cache_hit true/false), term(tenant), rare term stay exact MATCH.+(c) gate 1360/0/3, esp. pagination + scroll suites. (d) Δms: 1.4-1.7 →+0.9-1.1ms (cap alone), 0.7-0.9 with the norms cursor; size:0 stays 0.40ms.+Residual ~0.2-0.4ms is ES's block-max WAND floor — accept (see out-of-scope).++**Effort:** S. **Risk:** page under-fill — fully gated; do NOT touch the+sort:['_doc'] _id-string defect in the same change.++---++### Batch 7 — T5b: prefiltered-hydrate waste (sorted-Vec cache + deep-from skip-parse)+**Targets:** range(@timestamp) latency (2.12 vs 0.93 → ≤1.5ms), deep from+size+latency (from:500 ~1.9 → ~1.3ms; from:5000 12.5 → ~2ms), terms(model) residual.++**Mechanism (one):** stop re-doing per-query work the prefilter/scan already did.+- `range_prefilter_cache` / `build_term_prefilter_cached` (index.rs:538, 10606,+ 10764): cache positions as a pre-sorted `Arc<Vec<u32>>` (built+ position-ascending during the existing ords walk — no HashSet, no per-query+ collect+sort); `hydrate_prefiltered_unsorted` (index.rs:11493) iterates the+ sorted slice with its existing early-break; membership checks in+ `scan_stored_section_into` use binary search or a bitvec twin. Keep the+ bool-prefilter intersection working (sorted-vec intersect or retain HashSet+ there) — scope contained to term/terms/range leaves + hydrate.+- Deep-from: thread `from` into `scan_stored_section_into` (index.rs:11810);+ under the F1 gate (match_all, count_authoritative, no peeled scorer,+ `deletes_present==false`) brace-scan-skip the first `from` positions and+ simd_json-parse only [from, from+size); full parse on any gate failure.++**Files:** engine/crates/xerj-engine/src/index.rs:538, 10606, 10733-10785,+11493-11495, 11810-11894, 5196.++**Fair validation:** (a) xerj==ES (post-B1 corpus): range top-10 == 0..9,+totals 100000==100000 tracked; from:500 → 500..549; from:0 and from:5000+windows byte-identical. (b) fast==brute: prefiltered vs forced full-scan+byte-identical; skip-parse page == full-parse page byte-identical, and on a+deletes-bearing index the gate must fall back (verified by identical output).+(c) gate 1360/0/3. (d) Δms: range 2.12 → ~1.0-1.5 (DIFF already fixed in B1;+residual is 256-doc parse slack, further shrunk by B6); from:500 Δ→~0;+from:5000 linear term gone.++**Effort:** M. **Risk:** low-medium — membership semantics must stay identical+or `count_authoritative` totals regress; skip-parse strictly gated.++---++### Batch 0 (parallel, no build) — T6: refresh stale harness records+**Targets:** agg:composite. after_key parity shipped in d49410c and is+live-verified byte-identical today (including mid-stream `after` pagination and+the exhausted final page omitting after_key); the scorecard already records+MATCH (FULL_MATRIX_SCORECARD_2026-07-10_CACHEHARDENED.md:64). Re-run the fair+probe, delete any pre-d49410c cached expected-response snapshot. Zero engine code.++---++## Out of scope (honest)++- **dis_max (NOISE_ES_FLOOR):** Δ0.25ms inside the ±0.30ms TIE band; even a+ perfect LUT row-walk leaves it TIE. Do not spend a build. Already MATCH on+ correctness (bit-exact).+- **exists(cost_usd) latency:** Δ0.14ms, both engines at the HTTP+render floor.+ Correctness half is flipped by Batch 1.+- **agg:composite:** stale diff, harness refresh only (Batch 0).+- **ES block-max WAND / impact-metadata early termination (NEEDS_INFRA):** the+ last ~0.2-0.4ms on term(cache_hit)-class cells and the exact-count-under-tth+ enumeration floor need maxscore/impact postings metadata — real infra, not a+ batch here.+- **`sort:["_doc"]` renders lexicographic `_id` order** (parser.rs:2612 /+ sort.rs:69 / index.rs:18558): latent off-scorecard bug, matches neither ES nor+ XERJ's own stored order. Separate ticket — do NOT bundle with any batch above+ (YAML tests may pin orderings).+- **size:0 / _count brute cliff** (bool body ~107ms, dis_max size:0 ~162ms vs ES+ <1ms — `scored_fast_plan` bails on size==0 at index.rs:20216): NOT one of these+ cells, but the largest single ratio found during diagnosis. File as its own+ high-ROI follow-up batch (extend the columnar scorer to count+max_score-only).+- **Multi-segment (unsettled) exactness pre-merge:** after Batch 1, tie order is+ exact for settled/force-merged states (the benchmark state); interleaved+ per-shard flush segments are only approximately insertion-ordered until merged.+ Per-segment admission caps (exists diagnosis step 3) are a separate follow-up+ commit if live-write benchmarks ever pin it.++## Expected scorecard movement++| Batch | Cells flipped | Kind |+|-------|--------------|------|+| B1 | wildcard, prefix, range, terms, exists, deep-from (+match_all) | 6+ DIFF:top→MATCH |+| B2 | multi_match | DIFF→MATCH + LOSS→WIN |+| B3 | prefix, wildcard, mbp | 3 LOSS→parity/WIN + mbp DIFF→MATCH |+| B4 | fuzzy | DIFF→MATCH + LOSS→parity/WIN |+| B5 | bool m+f+s+mn | LOSS→WIN |+| B6 | term(cache_hit) | LOSS→near-parity (+~0.5ms off all size:10 cells) |+| B7 | range, deep-from | 2 LOSS→parity |+| B0 | composite | stale DIFF→MATCH (harness) |diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 9df15f771..aaca3d319 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -4862,9 +4862,13 @@ pub async fn search(// declared at create time (or after close/reopen picks up the current// mapping). Mirror that behaviour using the `__xy_index_sort_*` hints// stored in settings at create / reopen time.+ //+ // For the implicit auto-@timestamp hint this is LEAF ordering, not a+ // per-doc sort — see `implicit_leaf_ts` below.+ let mut implicit_leaf_ts: Option<String> = None;if index_names.len() == 1 {let idx_name = index_names[0];- let (sort_field, sort_order, explicit) = state+ let (sort_field, sort_order, explicit, reopen_derived) = state.engine.index_settings.get(idx_name)@@ -4883,9 +4887,13 @@ pub async fn search(.get("__xy_index_sort_explicit").and_then(Value::as_bool).unwrap_or(false);- (f, o, ex)+ let ro = s+ .get("__xy_index_sort_reopen")+ .and_then(Value::as_bool)+ .unwrap_or(false);+ (f, o, ex, ro)})- .unwrap_or((None, "desc".to_string(), false));+ .unwrap_or((None, "desc".to_string(), false, false));if let Some(f) = sort_field {// Apply the index sort ONLY for an EXPLICITLY declared// `index.sort` (settings), when no sort was requested or the@@ -4906,15 +4914,24 @@ pub async fn search(// 1.5 M docs (~2-7 s each, `walked=1500002 admitted=250252// allhits_cap=false`) while the same query with an explicit// `sort: _doc` took 45 ms.- // For the implicit auto-@timestamp hint, only match_all-shaped- // queries get the injected sort: that is the query shape the ES- // conformance contract observes (search/380_sort_segments_on_- // timestamp.yml), and the engine serves it via the bounded- // sorted-DV candidates path (O(from+size) per segment).- // Filtered queries (term/range/bool) skip the injection — in- // real ES the physical index sort doesn't impose a query-time- // global sort on them either, and routing them through the- // top-N heap costs a full O(total-docs) walk per request.+ // For the implicit auto-@timestamp hint, real ES (7.16++ // "sort segments on timestamp") orders the LEAVES — segments+ // are read newest-max-@timestamp-first while docs inside each+ // segment keep arrival order, scores stay 1.0 and max_score+ // stays 1.0. The old implementation approximated that with an+ // injected per-doc `@timestamp desc` sort, which diverges from+ // real ES on any settled index: page windows are cut by+ // timestamp value (from:500 returned timestamp-ranked ids, ES+ // returns arrival ids 500..549), equal timestamps re-ordered+ // lexicographically by _id, and the sort suppressed max_score+ // (null vs ES 1.0) — all three live-verified against ES 8.13.4.+ // So for the implicit hint pass a leaf-order flag to the engine+ // instead (`SearchRequest::leaf_ts_field`), only for the+ // match_all-shaped sortless requests the old injection covered+ // (the shape search/380_sort_segments_on_timestamp.yml+ // observes). Filtered queries (term/range/bool) are untouched+ // — real ES's segment ordering doesn't impose a query-time+ // global sort on them either.let implicit_ok = body.query.is_none()|| body.query@@ -4922,11 +4939,26 @@ pub async fn search(.and_then(|q| q.as_object()).map(|o| o.len() == 1 && o.contains_key("match_all")).unwrap_or(false);- let apply = (explicit || implicit_ok)- && (body.sort.is_none()- || (explicit && body.sort.as_ref().map(is_lone_doc_sort).unwrap_or(false)));- if apply {- body.sort = Some(json!([{ f: { "order": sort_order } }]));+ if explicit {+ // Explicitly declared `index.sort` (settings): keep the+ // injected per-doc sort — ES returns docs in index-sort+ // order for sortless and `sort: _doc` requests there.+ let apply = body.sort.is_none()+ || body.sort.as_ref().map(is_lone_doc_sort).unwrap_or(false);+ if apply {+ body.sort = Some(json!([{ f: { "order": sort_order } }]));+ }+ } else if implicit_ok && body.sort.is_none() {+ if reopen_derived {+ // Reopen-derived hint: the close-time full-drain flush+ // already consolidated the per-refresh leaves ES would+ // have kept, so settled leaf ordering can't reproduce+ // the reopen contract — keep the historical per-doc+ // sort for these indices (see the `_open` handler).+ body.sort = Some(json!([{ f: { "order": sort_order } }]));+ } else {+ implicit_leaf_ts = Some(f);+ }}}}@@ -6118,7 +6150,13 @@ pub async fn search(.collect();let search_req = match build_search_request(&body, aggs_value) {- Ok(r) => r,+ Ok(mut r) => {+ // ES 7.16 leaf ordering for the implicit auto-@timestamp hint+ // (see the injection block above) — internal field, never on+ // the wire.+ r.leaf_ts_field = implicit_leaf_ts;+ r+ }Err(e) => return ApiError::new(e).into_response(),};@@ -16956,6 +16994,16 @@ pub async fn open_index(};merged.insert("__xy_index_sort_field".to_string(), json!("@timestamp"));merged.insert("__xy_index_sort_order".to_string(), json!("desc"));+ // Reopen-derived hint (dynamic @timestamp picked up at+ // close/open): `_close` drains the whole memtable into ONE+ // consolidated segment, so the per-refresh leaf structure ES+ // preserves is already gone by the first post-reopen search.+ // Mark the provenance so the search path keeps the+ // historical per-doc `@timestamp desc` sort for these+ // indices (the 380_sort_segments_on_timestamp.yml reopen+ // contract) instead of the settled leaf-order semantics+ // reserved for create-time-mapped indices.+ merged.insert("__xy_index_sort_reopen".to_string(), json!(true));state.engine.index_settingsdiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 17a5bdfe8..43340ed34 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -2795,6 +2795,24 @@ impl Index {// all stored docs even if a doc fails the FTS/DV re-parse.let mut ids_pairs: Vec<(u64, String)> = Vec::new();+ // ES tie-order parity (LOSS_BATTLE_PLAN B1): collect every+ // surviving doc as (seq_no, _id, raw JSON) FIRST, then sort+ // globally by _seq_no and derive ALL four merge outputs+ // (stored bytes / .ids side-car / FTS input / DV input) from+ // that ONE sorted stream. Ingest hash-routes docs across 16+ // memtable shards and each shard flushes its own segment, so+ // input-order concatenation scrambled physical row order vs+ // insertion (_seq_no) order — ES tie-breaks equal scores by+ // internal doc id == arrival order, so every constant-score+ // top-k diverged from ES. With physical == seq order the+ // existing doc-position tie-breaks ARE the ES tie-break; no+ // comparator changes anywhere. Memory: the owned raw copy+ // replaces the same bytes that previously went straight into+ // `merged_json_buf` (drained into it below), so the peak+ // stays ~1× stored size + the fts_input Values, unchanged+ // from the M5.22 profile.+ let mut survivors: Vec<(u64, String, String)> = Vec::new();+for meta in &metas_for_task {min_seq = min_seq.min(meta.min_seq_no);max_seq = max_seq.max(meta.max_seq_no);@@ -2913,34 +2931,53 @@ impl Index {}}- // Survives — append raw bytes to the merged buffer.- if !first_doc {- merged_json_buf.push(b',');- }- first_doc = false;- merged_json_buf.extend_from_slice(raw_str.as_bytes());- live_doc_count += 1;- ids_pairs.push((id_seq.seq_no, id_seq.id.to_string()));+ // Survives — hold (seq, id, raw) for the global+ // _seq_no-ordered build below.+ survivors.push((+ id_seq.seq_no,+ id_seq.id.to_string(),+ raw_str.to_string(),+ ));+ }+ // raw_docs + stored_bytes drop here — segment RAM reclaimed.+ }- // Per-doc Value parse for FTS / DV builders.- let doc_value: Value = match serde_json::from_str(raw_str) {- Ok(v) => v,- Err(_) => continue,- };- let id_str = id_seq.id.to_string();- let source = doc_value.get("_source").cloned().unwrap_or(Value::Null);- let mut fields: HashMap<String, String> = HashMap::new();- if let Some(obj) = source.as_object() {- for (key, val) in obj {- let text = extract_field_text(val);- if !text.is_empty() {- fields.insert(key.clone(), text);- }+ // Global insertion-order (_seq_no) sort — see the B1 note+ // above. seq_nos are unique per surviving doc (stale /+ // duplicate copies were filtered by the version map).+ survivors.sort_unstable_by_key(|(seq, _, _)| *seq);++ // Single sorted stream → all four outputs. `into_iter`+ // frees each raw String right after its bytes are copied+ // into `merged_json_buf`, keeping peak memory ~1× stored.+ for (seq_no, id_str, raw_str) in survivors {+ if !first_doc {+ merged_json_buf.push(b',');+ }+ first_doc = false;+ merged_json_buf.extend_from_slice(raw_str.as_bytes());+ live_doc_count += 1;+ // Pushed BEFORE the per-doc Value parse that can+ // `continue` — the .ids side-car must cover every+ // stored doc (alignment invariant, see above).+ ids_pairs.push((seq_no, id_str.clone()));++ // Per-doc Value parse for FTS / DV builders.+ let doc_value: Value = match serde_json::from_str(&raw_str) {+ Ok(v) => v,+ Err(_) => continue,+ };+ let source = doc_value.get("_source").cloned().unwrap_or(Value::Null);+ let mut fields: HashMap<String, String> = HashMap::new();+ if let Some(obj) = source.as_object() {+ for (key, val) in obj {+ let text = extract_field_text(val);+ if !text.is_empty() {+ fields.insert(key.clone(), text);}}- fts_input.push((id_str, fields, source));}- // raw_docs + stored_bytes drop here — segment RAM reclaimed.+ fts_input.push((id_str, fields, source));}if live_doc_count == 0 {@@ -4063,6 +4100,7 @@ impl Index {script_fields: None,fields: Vec::new(),profile: false,+ leaf_ts_field: None,};// Box::pin to break the type-recursion (search_inner ↔// run_hybrid both async fn).@@ -4998,6 +5036,50 @@ impl Index {)));}+ // Implicit @timestamp "sort segments on timestamp" handling (see+ // `SearchRequest::leaf_ts_field`). Two regimes:+ //+ // * SETTLED (empty memtable, ≤1 segment — the read-mostly steady+ // state): apply NO ordering at all. A single seq-ordered segment+ // (the B1 merge invariant) already reads back in arrival order,+ // which is exactly what real ES 8.13.4 returns on the same+ // single-segment index — leaf ordering over one leaf is a no-op+ // there, hits keep `_score` 1.0 and `max_score` 1.0. The old+ // injected per-doc sort diverged from ES here on all three axes+ // (page windows, equal-ts lex re-order, max_score null) —+ // live-verified.+ //+ // * UNSETTLED (buffered memtable docs, or multiple segments):+ // xerj's visibility units don't line up with ES's+ // refresh-generation leaves (a `?refresh=true` doc stays+ // memtable-resident here but is its own newest-first leaf in ES),+ // and the final hit ordering ties break by global seq — so+ // segment-level iteration order can't reproduce ES's leaf order.+ // Keep the historical injected per-doc `@timestamp desc` sort for+ // this window — the behaviour the ES conformance contract+ // (search/380_sort_segments_on_timestamp.yml) validates.+ let leaf_fallback_request: Option<SearchRequest> = match request.leaf_ts_field.as_deref() {+ Some(ts_field)+ if matches!(request.query, QueryNode::MatchAll)+ && request.sort.is_empty()+ && (self.memtable.doc_count() > 0+ || self.store.snapshot().segments.len() > 1) =>+ {+ let mut r = request.clone();+ r.sort = vec![xerj_query::sort::SortField {+ field: ts_field.to_string(),+ order: xerj_query::sort::SortOrder::Desc,+ mode: xerj_query::sort::SortMode::default(),+ missing: xerj_query::sort::SortMissing::default(),+ format: None,+ }];+ r.leaf_ts_field = None;+ Some(r)+ }+ _ => None,+ };+ let request: &SearchRequest = leaf_fallback_request.as_ref().unwrap_or(request);+// ── Cooperative deadline ──────────────────────────────────────────// The `tokio::time::timeout` wrapper in `search()` is DEAD CODE on// the multi-thread runtime: the search body runs insidediff --git a/engine/crates/xerj-query/src/ast.rs b/engine/crates/xerj-query/src/ast.rsindex 21d7d269f..9c24b0e92 100644--- a/engine/crates/xerj-query/src/ast.rs+++ b/engine/crates/xerj-query/src/ast.rs@@ -972,6 +972,18 @@ pub struct SearchRequest {/// dropped before pagination, aggregations, and total counting.#[serde(default, skip_serializing_if = "Option::is_none")]pub min_score: Option<f64>,++ /// INTERNAL (never on the wire — `serde(skip)`): iterate segments in+ /// per-segment-max-of-this-date-field DESC order for sortless+ /// match_all-shaped requests. This is ES 7.16's "sort segments on+ /// timestamp" LEAF ordering for indices that map `@timestamp` at+ /// create time: segments are read newest-first while docs inside each+ /// segment keep arrival order and hits keep `_score` 1.0 / max_score+ /// 1.0. Set by the ES-compat layer; replaces the old injected per-doc+ /// `@timestamp desc` sort, which diverged from real ES on settled+ /// indices (wrong page windows + `max_score: null`).+ #[serde(skip)]+ pub leaf_ts_field: Option<String>,}fn default_query() -> QueryNode {@@ -1002,6 +1014,7 @@ impl Default for SearchRequest {collapse: None,rescore: Vec::new(),min_score: None,+ leaf_ts_field: None,}}}diff --git a/engine/crates/xerj-query/src/parser.rs b/engine/crates/xerj-query/src/parser.rsindex 40e8aa3db..6f4f83466 100644--- a/engine/crates/xerj-query/src/parser.rs+++ b/engine/crates/xerj-query/src/parser.rs@@ -238,6 +238,7 @@ pub fn parse_request(body: &Value) -> Result<SearchRequest> {collapse: None,rescore: Vec::new(),min_score: None,+ leaf_ts_field: None,})}diff --git a/engine/crates/xerj-storage/src/merge.rs b/engine/crates/xerj-storage/src/merge.rsindex ce4ed7d9d..de8d8535d 100644--- a/engine/crates/xerj-storage/src/merge.rs+++ b/engine/crates/xerj-storage/src/merge.rs@@ -302,6 +302,15 @@ impl MergeExecutor {}}+ // Emit the merged segment in global insertion (_seq_no) order. Inputs+ // are per-memtable-shard segments (hash-partitioned by _id), so plain+ // concatenation scrambles physical row order vs arrival order; ES+ // tie-breaks equal scores by internal doc id == arrival order, and the+ // engine-side merge (index.rs merge_pass_locked) applies the same sort.+ merged_docs.sort_unstable_by_key(|doc| {+ doc.get("_seq_no").and_then(|v| v.as_u64()).unwrap_or(0)+ });+let live_doc_count = merged_docs.len() as u64;// Write merged segment — inherit schema version from the snapshot if available