Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge pull request #420 from xerj-org/fix/issue-401

xerj-orgf4c33ff87c56
diff --cc CHANGELOG.md
index ed4d90fa8,1ad8c4c40..fa7f3a253
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,93 -7,68 +7,153 @@@ and this project adheres to [Semantic V
## [Unreleased]
+### Documentation
+
+- **The 10,000-document scroll snapshot cap is now published, and pinned to the
+ constant that enforces it**
+ ([#370](https://github.com/xerj-org/xerj/issues/370)). XERJ's `_search?scroll=`
+ is a bounded up-front snapshot, not a segment-walking cursor, so a query whose
+ exact result set exceeds `SCROLL_SNAPSHOT_MAX_HITS` is refused with a 400
+ rather than paged and silently truncated (#198). The refusal is the right
+ behaviour and its message already names `search_after` as the way past it —
+ but the ceiling appeared in no user-facing document, and scroll is precisely
+ what the ES ecosystem reaches for to read a *whole* index
+ (`elasticsearch-py`'s `helpers.scan()`, reindex internals, export and backup
+ tooling), where the result set is normally larger than the cap. "Scroll is
+ supported, with a 10,000-document snapshot cap; use `search_after` beyond
+ that" is a different compatibility claim from "scroll is supported", and only
+ the first one is true.
+
+ The cap, the 400 it produces and the `search_after` alternative are now stated
+ on `landing/docs/api-es-compat.html` (which also gains the three scroll
+ endpoints it never listed), in the "what needs rewriting" list on
+ `landing/docs/migration-from-es.html`, and in the honesty/capability-boundary
+ section of `landing/llms-full.txt`. Every published figure sits in a
+ `<!-- generated:scroll-snapshot-cap -->` region checked against
+ `xerj_api::es_compat::SCROLL_SNAPSHOT_MAX_HITS` by
+ `xerj-api/tests/scroll_snapshot_cap_is_documented.rs`, so changing the
+ constant now fails the build until the pages follow — the mechanism already
+ used for the capability counts in `docs_capability_lists.rs`. The same test
+ drives the real ES-compat router over a corpus one document past the cap and
+ reads the enforced number back out of the 400 the server produced, so the
+ published number is checked against the server's behaviour rather than
+ against a second hand-written copy of it.
+
+ **The published escape hatch is now executed, not just spell-checked.** A
+ pinned number is worth nothing if the recipe past it is wrong, and the first
+ draft of this page's `search_after` transcript was: it paged with
+ `search_after: ["999"]` against 11,450 numeric-looking `_id`s. `_id` is a
+ keyword and sorts lexicographically, so `"1000" < "999"` and page 1 of 1,000
+ actually ends at `_id` `"10897"`. Followed exactly as printed, including the
+ page's own stop rule ("until a page comes back empty"), that walk collected
+ 1,010 of 11,450 documents — 10,440 missing, every response a 200 — which is
+ the silently-truncated export that #198 and this very page exist to rule out.
+ The transcript now publishes the cursor the response actually returns, shows
+ the hit it comes from, and states the lexicographic ordering that produced the
+ mistake. `published_search_after_recipe_walks_the_whole_corpus` parses the
+ transcript out of the page, seeds the corpus the transcript itself names,
+ replays the printed *paging* request bodies (the two carrying `sort`) against the real router, and
+ asserts the walk collects every seeded `_id` over the page count the
+ transcript claims — so a wrong cursor, an early stop, a repeated page or a
+ drifted claim all fail the build.
+
+ Three further scroll claims elsewhere in the tree were corrected to match:
+ `landing/agent-search/index.html` said scroll *was* a deep-pagination cursor;
+ `landing/llms-full.txt` §6 and `README.md` carried the bare "scroll is
+ supported" form; and the ES→XERJ migration recipe in
+ `docs/recipes/production-deployment.md` told readers they could point a
+ whole-index scroll copy at a XERJ source with no mention of the ceiling.
+
+ One boundary is documented rather than fixed here: the cap is applied to the
+ request's summed total on `POST /{index}/_search?scroll=` (`es_compat.rs:14211`)
+ but *per index* on the `POST /{index}/_search_scroll` alias
+ (`es_compat.rs:19726`), so a multi-index scroll on that route can snapshot the
+ ceiling from each index. That direction is permissive, not lossy. The pages now
+ say so and point at
+ [#405](https://github.com/xerj-org/xerj/issues/405), where the engine
+ inconsistency is tracked.
+
+ Verified by `cargo test -p xerj-api --test
+ scroll_snapshot_cap_is_documented` — 3 passed in the `ci-test` profile
+ (8.67s, overflow-checks off) and 3 passed in `dev` (12.40s, overflow-checks
+ on). It drives `build_es_compat_router` over tempdir engines: a
+ 10,001-document index (`SCROLL_SNAPSHOT_MAX_HITS + 1`)
+ refuses the scroll open with `400 illegal_argument_exception` naming `[10001]`
+ and `[10000]`; a 128-document control scroll returns a `_scroll_id`; the
+ printed first page returns 1,000 hits ending at `_id` `"10897"` with
+ `sort: ["10897"]`; and the published walk from that cursor returns 11,450
+ distinct `_id`s over 12 pages with none missing and none repeated. Reverting
+ only the transcript hunk to the `["999"]` form fails the same test with
+ `left: Array [String("999")] / right: Array [String("10897")]`, and with the
+ cursor assertion inverted so the walk still runs, with "collected 1010 of
+ 11450 documents over 2 pages, silently missing 10440". The `_search_scroll`
+ divergence above was measured on two 6,000-document indices in one engine:
+ `POST /sa,sb/_search?scroll=1m` → 400 naming `[12000]`/`[10000]`;
+ `POST /sa,sb/_search_scroll?scroll=1m` → 200 with a `_scroll_id` and
+ `hits.total {value: 12000, relation: "eq"}`.
+
### Fixed
+ - **`sort` on an ES meta-field other than `_score` / `_doc` / `_id` resolved to
+ `null` on every hit** ([#401](https://github.com/xerj-org/xerj/issues/401)).
+ `compute_sort_values` special-cased exactly those three keys and sent
+ everything else to `get_field_value(source, field)` — a lookup *inside*
+ `_source`. `_seq_no`, `_version`, `_primary_term` and `_index` are engine
+ metadata and are never literal `_source` keys, so the lookup missed on every
+ document and the whole result set tied on `[null]`. The damage lands on
+ keyset pagination: `search_after: [null]` is never strictly greater than the
+ next hit's `[null]`, so `sort: [{"_seq_no":"asc"}]` + `search_after` — the
+ consistent full-scan pattern used for migrations — either re-reads page one
+ forever or stops after it, with no error either way. Measured on a 30-document
+ index before the fix: 4 ids collected out of 30, page two empty.
+
+ These fields now resolve through the same version-map accessors that populate
+ the response meta-fields (`lookup_seq_no` / `lookup_version`), so a hit's
+ `sort` value and its `_seq_no` / `_version` agree. `_primary_term` is the
+ constant `1` the hit meta-field already emits and `_index` is the index name;
+ both are constant within one index, so they tie like any other constant sort
+ key and a client paginating on them still needs a tie-breaker. Sorting on
+ `_seq_no` under `index.disable_sequence_numbers` is unchanged — es_compat
+ rejects it at the API edge before the request reaches the engine.
+
+ Real sort values are only half of it: the memtable's bounded sort-candidate
+ path narrows the heap's input using a doc-values column keyed by the sort
+ field, which for a meta-field is derived from `_source` rather than from the
+ version map the heap now ranks on. With no such key present it classified
+ every buffered document as "missing the field" and returned an arbitrary
+ `materialisation_limit`-sized prefix. A correctly ranked page over a wrong
+ candidate set is still a silently wrong page, so that path — and the
+ segment-side sort shadow — now decline meta-fields and take the full walk.
+ The same gate fixes an independent instance of the identical defect on `_id`,
+ which already resolved to a real sort value: over a 2 000-document memtable
+ `sort: [{"_id":"desc"}]` returned
+ `["d1995","d1989","d1981","d1973","d1961"]` instead of
+ `["d1999","d1998","d1997","d1996","d1995"]`.
+
+ There is a third such prefilter, and it is the one the two gates above route
+ meta-sorts *onto*: the memtable scan's pre-clone rejection
+ (`memtable_primary_key_rejects`) skips a document whose primary sort key,
+ read from `_source`, already loses to the full heap. Once the heap ranks on
+ version-map metadata, a document carrying a `_source` key of the same name
+ is compared against the wrong frontier and dropped before the heap ever sees
+ it. Measured over ES-compat HTTP on 2 000 documents where only `d0000`
+ carried `"_seq_no": 10000000`: `{"size":5,"sort":[{"_seq_no":"asc"}]}`
+ returned `[d0001, d0002, d0003, d0004, d0005]`, omitting the true first
+ document, while `"size": 3000` returned it — a silent, size-dependent
+ missing document. That path now declines meta-fields too.
+
+ **Behaviour change:** `sort` on `_seq_no` / `_version` / `_primary_term` /
+ `_index` now ignores a `_source` key of the same name and always ranks on
+ engine metadata. Previously it ranked on the `_source` value where one
+ existed and on `null` everywhere else. This is observable because xerj —
+ unlike Elasticsearch, which rejects a metadata-named key inside a document
+ with a `document_parsing_exception` — accepts `{"_seq_no": 1}` in a document
+ body and echoes it back in `_source`. Such a key is now inert for sorting;
+ it is still stored, still returned in `_source`, and still usable for
+ ordinary queries. Rejecting metadata-named `_source` keys at the API edge is
+ deliberately **not** part of this change — xerj writes `_routing` into
+ `_source` itself, so the edge rule needs its own design.
+
- **`xerj autoindex` aborted a whole run when one file declared two or more SQL
tables** ([#360](https://github.com/xerj-org/xerj/issues/360)). One file can
feed several datasets — a SQL dump is one file and N tables — but the sealed