AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge pull request #420 from xerj-org/fix/issue-401
xerj-orgf4c33ff87c56
diff --cc CHANGELOG.mdindex ed4d90fa8,1ad8c4c40..fa7f3a253--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,93 -7,68 +7,153 @@@ and this project adheres to [Semantic V## [Unreleased]+### Documentation++- **The 10,000-document scroll snapshot cap is now published, and pinned to the+ constant that enforces it**+ ([#370](https://github.com/xerj-org/xerj/issues/370)). XERJ's `_search?scroll=`+ is a bounded up-front snapshot, not a segment-walking cursor, so a query whose+ exact result set exceeds `SCROLL_SNAPSHOT_MAX_HITS` is refused with a 400+ rather than paged and silently truncated (#198). The refusal is the right+ behaviour and its message already names `search_after` as the way past it —+ but the ceiling appeared in no user-facing document, and scroll is precisely+ what the ES ecosystem reaches for to read a *whole* index+ (`elasticsearch-py`'s `helpers.scan()`, reindex internals, export and backup+ tooling), where the result set is normally larger than the cap. "Scroll is+ supported, with a 10,000-document snapshot cap; use `search_after` beyond+ that" is a different compatibility claim from "scroll is supported", and only+ the first one is true.++ The cap, the 400 it produces and the `search_after` alternative are now stated+ on `landing/docs/api-es-compat.html` (which also gains the three scroll+ endpoints it never listed), in the "what needs rewriting" list on+ `landing/docs/migration-from-es.html`, and in the honesty/capability-boundary+ section of `landing/llms-full.txt`. Every published figure sits in a+ `<!-- generated:scroll-snapshot-cap -->` region checked against+ `xerj_api::es_compat::SCROLL_SNAPSHOT_MAX_HITS` by+ `xerj-api/tests/scroll_snapshot_cap_is_documented.rs`, so changing the+ constant now fails the build until the pages follow — the mechanism already+ used for the capability counts in `docs_capability_lists.rs`. The same test+ drives the real ES-compat router over a corpus one document past the cap and+ reads the enforced number back out of the 400 the server produced, so the+ published number is checked against the server's behaviour rather than+ against a second hand-written copy of it.++ **The published escape hatch is now executed, not just spell-checked.** A+ pinned number is worth nothing if the recipe past it is wrong, and the first+ draft of this page's `search_after` transcript was: it paged with+ `search_after: ["999"]` against 11,450 numeric-looking `_id`s. `_id` is a+ keyword and sorts lexicographically, so `"1000" < "999"` and page 1 of 1,000+ actually ends at `_id` `"10897"`. Followed exactly as printed, including the+ page's own stop rule ("until a page comes back empty"), that walk collected+ 1,010 of 11,450 documents — 10,440 missing, every response a 200 — which is+ the silently-truncated export that #198 and this very page exist to rule out.+ The transcript now publishes the cursor the response actually returns, shows+ the hit it comes from, and states the lexicographic ordering that produced the+ mistake. `published_search_after_recipe_walks_the_whole_corpus` parses the+ transcript out of the page, seeds the corpus the transcript itself names,+ replays the printed *paging* request bodies (the two carrying `sort`) against the real router, and+ asserts the walk collects every seeded `_id` over the page count the+ transcript claims — so a wrong cursor, an early stop, a repeated page or a+ drifted claim all fail the build.++ Three further scroll claims elsewhere in the tree were corrected to match:+ `landing/agent-search/index.html` said scroll *was* a deep-pagination cursor;+ `landing/llms-full.txt` §6 and `README.md` carried the bare "scroll is+ supported" form; and the ES→XERJ migration recipe in+ `docs/recipes/production-deployment.md` told readers they could point a+ whole-index scroll copy at a XERJ source with no mention of the ceiling.++ One boundary is documented rather than fixed here: the cap is applied to the+ request's summed total on `POST /{index}/_search?scroll=` (`es_compat.rs:14211`)+ but *per index* on the `POST /{index}/_search_scroll` alias+ (`es_compat.rs:19726`), so a multi-index scroll on that route can snapshot the+ ceiling from each index. That direction is permissive, not lossy. The pages now+ say so and point at+ [#405](https://github.com/xerj-org/xerj/issues/405), where the engine+ inconsistency is tracked.++ Verified by `cargo test -p xerj-api --test+ scroll_snapshot_cap_is_documented` — 3 passed in the `ci-test` profile+ (8.67s, overflow-checks off) and 3 passed in `dev` (12.40s, overflow-checks+ on). It drives `build_es_compat_router` over tempdir engines: a+ 10,001-document index (`SCROLL_SNAPSHOT_MAX_HITS + 1`)+ refuses the scroll open with `400 illegal_argument_exception` naming `[10001]`+ and `[10000]`; a 128-document control scroll returns a `_scroll_id`; the+ printed first page returns 1,000 hits ending at `_id` `"10897"` with+ `sort: ["10897"]`; and the published walk from that cursor returns 11,450+ distinct `_id`s over 12 pages with none missing and none repeated. Reverting+ only the transcript hunk to the `["999"]` form fails the same test with+ `left: Array [String("999")] / right: Array [String("10897")]`, and with the+ cursor assertion inverted so the walk still runs, with "collected 1010 of+ 11450 documents over 2 pages, silently missing 10440". The `_search_scroll`+ divergence above was measured on two 6,000-document indices in one engine:+ `POST /sa,sb/_search?scroll=1m` → 400 naming `[12000]`/`[10000]`;+ `POST /sa,sb/_search_scroll?scroll=1m` → 200 with a `_scroll_id` and+ `hits.total {value: 12000, relation: "eq"}`.+### Fixed+ - **`sort` on an ES meta-field other than `_score` / `_doc` / `_id` resolved to+ `null` on every hit** ([#401](https://github.com/xerj-org/xerj/issues/401)).+ `compute_sort_values` special-cased exactly those three keys and sent+ everything else to `get_field_value(source, field)` — a lookup *inside*+ `_source`. `_seq_no`, `_version`, `_primary_term` and `_index` are engine+ metadata and are never literal `_source` keys, so the lookup missed on every+ document and the whole result set tied on `[null]`. The damage lands on+ keyset pagination: `search_after: [null]` is never strictly greater than the+ next hit's `[null]`, so `sort: [{"_seq_no":"asc"}]` + `search_after` — the+ consistent full-scan pattern used for migrations — either re-reads page one+ forever or stops after it, with no error either way. Measured on a 30-document+ index before the fix: 4 ids collected out of 30, page two empty.++ These fields now resolve through the same version-map accessors that populate+ the response meta-fields (`lookup_seq_no` / `lookup_version`), so a hit's+ `sort` value and its `_seq_no` / `_version` agree. `_primary_term` is the+ constant `1` the hit meta-field already emits and `_index` is the index name;+ both are constant within one index, so they tie like any other constant sort+ key and a client paginating on them still needs a tie-breaker. Sorting on+ `_seq_no` under `index.disable_sequence_numbers` is unchanged — es_compat+ rejects it at the API edge before the request reaches the engine.++ Real sort values are only half of it: the memtable's bounded sort-candidate+ path narrows the heap's input using a doc-values column keyed by the sort+ field, which for a meta-field is derived from `_source` rather than from the+ version map the heap now ranks on. With no such key present it classified+ every buffered document as "missing the field" and returned an arbitrary+ `materialisation_limit`-sized prefix. A correctly ranked page over a wrong+ candidate set is still a silently wrong page, so that path — and the+ segment-side sort shadow — now decline meta-fields and take the full walk.+ The same gate fixes an independent instance of the identical defect on `_id`,+ which already resolved to a real sort value: over a 2 000-document memtable+ `sort: [{"_id":"desc"}]` returned+ `["d1995","d1989","d1981","d1973","d1961"]` instead of+ `["d1999","d1998","d1997","d1996","d1995"]`.++ There is a third such prefilter, and it is the one the two gates above route+ meta-sorts *onto*: the memtable scan's pre-clone rejection+ (`memtable_primary_key_rejects`) skips a document whose primary sort key,+ read from `_source`, already loses to the full heap. Once the heap ranks on+ version-map metadata, a document carrying a `_source` key of the same name+ is compared against the wrong frontier and dropped before the heap ever sees+ it. Measured over ES-compat HTTP on 2 000 documents where only `d0000`+ carried `"_seq_no": 10000000`: `{"size":5,"sort":[{"_seq_no":"asc"}]}`+ returned `[d0001, d0002, d0003, d0004, d0005]`, omitting the true first+ document, while `"size": 3000` returned it — a silent, size-dependent+ missing document. That path now declines meta-fields too.++ **Behaviour change:** `sort` on `_seq_no` / `_version` / `_primary_term` /+ `_index` now ignores a `_source` key of the same name and always ranks on+ engine metadata. Previously it ranked on the `_source` value where one+ existed and on `null` everywhere else. This is observable because xerj —+ unlike Elasticsearch, which rejects a metadata-named key inside a document+ with a `document_parsing_exception` — accepts `{"_seq_no": 1}` in a document+ body and echoes it back in `_source`. Such a key is now inert for sorting;+ it is still stored, still returned in `_source`, and still usable for+ ordinary queries. Rejecting metadata-named `_source` keys at the API edge is+ deliberately **not** part of this change — xerj writes `_routing` into+ `_source` itself, so the edge rule needs its own design.+- **`xerj autoindex` aborted a whole run when one file declared two or more SQLtables** ([#360](https://github.com/xerj-org/xerj/issues/360)). One file canfeed several datasets — a SQL dump is one file and N tables — but the sealed