Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into pr2442

xerj-org2b56b6d96140
diff --cc CHANGELOG.md
index 54f93be6f,04026c4ca..b3c0eed2d
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -77,15 -117,109 +117,116 @@@ continues in #2427 and #1122, which sta
[#1170](https://github.com/xerj-org/xerj/issues/1170); the issue stays
open).
+ - **Release performance gate** for autoindex and search. It adds
+ `gen_corpus.py`, `autoindex_release_gate.sh` and `gate.py`, with per-phase
+ budgets, an Elasticsearch floor and an rc.99 comparison harness, and it
+ runs before a release is cut (#2432). The live-tree memory follow-up is
+ tracked in #2433, which stays open.
+
+ - `xerj init` prints the Kimi CLI MCP config path (`~/.kimi-code/mcp.json`),
+ and the README marks the Kimi CLI row as tested (#2386).
+
+ - Read-only post-deployment agent-discovery verifier for final GET/HEAD Link
+ headers, documentation targets and optional public-skill digest checks.
+ CI exercises local HTTP fixtures rather than contacting production (PR #2299).
+
+ ### Changed
+
+ - A `semantic_text` field with the built-in lexical embedder no longer
+ stores its embedding twice. The pooled and per-passage vectors were kept
+ in the stored `_source` as well as in the vector index; the columnar
+ stored encoder now drops them when they rebuild bit-exactly from the
+ text, and every reader (`_source`, `GET`, kNN, reindex, snapshot)
+ rebuilds them, so responses are unchanged. On a 6,000-document test the
+ segment file went from 20.1 MB to 1.8 MB and the index from 3.67x to
+ 1.63x its source size. The saving is on disk; the HNSW graph in memory is
+ unchanged. Sections under 128 documents keep their vectors.
+ `XERJ_STORE_SEMANTIC_VECTORS=1` turns this off; see the upgrade note
+ (part of [#1122](https://github.com/xerj-org/xerj/issues/1122) and
+ [#2433](https://github.com/xerj-org/xerj/issues/2433)).
+
### Fixed
+- `_reindex` follows aliases on both sides: a source alias is searched across
+ every backing index and each member's alias filter is applied, and a
+ destination alias writes to its single write index (an ambiguous alias is
+ refused with `illegal_argument_exception`). Previously only the first
+ backing index was read or written and filters were ignored, so the call
+ reported success over the wrong data.
+
+ - `xerj corpus add` accepts a local Windows path such as
+ `F:\Projects\Foo` (#2472). The repo name was the text after the last `/`,
+ so a `\`-separated path stayed whole and was refused as an unsafe repo
+ name; it now names the repo `Foo`, as `F:/Projects/Foo` already did.
+
+ - `xerj code` shows the part of a long definition that matched the query,
+ not its first `--full` characters (#2464). The matching text could sit
+ thousands of characters below the signature, so a hit named the right
+ function but showed none of the query terms. Such passages now end with
+ `... [function name: showing N of M chars from offset O, window centred on the match]`;
+ the `@ line N` label and citations are unchanged, and a definition whose
+ head is already the best window still ends `truncated at N chars`. Hub ask
+ passages (`xerj ask`) and hub search API hits change the same way (PR #2474).
+
+ - A segment merge's output-write step (doc-values build, stored encode, FTS
+ side-car build) is now charged to the merge budget before it starts, and a
+ background batch yields instead of starting it when it would take RSS to
+ the process limit (#2427). That step runs after the last yield checkpoint,
+ and neither admission nor the yield checks saw it: large pg-docs merges
+ (29k-166k docs) took a 3,072 MB search node to 4.4-4.9 GB about 5 s before
+ they completed. The step is estimated at 6x the merged stored JSON or 2x
+ the FTS build input, whichever is larger. An admitted step grows the
+ batch's merge reservation, so concurrent merge admission sees it.
+ `_forcemerge` batches are charged but never refused, and neither is a
+ small batch that the yield-streak rule (#2468) lets run to completion (PR #2470).
+
+ - The parent memory breaker no longer flaps, and background merges and flush
+ finalizes are bounded while it is engaged (#2427). A 3,000 MB autoindex run
+ engaged the breaker 20 times in 16 minutes, often releasing 100-500 ms
+ later, and RSS still reached 3,808 MB because merges, FST builds and flush
+ buffers kept allocating after writes were getting 429s. The breaker now
+ releases below 85% of the watermark (`XERJ_MEMORY_BREAKER_RELEASE_PCT`,
+ 50-99) and only after it has been engaged for 2 s
+ (`XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS`). RSS that stays flat inside that
+ band for 30 s still releases, so a live set no drain can shrink does not
+ block writes forever. The breaker never releases while RSS is at or above
+ the watermark: the 1 GB release gate (watermark 2,918 MB) released at
+ 5,121 MB through the old flat-footprint release (#1981), re-engaged 100 ms
+ later and was OOM-killed at 8.4 GB. A footprint that climbs while the
+ breaker is engaged is drained again within 2 s instead of waiting out the
+ drain backoff. While the breaker is engaged, searches run one at a time,
+ and at or above the process memory limit new searches get a 429
+ `circuit_breaking_exception`. Every segment merge reserves its estimated working
+ set (8x its inputs) against the watermark in one process-wide ledger.
+ Background merges wait while the breaker is engaged or when the
+ reservation does not fit, and run on the next merge check. One merge
+ always runs when nothing else is merging and the breaker is not engaged.
+ `_forcemerge` may run alone while engaged. While the breaker is engaged,
+ flush finalizes (segment, FST and doc-values builds) run one at a time.
+ A background merge batch that is already running stops once the breaker is
+ engaged and RSS reaches the process limit, keeps its inputs, and halves the
+ input cap of later batches until batches complete again. On the 1 GB gate
+ corpus one admitted batch (JSON segments that compress 10-20x) grew a
+ 3,072 MB `xerj autoindex` node to 8,122 MB in 18 s while writes were
+ getting 429s; plain `_bulk` into one index peaked at 4,967 MB (PR #2452).
+
+ - `xerj autoindex` no longer aborts a long run with `Resource temporarily
+ unavailable (os error 11)` when the machine refuses to start a thread. The
+ sync replay used to start a new thread for every operation (hundreds of
+ thousands on a large folder); it now starts at most the window width of
+ workers once per replay and reuses them. Starting a replay worker or an
+ indexing worker now retries with a short backoff when thread creation is
+ refused, and the replay finishes on the calling thread if no worker can be
+ started, so a refusal is no longer fatal (#2428) (PR #2437).
+ - `_bulk` now checks the memory limiter before it parses the request, not
+ only when each index's batch is written. A bulk that arrived while the memory
+ or memtable breaker was engaged still split and copied its whole body first.
+ Several concurrent bulks between two RSS samples were each admitted on the
+ same old sample, which could carry the node past its memory cap. Each
+ request now counts its body against live RSS plus every other in-flight
+ reservation until its parse ends, and is answered with one HTTP 429 per action (top-level 429,
+ `Retry-After: 1`) when that would reach the watermark or a breaker is
+ engaged, so clients back off and re-offer every record (part of #2427) (PR #2440).
- `xerj autoindex` no longer runs out of memory while planning a large folder.
It used to keep the sampled values of every file in memory at once before
grouping them into datasets, and kept them all until the end of the run;