AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge remote-tracking branch 'origin/main' into pr2442
xerj-org2b56b6d96140
diff --cc CHANGELOG.mdindex 54f93be6f,04026c4ca..b3c0eed2d--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -77,15 -117,109 +117,116 @@@ continues in #2427 and #1122, which sta[#1170](https://github.com/xerj-org/xerj/issues/1170); the issue staysopen).+ - **Release performance gate** for autoindex and search. It adds+ `gen_corpus.py`, `autoindex_release_gate.sh` and `gate.py`, with per-phase+ budgets, an Elasticsearch floor and an rc.99 comparison harness, and it+ runs before a release is cut (#2432). The live-tree memory follow-up is+ tracked in #2433, which stays open.++ - `xerj init` prints the Kimi CLI MCP config path (`~/.kimi-code/mcp.json`),+ and the README marks the Kimi CLI row as tested (#2386).++ - Read-only post-deployment agent-discovery verifier for final GET/HEAD Link+ headers, documentation targets and optional public-skill digest checks.+ CI exercises local HTTP fixtures rather than contacting production (PR #2299).++ ### Changed++ - A `semantic_text` field with the built-in lexical embedder no longer+ stores its embedding twice. The pooled and per-passage vectors were kept+ in the stored `_source` as well as in the vector index; the columnar+ stored encoder now drops them when they rebuild bit-exactly from the+ text, and every reader (`_source`, `GET`, kNN, reindex, snapshot)+ rebuilds them, so responses are unchanged. On a 6,000-document test the+ segment file went from 20.1 MB to 1.8 MB and the index from 3.67x to+ 1.63x its source size. The saving is on disk; the HNSW graph in memory is+ unchanged. Sections under 128 documents keep their vectors.+ `XERJ_STORE_SEMANTIC_VECTORS=1` turns this off; see the upgrade note+ (part of [#1122](https://github.com/xerj-org/xerj/issues/1122) and+ [#2433](https://github.com/xerj-org/xerj/issues/2433)).+### Fixed+- `_reindex` follows aliases on both sides: a source alias is searched across+ every backing index and each member's alias filter is applied, and a+ destination alias writes to its single write index (an ambiguous alias is+ refused with `illegal_argument_exception`). Previously only the first+ backing index was read or written and filters were ignored, so the call+ reported success over the wrong data.++ - `xerj corpus add` accepts a local Windows path such as+ `F:\Projects\Foo` (#2472). The repo name was the text after the last `/`,+ so a `\`-separated path stayed whole and was refused as an unsafe repo+ name; it now names the repo `Foo`, as `F:/Projects/Foo` already did.++ - `xerj code` shows the part of a long definition that matched the query,+ not its first `--full` characters (#2464). The matching text could sit+ thousands of characters below the signature, so a hit named the right+ function but showed none of the query terms. Such passages now end with+ `... [function name: showing N of M chars from offset O, window centred on the match]`;+ the `@ line N` label and citations are unchanged, and a definition whose+ head is already the best window still ends `truncated at N chars`. Hub ask+ passages (`xerj ask`) and hub search API hits change the same way (PR #2474).++ - A segment merge's output-write step (doc-values build, stored encode, FTS+ side-car build) is now charged to the merge budget before it starts, and a+ background batch yields instead of starting it when it would take RSS to+ the process limit (#2427). That step runs after the last yield checkpoint,+ and neither admission nor the yield checks saw it: large pg-docs merges+ (29k-166k docs) took a 3,072 MB search node to 4.4-4.9 GB about 5 s before+ they completed. The step is estimated at 6x the merged stored JSON or 2x+ the FTS build input, whichever is larger. An admitted step grows the+ batch's merge reservation, so concurrent merge admission sees it.+ `_forcemerge` batches are charged but never refused, and neither is a+ small batch that the yield-streak rule (#2468) lets run to completion (PR #2470).++ - The parent memory breaker no longer flaps, and background merges and flush+ finalizes are bounded while it is engaged (#2427). A 3,000 MB autoindex run+ engaged the breaker 20 times in 16 minutes, often releasing 100-500 ms+ later, and RSS still reached 3,808 MB because merges, FST builds and flush+ buffers kept allocating after writes were getting 429s. The breaker now+ releases below 85% of the watermark (`XERJ_MEMORY_BREAKER_RELEASE_PCT`,+ 50-99) and only after it has been engaged for 2 s+ (`XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS`). RSS that stays flat inside that+ band for 30 s still releases, so a live set no drain can shrink does not+ block writes forever. The breaker never releases while RSS is at or above+ the watermark: the 1 GB release gate (watermark 2,918 MB) released at+ 5,121 MB through the old flat-footprint release (#1981), re-engaged 100 ms+ later and was OOM-killed at 8.4 GB. A footprint that climbs while the+ breaker is engaged is drained again within 2 s instead of waiting out the+ drain backoff. While the breaker is engaged, searches run one at a time,+ and at or above the process memory limit new searches get a 429+ `circuit_breaking_exception`. Every segment merge reserves its estimated working+ set (8x its inputs) against the watermark in one process-wide ledger.+ Background merges wait while the breaker is engaged or when the+ reservation does not fit, and run on the next merge check. One merge+ always runs when nothing else is merging and the breaker is not engaged.+ `_forcemerge` may run alone while engaged. While the breaker is engaged,+ flush finalizes (segment, FST and doc-values builds) run one at a time.+ A background merge batch that is already running stops once the breaker is+ engaged and RSS reaches the process limit, keeps its inputs, and halves the+ input cap of later batches until batches complete again. On the 1 GB gate+ corpus one admitted batch (JSON segments that compress 10-20x) grew a+ 3,072 MB `xerj autoindex` node to 8,122 MB in 18 s while writes were+ getting 429s; plain `_bulk` into one index peaked at 4,967 MB (PR #2452).++ - `xerj autoindex` no longer aborts a long run with `Resource temporarily+ unavailable (os error 11)` when the machine refuses to start a thread. The+ sync replay used to start a new thread for every operation (hundreds of+ thousands on a large folder); it now starts at most the window width of+ workers once per replay and reuses them. Starting a replay worker or an+ indexing worker now retries with a short backoff when thread creation is+ refused, and the replay finishes on the calling thread if no worker can be+ started, so a refusal is no longer fatal (#2428) (PR #2437).+ - `_bulk` now checks the memory limiter before it parses the request, not+ only when each index's batch is written. A bulk that arrived while the memory+ or memtable breaker was engaged still split and copied its whole body first.+ Several concurrent bulks between two RSS samples were each admitted on the+ same old sample, which could carry the node past its memory cap. Each+ request now counts its body against live RSS plus every other in-flight+ reservation until its parse ends, and is answered with one HTTP 429 per action (top-level 429,+ `Retry-After: 1`) when that would reach the watermark or a breaker is+ engaged, so clients back off and re-offer every record (part of #2427) (PR #2440).- `xerj autoindex` no longer runs out of memory while planning a large folder.It used to keep the sampled values of every file in memory at once beforegrouping them into datasets, and kept them all until the end of the run;