AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge contributor PR #2463 for composed back-pressure review
xerj-org6a4d14bd3a7f
diff --cc CHANGELOG.mdindex c1b61677e,e7efe2b14..338d70499--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -124,80 -77,17 +124,89 @@@ continues in #2427 and #1122, which sta[#1170](https://github.com/xerj-org/xerj/issues/1170); the issue staysopen).+- **Release performance gate** for autoindex and search. It adds+ `gen_corpus.py`, `autoindex_release_gate.sh` and `gate.py`, with per-phase+ budgets, an Elasticsearch floor and an rc.99 comparison harness, and it+ runs before a release is cut (#2432). The live-tree memory follow-up is+ tracked in #2433, which stays open.++- `xerj init` prints the Kimi CLI MCP config path (`~/.kimi-code/mcp.json`),+ and the README marks the Kimi CLI row as tested (#2386).++- Read-only post-deployment agent-discovery verifier for final GET/HEAD Link+ headers, documentation targets and optional public-skill digest checks.+ CI exercises local HTTP fixtures rather than contacting production (PR #2299).++### Changed++- A `semantic_text` field with the built-in lexical embedder no longer+ stores its embedding twice. The pooled and per-passage vectors were kept+ in the stored `_source` as well as in the vector index; the columnar+ stored encoder now drops them when they rebuild bit-exactly from the+ text, and every reader (`_source`, `GET`, kNN, reindex, snapshot)+ rebuilds them, so responses are unchanged. On a 6,000-document test the+ segment file went from 20.1 MB to 1.8 MB and the index from 3.67x to+ 1.63x its source size. The saving is on disk; the HNSW graph in memory is+ unchanged. Sections under 128 documents keep their vectors.+ `XERJ_STORE_SEMANTIC_VECTORS=1` turns this off; see the upgrade note+ (part of [#1122](https://github.com/xerj-org/xerj/issues/1122) and+ [#2433](https://github.com/xerj-org/xerj/issues/2433)).+### Fixed+- `xerj search` names an unknown flag on one line and exits 2 instead of+ following "unexpected argument" with the whole usage text (#2458). An+ agent that guessed `--index` and `--max-hits` read the help dump as the+ answer and gave up on the CLI. Those guesses, and `--size`, `--limit`,+ `--top`, `--top-k`, `-n` and `-i`, now point at `--prefix <P>` or+ `-k <N>`.+- `xerj_batch_search` (and `xerj batch-search`) accept a plain string in+ `queries` as shorthand for `{"query": <string>}` (#2497). A string item+ used to fail alone with "arguments must be a JSON object", so an agent+ that sent `["a", "b"]` lost a call; the advertised schema now allows it.++- `xerj corpus add` accepts a local Windows path such as+ `F:\Projects\Foo` (#2472). The repo name was the text after the last `/`,+ so a `\`-separated path stayed whole and was refused as an unsafe repo+ name; it now names the repo `Foo`, as `F:/Projects/Foo` already did.++- `xerj autoindex` no longer leaves large `.autoindex-stage-*` files in the+ state dir (#2427, PR [#2493](https://github.com/xerj-org/xerj/pull/2493)).+ Phase B staged each file as uncompressed NDJSON in a named temp file, which+ was several times the size of the source for row-shaped data (a 31.7 MB+ trace staged as about 215 MB) and was left behind when the client was+ killed by a signal. Stages are now zstd-compressed and written to an+ unnamed temp file that the kernel frees when the client exits, however it+ exits.+- `xerj code` shows the part of a long definition that matched the query,+ not its first `--full` characters (#2464). The matching text could sit+ thousands of characters below the signature, so a hit named the right+ function but showed none of the query terms. Such passages now end with+ `... [function name: showing N of M chars from offset O, window centred on the match]`;+ the `@ line N` label and citations are unchanged, and a definition whose+ head is already the best window still ends `truncated at N chars`. Hub ask+ passages (`xerj ask`) and hub search API hits change the same way (PR #2474).++- A segment merge's output-write step (doc-values build, stored encode, FTS+ side-car build) is now charged to the merge budget before it starts, and a+ background batch yields instead of starting it when it would take RSS to+ the process limit (#2427). That step runs after the last yield checkpoint,+ and neither admission nor the yield checks saw it: large pg-docs merges+ (29k-166k docs) took a 3,072 MB search node to 4.4-4.9 GB about 5 s before+ they completed. The step is estimated at 6x the merged stored JSON or 2x+ the FTS build input, whichever is larger. An admitted step grows the+ batch's merge reservation, so concurrent merge admission sees it.+ `_forcemerge` batches are charged but never refused, and neither is a+ small batch that the yield-streak rule (#2468) lets run to completion (PR #2470).+ - `autoindex` bulk back-pressure patience is now measured since the last+ progress, not since the first round. A window the server drains a few+ records at a time (per-item 429 `resource_exhausted` on an embedding+ deadline or admission refusal) used to abort with `bulk backend failed`+ once the rounds together passed the 600 s `THROTTLE_PATIENCE`, even though+ every round settled records. Each round that settles at least one item now+ restarts the patience clock and the backoff; rounds that settle nothing+ still count, so a server that answers 429 forever fails at the same limits+ as before.- The parent memory breaker no longer flaps, and background merges and flushfinalizes are bounded while it is engaged (#2427). A 3,000 MB autoindex run