AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge remote-tracking branch 'origin/main' into unblock-260-local
xerj-orgf2e9b1a835e5
diff --cc CHANGELOG.mdindex 46380a773,9160e83f7..a68f46a11--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,110 -7,194 +7,298 @@@ and this project adheres to [Semantic V## [Unreleased]+### Fixed++- **`index` on the mapping is now honoured instead of silently ignored**+ ([#204](https://github.com/xerj-org/xerj/issues/204)). `"index": false` was+ accepted by `PUT /{index}`, echoed back verbatim by `GET /{index}/_mapping`,+ and then had no effect at all: the field kept a full inverted index and+ `match`, `term`, `match_phrase`, `prefix` and `wildcard` against it all+ returned the document, where Elasticsearch answers 400. It is the sibling of+ the `doc_values` fix in rc.12 and follows its shape.++ The line is drawn where ES draws it. Since 8.1, `"index": false` on a+ keyword / numeric / date / boolean / ip field keeps the doc-values column and+ stays queryable — `MappedFieldType.isSearchable()` is "has postings **or** has+ doc values" — so those queries keep working here, unchanged. Only a field with+ neither (a `text` field, or anything with an explicit `"doc_values": false`)+ is unsearchable, and a query naming one is now rejected with ES's own+ sentence, `Cannot search on field [f] since it is not indexed nor has doc+ values.`, as a 400 `search_phase_execution_exception` /+ `query_shard_exception`. The check lives in one place — `search_inner` — and+ these surfaces were each measured reporting the rejection rather than+ absorbing it, including the three that were swallowing it: `_search`,+ `_count` on all four selectors (single index, `logs-*`, a comma list and the+ cluster-wide `POST /_count`), `_msearch`, `_msearch/template`, `_explain`,+ `_delete_by_query` and `_rank_eval`. `_explain` answers the 400 instead of a+ confident `"matched": false` for a query it never ran; `_msearch` and+ `_msearch/template` render it as a per-response 400 envelope with `type` and+ `root_cause`, leaving the sibling sub-requests in the batch untouched; and+ `_rank_eval` records the refused request under `failures` instead of dropping+ it from `details` *and* `failures` while still publishing a `metric_score`.++ A multi-index `_count` follows ES's broadcast rule rather than failing+ outright: the failure status is returned only when **no** index answered+ (`RestStatus.status`, semantics only), so `/t/_count` and a `logs-*` whose+ indices all carry the offending mapping both 400 exactly as `_search` does,+ while a mixed selector returns 200 with the partial count and the refusal+ visible as `_shards.failed: 1` plus a `_shards.failures` entry carrying the+ `query_shard_exception`. What it will not do any more is what it did before:+ report `count: 0` with `"successful": 2, "failed": 0`.++ `exists` is unaffected and still returns zero hits rather than an error, which+ is also what ES does.++ The multi-field query types follow ES's own split rather than a blanket rule.+ An explicitly *named* field is rejected like any other named field —+ `multi_match` / `simple_query_string` / `query_string` with+ `"fields": ["note"]`, and `query_string` with `"default_field": "note"` —+ while a *wildcard* spec such as `"fields": ["*"]` is never an error, because+ ES expands a pattern over the searchable fields and silently drops the rest+ (`QueryParserHelper.resolveMappingField`). Before this, the same user intent+ got two different answers: the `simple_query_string` form was rejected (the+ parser lowers it to a `match`) while the `multi_match` form returned the+ document through the unsearchable field.++ Reaching the `query_string` half of that meant fixing a second+ accepted-and-ignored key in the same family: **`query_string`'s `fields`+ array was never read by the parser**, so `{"query":"x","fields":["title"]}`+ searched every field, and — once `index: false` became a real rejection —+ disagreed with `{"default_field":"title"}`, which is the same request written+ the other way. `fields` now selects the targets for a clause that named no+ field of its own (a `dis_max` over one leaf per field when there is more than+ one, `^boost` honoured) — the same job `simple_query_string`'s `fields` list+ was already doing. A clause that does name its own field keeps it, as in ES.++ The field is dropped from the full-text index at flush and merge only where+ the stored-doc fallback is *equivalent* — a non-indexed `text` field. An+ exact-typed field that kept its doc values keeps its whole-value postings,+ because the fallback scan splits on non-alphanumerics and would start matching+ `192.168.0.1` against `192.168.0.2`. So no byte-saving claim is made for this+ change: for those types the footprint the option implies is knowingly forgone+ rather than bought with wrong answers.++ Known gaps, measured and left open rather than claimed shut. **Aggregations+ and sort do not inherit the check**: only the `query` clause is walked, so+ `{"size":0,"aggs":{"a":{"terms":{"field":"note"}}}}` still answers 200 with a+ bucket built from `_source`, and `{"sort":[{"note":"asc"}]}` still answers 200.+ ES rejects those too, but for an unrelated reason and with an unrelated+ sentence — `Fielddata is disabled on [f] in [i]…`, which it raises for *any*+ `text` field whether or not `index: false` was declared — so reusing this+ error there would misreport a wider divergence as this one. **The field-less+ arms of the stored-document scan are schema-free**, walking every `_source`+ key, so a token living only in an unsearchable field can still match when+ **no** field is named: `query_string {"query":"…"}` and `more_like_this` with+ no `fields` both still return the document. (Their *named* forms are+ rejected — `more_like_this` with `fields` lowers to a `bool.should` of+ `match` at parse time and inherits the check.) **A wildcard `fields` spec+ reaches that same field-less arm**: `simple_query_string` / `query_string`+ with `"fields": ["*"]` lowers to the `"*"` placeholder and is answered by the+ schema-free scan, so it can still match through an unsearchable field. It is+ never an *error*, which is ES's rule and deliberate — but it is not "answered+ over the searchable fields" either, and that is a pre-existing property of+ wildcard expansion, identical on `main`. (`multi_match` with a pattern is+ different: it is answered by the FTS projection, which does apply+ `isSearchable()`, and finds nothing in an unsearchable field.) **The opaque+ `query_string` fallback still ignores a multi-field `fields` list**: a query+ string the parser declines to lower keeps the whole string for the FTS path,+ where a one-element `fields` is carried across as `default_field` and a+ longer list has nowhere to go. And **`_validate/query` never opens the+ index** — it answers from the parse alone, so it reports `{"valid": true}`+ for a body `_search` refuses. That too is unchanged from `main`, but it now+ disagrees with `_search`, so it is written down rather than left to be+ discovered. Every one of these gaps is pinned by an assertion in+ `index_false_is_honoured::documented_gaps_are_still_open_and_still_documented`,+ so closing one fails the test that says it is open.++ ## [1.0.0-rc.14] - 2026-08-10++ ### Changed++ - **One documented core and memory policy, and the worker knobs are live**+ ([#240](https://github.com/xerj-org/xerj/issues/240)). XERJ had no single+ answer to "how much of this machine may I take?", and the scattered answers it+ did have disagreed with each other — and, on Darwin, with the OS.+ `engine.flush_workers`, `engine.merge_workers` and `engine.search_workers`+ were documented but inert; they now drive real pool widths, and a value of `0`+ refuses startup instead of being silently reinterpreted. `--bulk-mb` above 24+ and `--workers 0` are likewise refused rather than clamped, so a request the+ engine will not honour fails loudly. Only hand-written configs are affected:+ the shipped `xerj.default.toml` has no `[engine]` section.++ - **One 16 MiB file no longer costs 65 GB of RSS**+ ([#239](https://github.com/xerj-org/xerj/issues/239)). `split_sections` was+ quadratic in both time and memory: a 16 MiB text file with no blank line+ needed **65.6 GB of peak RSS and 16.3 s** to produce 16.8 MB of sections.+ Now linear — the same input takes **8.3 ms and 35 MB**.++ - **BREAKING — `server.bind_address` now defaults to `127.0.0.1`, and a+ cleartext node refuses to publish itself to the network**+ ([#228](https://github.com/xerj-org/xerj/issues/228)). The old default was+ `0.0.0.0` while TLS is off by default, so an out-of-the-box node accepted its+ admin API key over plain HTTP on every interface the host had — verified on a+ stock boot, where `curl -H "Authorization: ApiKey …" http://<lan-ip>:9200/…`+ answered `200` and the same URL over `https` had no listener to hand shake+ with. Auth being on did not help: the credential *is* the thing on the wire.++ A fresh node is now reachable from its own host and nowhere else. Exposing it+ is two statements, not zero: set `server.bind_address` (or `--bind` /+ `XERJ_BIND_ADDRESS`), and — while `tls.enabled = false` — also set+ `server.allow_insecure_network_bind = true` (env+ `XERJ_ALLOW_INSECURE_NETWORK_BIND`). Without the second, startup exits+ non-zero before the data directory is created or a first-run admin key is+ minted. `--insecure` does not evade it; it clears `tls.enabled`, which is+ exactly what the check keys on. Same fail-closed shape as+ [#229](https://github.com/xerj-org/xerj/issues/229), whose TLS-on gRPC check+ is untouched and cannot be relaxed by this opt-out.++ **Upgrading:** a deployment that relied on the old default becomes+ unreachable from other hosts until it sets both; one that already wrote+ `bind_address = "0.0.0.0"` without TLS now fails to start with a message+ naming the setting that unblocks it. The shipped Docker image, compose file+ and Helm chart set both, because a container's network namespace is the+ boundary and its published port is where TLS belongs.++ Two documentation claims that were false as written are now true: the CLI and+ security pages both said `--insecure` "refuses to run with a non-loopback+ bind", and it did not — a `--insecure` node on `0.0.0.0` accepted an+ unauthenticated `PUT /leak/_doc/1` from the LAN (`201`). The startup banner+ also prints the bind address on every listener line, so a loopback node and a+ world-facing one no longer look identical.++ A `bind_address` that is not an IP literal is rejected first, ahead of both+ exposure checks, with the fault it actually has. Host names have never been+ resolved — the pre-#228 code parsed `"{bind}:{port}"` as a socket address and+ failed on them too — but the exposure predicates fail closed on anything they+ cannot parse, so without that ordering `bind_address = "localhost"` was+ refused with a message asserting that localhost "is not loopback" and would+ "serve plain HTTP on a network-reachable interface", and pointed at an opt-out+ that fixed nothing: setting it let the boot run on to the bind and fail there+ instead, after the data directory, the `.xerj_*` system indices, the master+ key and a printed first-run `admin.key` already existed.++ ### Security++ - **API key secrets are no longer stored in the clear**+ ([#201](https://github.com/xerj-org/xerj/issues/201)). `api_keys.json` held+ the secret verbatim. 0600 plus atomic rename is right for a secret a *process*+ owns, but a file has more readers than a process: a backup, a snapshot, a+ container layer, a support bundle, a decommissioned disk. Secrets are now+ stored as a salted SHA-256 (`$ssha256$<salt>$<digest>`) and compared in+ constant time, with existing plaintext records migrated on load. The fast hash+ is deliberate: the secret is two v4 UUIDs — 244 bits of CSPRNG output, never+ human-chosen — so offline guessing is out of reach whatever the hash costs,+ while this comparison runs on *every authenticated request*, where an Argon2id+ would add tens of milliseconds per request and hand out a free CPU-exhaustion+ DoS. The same release makes `_security/_authenticate` report the key's real+ identity instead of a hardcoded `superuser`, and gives the audit chain durable+ storage.++ - **The node refuses to start when TLS is on but gRPC would answer in cleartext+ off-loopback** ([#229](https://github.com/xerj-org/xerj/issues/229)).+ `tls.enabled = true` encrypted two of the three data-plane listeners; the+ third never participated, because the gRPC server is built without tonic's+ `tls` feature. Nothing surfaced it — gRPC clients connected and worked+ identically either way — so an operator who enabled TLS and bound a network+ interface got no error, no failed handshake and no symptom, while API keys and+ document bodies crossed the network in the clear. Auth was never the gap;+ confidentiality was. Boot now fails closed unless+ `tls.allow_insecure_grpc_h2c` says otherwise.++ ### Fixed++ - **`multi_match` returns the same hit set before and after `_flush`**+ ([#218](https://github.com/xerj-org/xerj/issues/218)). The memtable and+ segment paths disagreed on multi-token semantics, so flushing an index could+ change which documents a query matched — verified live on a one-document+ index where an OR query returned 0 hits before `_flush` and 1 after, with ES+ returning 1 throughout.++ - **Dynamic string fields get Elasticsearch's default `.keyword` multi-field**+ ([#209](https://github.com/xerj-org/xerj/issues/209)). Dynamic mapping+ inferred every string as bare `text`, so `GET _mapping` never showed a+ `.keyword` sub-field and `_field_caps` never listed one. Anyone arriving from+ Elasticsearch — or any tool that reads `_field_caps` for field discovery,+ Kibana included — could not see that `category.keyword` existed.++ - **`min_score` is honoured at `size:0`, and an external scalar N keeps its+ ghosts** ([#193](https://github.com/xerj-org/xerj/issues/193)). A+ `size:0 + min_score` body never materialised a hit, so the threshold had+ nothing to filter and `hits.total` reported the raw match count — measured at+ **65** where the same query at `size:100` counted **5**. The tie-break+ comparator item from that issue is deliberately not included here; it belongs+ with [#191](https://github.com/xerj-org/xerj/issues/191).++ - **Index blocks can be removed, `read_only_allow_delete` means what its name+ says, and a blocked write answers 403 rather than 500.** Three compounding+ defects: `PUT /_settings` wrote only the display-side map while enforcement+ read the per-index `settings.json`, so a block was acknowledged, invisible to+ `GET /_settings`, and — with `_block` registered PUT-only — removable only by+ restarting with a hand-edited file. The one block that fires automatically did+ the inverse of its name, and clients hitting any of it were told the server+ had faulted.++ - **`llms.txt` no longer tells agents six things the binary does not do.** The+ "Running an index for a human" section is executed, not just read: an agent+ follows it literally and then reports the result to its user. Corrected: the+ job-size line is not the first line printed; `--dry-run` does not write+ *nothing* (it creates and locks the state dir and appends a journal record);+ a dry run reuses a frozen plan instead of re-costing the job, so it is not a+ floor for the real run; exit `1` is the catch-all for every error, not+ "endpoint unreachable"; `status` needs the same explicit `--state-dir` the run+ used; and `--workers`/`--pdf-workers` bound the client-side extractor, not the+ server (with `--pdf-workers` capped to 1–4).++ - **A failed index can now be inspected, deleted and retried without stopping+ the server** ([#206](https://github.com/xerj-org/xerj/issues/206)). An index+ directory that refused to open at boot was recorded in a map that only+ `Engine::health` ever read. Live-reproduced with a corrupt `snapshot.json`:+ it was absent from `_cat/indices` and `_cluster/state`, `DELETE /{index}`+ answered `404 index_not_found` and left the bytes on disk, and the only+ recovery was to stop the server and edit the data directory by hand. A+ failed index is now a real state: listed as `red` in `_cat/indices`, present+ in `_cluster/state` as an `UNASSIGNED` primary with `ALLOCATION_FAILED` and+ the verbatim open error, enumerated with its reason by+ `GET /_cluster/indices/failed`, reopenable via+ `POST /_cluster/indices/failed/{name}/_retry` once the cause is fixed, and+ deletable through the ordinary `DELETE /{index}`. Searches, writes and+ creates against it return `503 no_shard_available_action_exception` carrying+ the reason instead of a `404` that claimed it did not exist, while the+ metadata surfaces — `GET /{index}`, `HEAD /{index}` (the `indices.exists()`+ every ES client calls), `_mapping`, `_settings`, `_cat/indices` — report it+ as an index that exists, the way ES answers those from cluster metadata+ rather than from a shard. Before this, `HEAD` said the name was free and the+ following `PUT` said it was unavailable.++ Two related defects in the same report are fixed with it. **Readiness no+ longer hard-fails on a partly degraded node** — `/health/ready` returned+ `503` for any red status, so one broken index pulled a pod holding 200+ healthy ones out of service permanently; it now reports `200 ready+ (degraded)` while the node can still serve something and `503` only when+ every index it holds failed to open. And **`_cat/health` no longer prints a+ hardcoded `green`** — the one health surface an existing ES dashboard points+ at was the one that could not report a broken node; it and `_cluster/health`+ now count an unopenable index as an unassigned primary, agreeing with+ `/v1/health` and `/v1/cluster/health`.++ Because `red` is now reachable on `_cluster/health` for the first time, the+ two conditions that gate on it are consulted rather than assumed:+ `wait_for_active_shards=all` and `wait_for_status=green|yellow` both answer+ `408` with `timed_out: true` on a red node instead of `200 timed_out: false`.+ `GET /_cluster/health?wait_for_status=green&timeout=30s` is the bootstrap gate+ every docker healthcheck, CI wait loop and Kibana startup uses; a red node+ used to sail straight through it. A green or yellow cluster is unaffected.++ - **A `DELETE /{index}` that could not remove the bytes no longer strands the+ index.** The open-index path pulled the handle out of the engine before+ `remove_dir_all`, so a removal that failed (read-only mount, `EACCES`) freed+ the name while the directory survived: no handle, no failed-index entry,+ nothing on `_cat/indices`, and the next `DELETE` answered `404` — the same+ dead end as [#206](https://github.com/xerj-org/xerj/issues/206), reached from+ the other side. The handle is now restored on failure, so the index stays in+ service and addressable and the operator can retry the delete once the cause+ is fixed.+- **`autoindex` no longer leaves immortal catalog entries for skipped files**([#238](https://github.com/xerj-org/xerj/issues/238)). A file added after theresume plan was frozen is skipped and reported in the catalog — but that