Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into unblock-260-local

xerj-orgf2e9b1a835e5
diff --cc CHANGELOG.md
index 46380a773,9160e83f7..a68f46a11
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,110 -7,194 +7,298 @@@ and this project adheres to [Semantic V
## [Unreleased]
+### Fixed
+
+- **`index` on the mapping is now honoured instead of silently ignored**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)). `"index": false` was
+ accepted by `PUT /{index}`, echoed back verbatim by `GET /{index}/_mapping`,
+ and then had no effect at all: the field kept a full inverted index and
+ `match`, `term`, `match_phrase`, `prefix` and `wildcard` against it all
+ returned the document, where Elasticsearch answers 400. It is the sibling of
+ the `doc_values` fix in rc.12 and follows its shape.
+
+ The line is drawn where ES draws it. Since 8.1, `"index": false` on a
+ keyword / numeric / date / boolean / ip field keeps the doc-values column and
+ stays queryable — `MappedFieldType.isSearchable()` is "has postings **or** has
+ doc values" — so those queries keep working here, unchanged. Only a field with
+ neither (a `text` field, or anything with an explicit `"doc_values": false`)
+ is unsearchable, and a query naming one is now rejected with ES's own
+ sentence, `Cannot search on field [f] since it is not indexed nor has doc
+ values.`, as a 400 `search_phase_execution_exception` /
+ `query_shard_exception`. The check lives in one place — `search_inner` — and
+ these surfaces were each measured reporting the rejection rather than
+ absorbing it, including the three that were swallowing it: `_search`,
+ `_count` on all four selectors (single index, `logs-*`, a comma list and the
+ cluster-wide `POST /_count`), `_msearch`, `_msearch/template`, `_explain`,
+ `_delete_by_query` and `_rank_eval`. `_explain` answers the 400 instead of a
+ confident `"matched": false` for a query it never ran; `_msearch` and
+ `_msearch/template` render it as a per-response 400 envelope with `type` and
+ `root_cause`, leaving the sibling sub-requests in the batch untouched; and
+ `_rank_eval` records the refused request under `failures` instead of dropping
+ it from `details` *and* `failures` while still publishing a `metric_score`.
+
+ A multi-index `_count` follows ES's broadcast rule rather than failing
+ outright: the failure status is returned only when **no** index answered
+ (`RestStatus.status`, semantics only), so `/t/_count` and a `logs-*` whose
+ indices all carry the offending mapping both 400 exactly as `_search` does,
+ while a mixed selector returns 200 with the partial count and the refusal
+ visible as `_shards.failed: 1` plus a `_shards.failures` entry carrying the
+ `query_shard_exception`. What it will not do any more is what it did before:
+ report `count: 0` with `"successful": 2, "failed": 0`.
+
+ `exists` is unaffected and still returns zero hits rather than an error, which
+ is also what ES does.
+
+ The multi-field query types follow ES's own split rather than a blanket rule.
+ An explicitly *named* field is rejected like any other named field —
+ `multi_match` / `simple_query_string` / `query_string` with
+ `"fields": ["note"]`, and `query_string` with `"default_field": "note"` —
+ while a *wildcard* spec such as `"fields": ["*"]` is never an error, because
+ ES expands a pattern over the searchable fields and silently drops the rest
+ (`QueryParserHelper.resolveMappingField`). Before this, the same user intent
+ got two different answers: the `simple_query_string` form was rejected (the
+ parser lowers it to a `match`) while the `multi_match` form returned the
+ document through the unsearchable field.
+
+ Reaching the `query_string` half of that meant fixing a second
+ accepted-and-ignored key in the same family: **`query_string`'s `fields`
+ array was never read by the parser**, so `{"query":"x","fields":["title"]}`
+ searched every field, and — once `index: false` became a real rejection —
+ disagreed with `{"default_field":"title"}`, which is the same request written
+ the other way. `fields` now selects the targets for a clause that named no
+ field of its own (a `dis_max` over one leaf per field when there is more than
+ one, `^boost` honoured) — the same job `simple_query_string`'s `fields` list
+ was already doing. A clause that does name its own field keeps it, as in ES.
+
+ The field is dropped from the full-text index at flush and merge only where
+ the stored-doc fallback is *equivalent* — a non-indexed `text` field. An
+ exact-typed field that kept its doc values keeps its whole-value postings,
+ because the fallback scan splits on non-alphanumerics and would start matching
+ `192.168.0.1` against `192.168.0.2`. So no byte-saving claim is made for this
+ change: for those types the footprint the option implies is knowingly forgone
+ rather than bought with wrong answers.
+
+ Known gaps, measured and left open rather than claimed shut. **Aggregations
+ and sort do not inherit the check**: only the `query` clause is walked, so
+ `{"size":0,"aggs":{"a":{"terms":{"field":"note"}}}}` still answers 200 with a
+ bucket built from `_source`, and `{"sort":[{"note":"asc"}]}` still answers 200.
+ ES rejects those too, but for an unrelated reason and with an unrelated
+ sentence — `Fielddata is disabled on [f] in [i]…`, which it raises for *any*
+ `text` field whether or not `index: false` was declared — so reusing this
+ error there would misreport a wider divergence as this one. **The field-less
+ arms of the stored-document scan are schema-free**, walking every `_source`
+ key, so a token living only in an unsearchable field can still match when
+ **no** field is named: `query_string {"query":"…"}` and `more_like_this` with
+ no `fields` both still return the document. (Their *named* forms are
+ rejected — `more_like_this` with `fields` lowers to a `bool.should` of
+ `match` at parse time and inherits the check.) **A wildcard `fields` spec
+ reaches that same field-less arm**: `simple_query_string` / `query_string`
+ with `"fields": ["*"]` lowers to the `"*"` placeholder and is answered by the
+ schema-free scan, so it can still match through an unsearchable field. It is
+ never an *error*, which is ES's rule and deliberate — but it is not "answered
+ over the searchable fields" either, and that is a pre-existing property of
+ wildcard expansion, identical on `main`. (`multi_match` with a pattern is
+ different: it is answered by the FTS projection, which does apply
+ `isSearchable()`, and finds nothing in an unsearchable field.) **The opaque
+ `query_string` fallback still ignores a multi-field `fields` list**: a query
+ string the parser declines to lower keeps the whole string for the FTS path,
+ where a one-element `fields` is carried across as `default_field` and a
+ longer list has nowhere to go. And **`_validate/query` never opens the
+ index** — it answers from the parse alone, so it reports `{"valid": true}`
+ for a body `_search` refuses. That too is unchanged from `main`, but it now
+ disagrees with `_search`, so it is written down rather than left to be
+ discovered. Every one of these gaps is pinned by an assertion in
+ `index_false_is_honoured::documented_gaps_are_still_open_and_still_documented`,
+ so closing one fails the test that says it is open.
+
+ ## [1.0.0-rc.14] - 2026-08-10
+
+ ### Changed
+
+ - **One documented core and memory policy, and the worker knobs are live**
+ ([#240](https://github.com/xerj-org/xerj/issues/240)). XERJ had no single
+ answer to "how much of this machine may I take?", and the scattered answers it
+ did have disagreed with each other — and, on Darwin, with the OS.
+ `engine.flush_workers`, `engine.merge_workers` and `engine.search_workers`
+ were documented but inert; they now drive real pool widths, and a value of `0`
+ refuses startup instead of being silently reinterpreted. `--bulk-mb` above 24
+ and `--workers 0` are likewise refused rather than clamped, so a request the
+ engine will not honour fails loudly. Only hand-written configs are affected:
+ the shipped `xerj.default.toml` has no `[engine]` section.
+
+ - **One 16 MiB file no longer costs 65 GB of RSS**
+ ([#239](https://github.com/xerj-org/xerj/issues/239)). `split_sections` was
+ quadratic in both time and memory: a 16 MiB text file with no blank line
+ needed **65.6 GB of peak RSS and 16.3 s** to produce 16.8 MB of sections.
+ Now linear — the same input takes **8.3 ms and 35 MB**.
+
+ - **BREAKING — `server.bind_address` now defaults to `127.0.0.1`, and a
+ cleartext node refuses to publish itself to the network**
+ ([#228](https://github.com/xerj-org/xerj/issues/228)). The old default was
+ `0.0.0.0` while TLS is off by default, so an out-of-the-box node accepted its
+ admin API key over plain HTTP on every interface the host had — verified on a
+ stock boot, where `curl -H "Authorization: ApiKey …" http://<lan-ip>:9200/…`
+ answered `200` and the same URL over `https` had no listener to hand shake
+ with. Auth being on did not help: the credential *is* the thing on the wire.
+
+ A fresh node is now reachable from its own host and nowhere else. Exposing it
+ is two statements, not zero: set `server.bind_address` (or `--bind` /
+ `XERJ_BIND_ADDRESS`), and — while `tls.enabled = false` — also set
+ `server.allow_insecure_network_bind = true` (env
+ `XERJ_ALLOW_INSECURE_NETWORK_BIND`). Without the second, startup exits
+ non-zero before the data directory is created or a first-run admin key is
+ minted. `--insecure` does not evade it; it clears `tls.enabled`, which is
+ exactly what the check keys on. Same fail-closed shape as
+ [#229](https://github.com/xerj-org/xerj/issues/229), whose TLS-on gRPC check
+ is untouched and cannot be relaxed by this opt-out.
+
+ **Upgrading:** a deployment that relied on the old default becomes
+ unreachable from other hosts until it sets both; one that already wrote
+ `bind_address = "0.0.0.0"` without TLS now fails to start with a message
+ naming the setting that unblocks it. The shipped Docker image, compose file
+ and Helm chart set both, because a container's network namespace is the
+ boundary and its published port is where TLS belongs.
+
+ Two documentation claims that were false as written are now true: the CLI and
+ security pages both said `--insecure` "refuses to run with a non-loopback
+ bind", and it did not — a `--insecure` node on `0.0.0.0` accepted an
+ unauthenticated `PUT /leak/_doc/1` from the LAN (`201`). The startup banner
+ also prints the bind address on every listener line, so a loopback node and a
+ world-facing one no longer look identical.
+
+ A `bind_address` that is not an IP literal is rejected first, ahead of both
+ exposure checks, with the fault it actually has. Host names have never been
+ resolved — the pre-#228 code parsed `"{bind}:{port}"` as a socket address and
+ failed on them too — but the exposure predicates fail closed on anything they
+ cannot parse, so without that ordering `bind_address = "localhost"` was
+ refused with a message asserting that localhost "is not loopback" and would
+ "serve plain HTTP on a network-reachable interface", and pointed at an opt-out
+ that fixed nothing: setting it let the boot run on to the bind and fail there
+ instead, after the data directory, the `.xerj_*` system indices, the master
+ key and a printed first-run `admin.key` already existed.
+
+ ### Security
+
+ - **API key secrets are no longer stored in the clear**
+ ([#201](https://github.com/xerj-org/xerj/issues/201)). `api_keys.json` held
+ the secret verbatim. 0600 plus atomic rename is right for a secret a *process*
+ owns, but a file has more readers than a process: a backup, a snapshot, a
+ container layer, a support bundle, a decommissioned disk. Secrets are now
+ stored as a salted SHA-256 (`$ssha256$<salt>$<digest>`) and compared in
+ constant time, with existing plaintext records migrated on load. The fast hash
+ is deliberate: the secret is two v4 UUIDs — 244 bits of CSPRNG output, never
+ human-chosen — so offline guessing is out of reach whatever the hash costs,
+ while this comparison runs on *every authenticated request*, where an Argon2id
+ would add tens of milliseconds per request and hand out a free CPU-exhaustion
+ DoS. The same release makes `_security/_authenticate` report the key's real
+ identity instead of a hardcoded `superuser`, and gives the audit chain durable
+ storage.
+
+ - **The node refuses to start when TLS is on but gRPC would answer in cleartext
+ off-loopback** ([#229](https://github.com/xerj-org/xerj/issues/229)).
+ `tls.enabled = true` encrypted two of the three data-plane listeners; the
+ third never participated, because the gRPC server is built without tonic's
+ `tls` feature. Nothing surfaced it — gRPC clients connected and worked
+ identically either way — so an operator who enabled TLS and bound a network
+ interface got no error, no failed handshake and no symptom, while API keys and
+ document bodies crossed the network in the clear. Auth was never the gap;
+ confidentiality was. Boot now fails closed unless
+ `tls.allow_insecure_grpc_h2c` says otherwise.
+
+ ### Fixed
+
+ - **`multi_match` returns the same hit set before and after `_flush`**
+ ([#218](https://github.com/xerj-org/xerj/issues/218)). The memtable and
+ segment paths disagreed on multi-token semantics, so flushing an index could
+ change which documents a query matched — verified live on a one-document
+ index where an OR query returned 0 hits before `_flush` and 1 after, with ES
+ returning 1 throughout.
+
+ - **Dynamic string fields get Elasticsearch's default `.keyword` multi-field**
+ ([#209](https://github.com/xerj-org/xerj/issues/209)). Dynamic mapping
+ inferred every string as bare `text`, so `GET _mapping` never showed a
+ `.keyword` sub-field and `_field_caps` never listed one. Anyone arriving from
+ Elasticsearch — or any tool that reads `_field_caps` for field discovery,
+ Kibana included — could not see that `category.keyword` existed.
+
+ - **`min_score` is honoured at `size:0`, and an external scalar N keeps its
+ ghosts** ([#193](https://github.com/xerj-org/xerj/issues/193)). A
+ `size:0 + min_score` body never materialised a hit, so the threshold had
+ nothing to filter and `hits.total` reported the raw match count — measured at
+ **65** where the same query at `size:100` counted **5**. The tie-break
+ comparator item from that issue is deliberately not included here; it belongs
+ with [#191](https://github.com/xerj-org/xerj/issues/191).
+
+ - **Index blocks can be removed, `read_only_allow_delete` means what its name
+ says, and a blocked write answers 403 rather than 500.** Three compounding
+ defects: `PUT /_settings` wrote only the display-side map while enforcement
+ read the per-index `settings.json`, so a block was acknowledged, invisible to
+ `GET /_settings`, and — with `_block` registered PUT-only — removable only by
+ restarting with a hand-edited file. The one block that fires automatically did
+ the inverse of its name, and clients hitting any of it were told the server
+ had faulted.
+
+ - **`llms.txt` no longer tells agents six things the binary does not do.** The
+ "Running an index for a human" section is executed, not just read: an agent
+ follows it literally and then reports the result to its user. Corrected: the
+ job-size line is not the first line printed; `--dry-run` does not write
+ *nothing* (it creates and locks the state dir and appends a journal record);
+ a dry run reuses a frozen plan instead of re-costing the job, so it is not a
+ floor for the real run; exit `1` is the catch-all for every error, not
+ "endpoint unreachable"; `status` needs the same explicit `--state-dir` the run
+ used; and `--workers`/`--pdf-workers` bound the client-side extractor, not the
+ server (with `--pdf-workers` capped to 1–4).
+
+ - **A failed index can now be inspected, deleted and retried without stopping
+ the server** ([#206](https://github.com/xerj-org/xerj/issues/206)). An index
+ directory that refused to open at boot was recorded in a map that only
+ `Engine::health` ever read. Live-reproduced with a corrupt `snapshot.json`:
+ it was absent from `_cat/indices` and `_cluster/state`, `DELETE /{index}`
+ answered `404 index_not_found` and left the bytes on disk, and the only
+ recovery was to stop the server and edit the data directory by hand. A
+ failed index is now a real state: listed as `red` in `_cat/indices`, present
+ in `_cluster/state` as an `UNASSIGNED` primary with `ALLOCATION_FAILED` and
+ the verbatim open error, enumerated with its reason by
+ `GET /_cluster/indices/failed`, reopenable via
+ `POST /_cluster/indices/failed/{name}/_retry` once the cause is fixed, and
+ deletable through the ordinary `DELETE /{index}`. Searches, writes and
+ creates against it return `503 no_shard_available_action_exception` carrying
+ the reason instead of a `404` that claimed it did not exist, while the
+ metadata surfaces — `GET /{index}`, `HEAD /{index}` (the `indices.exists()`
+ every ES client calls), `_mapping`, `_settings`, `_cat/indices` — report it
+ as an index that exists, the way ES answers those from cluster metadata
+ rather than from a shard. Before this, `HEAD` said the name was free and the
+ following `PUT` said it was unavailable.
+
+ Two related defects in the same report are fixed with it. **Readiness no
+ longer hard-fails on a partly degraded node** — `/health/ready` returned
+ `503` for any red status, so one broken index pulled a pod holding 200
+ healthy ones out of service permanently; it now reports `200 ready
+ (degraded)` while the node can still serve something and `503` only when
+ every index it holds failed to open. And **`_cat/health` no longer prints a
+ hardcoded `green`** — the one health surface an existing ES dashboard points
+ at was the one that could not report a broken node; it and `_cluster/health`
+ now count an unopenable index as an unassigned primary, agreeing with
+ `/v1/health` and `/v1/cluster/health`.
+
+ Because `red` is now reachable on `_cluster/health` for the first time, the
+ two conditions that gate on it are consulted rather than assumed:
+ `wait_for_active_shards=all` and `wait_for_status=green|yellow` both answer
+ `408` with `timed_out: true` on a red node instead of `200 timed_out: false`.
+ `GET /_cluster/health?wait_for_status=green&timeout=30s` is the bootstrap gate
+ every docker healthcheck, CI wait loop and Kibana startup uses; a red node
+ used to sail straight through it. A green or yellow cluster is unaffected.
+
+ - **A `DELETE /{index}` that could not remove the bytes no longer strands the
+ index.** The open-index path pulled the handle out of the engine before
+ `remove_dir_all`, so a removal that failed (read-only mount, `EACCES`) freed
+ the name while the directory survived: no handle, no failed-index entry,
+ nothing on `_cat/indices`, and the next `DELETE` answered `404` — the same
+ dead end as [#206](https://github.com/xerj-org/xerj/issues/206), reached from
+ the other side. The handle is now restored on failure, so the index stays in
+ service and addressable and the operator can retry the delete once the cause
+ is fixed.
+
- **`autoindex` no longer leaves immortal catalog entries for skipped files**
([#238](https://github.com/xerj-org/xerj/issues/238)). A file added after the
resume plan was frozen is skipped and reported in the catalog — but that