Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge main (v1.0.0-rc.14) into fix/issue-207-security-tooling

xerj-orgfe8233a76085
diff --cc CHANGELOG.md
index d7e51397c,9160e83f7..fe0520c4e
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,208 -7,194 +7,398 @@@ and this project adheres to [Semantic V
## [Unreleased]
+### Security
+
+- **`cargo-audit` and `cargo-fuzz` now actually run in CI — they were only ever
+ documented** ([#207](https://github.com/xerj-org/xerj/issues/207)).
+ `user-feedback/09-security/cves-and-vulnerabilities.md` listed "`cargo-audit`
+ for dependency vulnerability scanning" and "fuzz testing (cargo-fuzz) on all
+ input parsing paths" as part of XERJ's answer to Elasticsearch's CVE history.
+ Neither existed: no audit job, no fuzz job, no fuzz target anywhere in the
+ tree. A reader comparing the two engines priced in a fuzzed parser surface
+ that was not there.
+ - New CI job `security-audit` runs `cargo audit` on every push and pull
+ request; a RUSTSEC vulnerability advisory fails the build.
+ - New CI job `fuzz` builds and runs seven libFuzzer harnesses
+ (`engine/fuzz/`) over checked-in seeds: the Elasticsearch query DSL, the
+ Lucene `query_string` grammar, date math and date-format patterns,
+ index-name date math, SQL, Painless scripts, and the aggregation engine's
+ two separate script tokenisers. `.github/scripts/fuzz-smoke.sh` is the same
+ entry point locally and in CI.
+ - `xerj-engine/tests/security_tooling_claims.rs` fails if a documented tool
+ stops running, if a fuzz target ships without a seed corpus, or if the
+ "all input parsing paths" overclaim reappears in any prose file.
+ - The claim itself is now specific about what is and is not covered.
+ - **Known gap, stated because the point of this entry is not to overclaim
+ again:** seven harnesses are seven parsers, not the engine's whole input
+ surface, and one parser outside them is already known to abort.
+ `aggs::parse_time_zone_offset` slices a `time_zone` string at byte 2 after
+ a *byte*-length check, so a `date_histogram` with `"time_zone": "+中a"` in
+ an unauthenticated `_search` body dies on a character boundary. It is
+ pre-existing, is not touched by this change, and is tracked in
+ [#272](https://github.com/xerj-org/xerj/issues/272) with a fix and a
+ proposed harness for the aggregation date parsers.
+
+- **Fixed two high-severity advisories in the document-ingest XML parser.**
+ Turning the audit gate on found `quick-xml` 0.36.2 in `xerj-autoindex`, which
+ parses `.docx`, `.xlsx` and `.xml` files supplied by whoever owns the folder
+ being indexed: RUSTSEC-2026-0195 (unbounded namespace-declaration allocation →
+ memory-exhaustion DoS, CVSS 7.5) and RUSTSEC-2026-0194 (quadratic run time on
+ duplicate attribute names, CVSS 7.5). Upgraded to 0.41, which also
+ de-duplicates the crate — 0.41 was already in the tree via `pdf_oxide`.
+ `crossbeam-epoch` moved 0.9.18 → 0.9.20 for RUSTSEC-2026-0204.
+
+### Changed
+
+- **`merge.strategy = "log_structured"` is now refused at startup instead of
+ silently running the size-tiered policy**
+ ([#207](https://github.com/xerj-org/xerj/issues/207)). Nothing in the tree
+ ever read `merge.strategy`; `run_merge_once` always builds a
+ `SizeTieredMergePolicy`. An operator who chose a levelled policy for its read
+ amplification got the other one, quietly. This follows the `storage.backend`
+ and `vector.default_quantization` guards already in `Config::validate`.
+
+- **The three dormant `[merge]` settings now say so, in the code, in the shipped
+ TOML, on the docs site, and in the log at startup**
+ ([#207](https://github.com/xerj-org/xerj/issues/207)). The issue reported
+ merge I/O rate limiting as "claimed but dormant" and it is: the `RateLimiter`
+ that honours `io_rate_mb_per_sec` is wired only into `xerj-storage`'s legacy
+ `MergeExecutor`, which the engine never constructs. `min_segments` and
+ `max_concurrent` are unread too — the real per-tier trigger is
+ `min_merge_count`, and merge parallelism comes from `XERJ_MERGE_PARALLELISM`.
+ `MergeConfig::dormant_overrides` names any of the three the operator has moved
+ off its default and `xerj-server` logs each at WARN, so the operator throttling
+ merges to protect query latency finds out instead of guessing. They stay
+ accepted rather than becoming hard errors because the wrong value costs
+ latency, not data, and `io_rate_mb_per_sec = 100` has shipped in
+ `xerj.default.toml` since v0.1. Wiring a real throttle into `run_merge_once`
+ is the fix that makes this list shorter.
+
+- **`engine/xerj.default.toml` disagreed with the defaults it claims to
+ document, in four places.** `max_segment_mb` said 5120 against a real 8192,
+ `wal_max_size_mb` 512 against 1024, `flush_size_mb` 256 against 512, and
+ `default_quantization` `"scalar8"` against `"none"` — so `cp xerj.default.toml
+ xerj.toml`, the documented first step, silently changed four engine behaviours
+ including turning on 8-bit vector quantization. The file is corrected and
+ `shipped_default_config_documents_the_real_defaults` now diffs every leaf key
+ against `Config::default()`, so it cannot drift again.
+
+- **The docs site had drifted the same way, and now has the same guard.**
+ `landing/docs/config.html` shipped `wal_max_size_mb = 512`,
+ `flush_size_mb = 256` and `default_quantization = "scalar8"` in its
+ copy-pasteable `[storage]` and `[vector]` blocks while its own DEFAULT
+ table — thirty lines up, on the same page — said 1024, 512 and `"none"`;
+ `landing/docs/storage.html` said the WAL rolls at 512 MiB;
+ `landing/docs/vectors.html` called `scalar8` "the default" and listed a
+ `scalar4` mode that no config or mapping value can reach; and
+ `engine/README.md` put `flush_size_mb` at 256 MiB. All four are corrected,
+ along with the docs search-index blob duplicated across 44 landing pages,
+ which offered `scalar4` as a selectable quantization mode.
+ A second guard, `the_docs_site_config_page_agrees_with_the_real_defaults`,
+ now diffs *both* halves of that page — every DEFAULT cell in the reference
+ table and every assignment in the example blocks — against
+ `Config::default()`. An example that deliberately differs (the blocks whose
+ point is switching TLS or cluster mode on) has to carry `# not a default` on
+ the line, so the reader is told as well as the test.
+
+- **Four other claims from [#207](https://github.com/xerj-org/xerj/issues/207)
+ now match the source.**
+ - *Settings count.* `config.rs` said 38 in its header and 56 twenty lines
+ down while the count test asserted 60 — and the test asserted it by
+ re-adding a hardcoded sum (`5 + 2 + 3 + … - 1 == 38`), an identity that
+ could not fail whatever `Config` held. There was a second copy of the same
+ non-test in `config.rs` itself (`count_user_facing_settings`, `61 == 61`).
+ Both now serialise `Config::default()` and count leaf keys, section by
- section. The measured figure is **103**, and 38 / 50 / 56 / 60 / 61 / "<50"
++ section. The measured figure is **104** (103 when this work was done; the
++ rc.14 `server.allow_insecure_network_bind` key makes it 104), and 38 / 50 /
++ 56 / 60 / 61 / "<50"
+ are corrected in `config.rs`, `xerj-common/src/lib.rs`, `engine/README.md`,
+ `xerj.default.toml` and the feedback responses, along with the stale
+ per-sub-config annotations (limits 3 → 13, storage 5 → 10, embedding 4 → 19,
+ merge 5 → 8, cluster 4 → 5, tls 3 → 4, auth 2 → 3). This closes item 10 of
- `user-feedback/ROADMAP-TO-GA.md`. 103 versus 3,000+ is still the winning
++ `user-feedback/ROADMAP-TO-GA.md`. 104 versus 3,000+ is still the winning
+ story, told truthfully.
+ - *Wire-compatibility test framing.* `journey_es_migration` was described as
+ proving "the same curl commands" work. It calls the Rust engine API
+ directly and never makes an HTTP request. The response-shape assertions are
+ real and stay; the comment now says what the test does and points at the
+ ES-compat YAML conformance suite, which is what actually covers the wire.
+ - *OpenAPI spec.* `landing/openapi.json` declared `version: 1.0` while
- specifying 7 routes of roughly 210. It is now titled and versioned as a
- partial spec, and says so.
++ specifying 7 routes out of the 200-plus the router registers. It is now
++ titled and versioned as a partial spec, and says so.
+
+### Fixed
+
+- **A `query_string` containing one non-ASCII character aborted the process.**
+ `{"query_string":{"query":"_\u0660"}}` — or the same text in `?q=` — took the
+ server down. The Lucene tokeniser walks bytes but slices `&q[start..i]` as a
+ `&str`, and it broke on `is_whitespace()`, which is true for Latin-1 NBSP
+ (0xA0). In valid UTF-8 that byte is only ever a *continuation* byte, so the
+ scan stopped between the two bytes of U+0660 and the slice panicked on a
+ character boundary. Every delimiter test in that scanner is now `is_ascii_*`,
+ which makes "the scan can only stop on a character boundary" true by
+ construction. Nothing is lost: the only non-ASCII bytes the old test accepted
+ were 0x85 and 0xA0, neither of which is a standalone character in UTF-8.
+
+- **An index name of `<{}{}>` aborted the process.** Index-name date math finds
+ the first `{` and the last `}` inside the braces; when the `}` comes first,
+ `date_part[2..0]` is a panic, not an empty slice. Reachable from any request
+ that names an index. The closing brace is now required to come after the
+ opening one.
+
+- **A Painless script containing one non-ASCII character aborted the process.**
+ The second thing the new fuzz harnesses found, inside a second. The tokeniser
+ walks *bytes* (`bytes[i] as char` reinterprets each byte as Latin-1) but
+ slices `&src[start..i]` as a `&str`. Every continuation byte of a multi-byte
+ character reads as an alphabetic Latin-1 char, so an identifier scan could
+ start or stop inside one and the slice panicked on a char boundary —
+ `end byte index 24 is not a char boundary; it is inside 'ʋ'`. Scripts arrive
+ in ordinary search and update bodies and the release profile sets
+ `panic = "abort"`, so this was not a 400, it was the whole server. The
+ identifier scan is now ASCII-only, which keeps both ends on a character
+ boundary by construction; a non-ASCII byte reaches the tokeniser's
+ `unexpected char` error instead. String literals still carry any Unicode.
+
+- **A search body could abort the whole process: `{"range":{"ts":{"gte":"now+33333333333333H"}}}`.**
+ The first thing the new `date_math` fuzz target found, 20 seconds into its
+ first run. `chrono`'s `Duration::hours` and its siblings **panic** when the
+ count does not fit a `TimeDelta`, and they run before the
+ `checked_add_signed` that was supposed to make
+ `xerj_query::dates::add_unit` total — so the careful checking around them
+ bought nothing. A range bound goes straight from the request body into this
+ code (`parser.rs:931`), the release profile sets `panic = "abort"`, and the
+ result is an unauthenticated remote denial of service on a released binary.
+ Fixed with the fallible `try_*` constructors. Grepping for the same pattern
+ found **three** more copies in the index-name date-math resolvers —
+ `xerj-engine/src/index.rs`, and *both* branches of
+ `xerj-api/src/es_compat.rs`: the `now` form (`<logs-{now+9999999999999d}>`)
+ and the anchored form (`<logs-{2026-01-01||+9999999999999d}>`), which are
+ handled by two different functions sixty lines apart. All three are fixed,
+ their `n * 30` / `n * 365` multiplications are `checked_mul`, the unit is read
+ as one character rather than one byte, and the addition goes through
+ `checked_add_signed`. `es_compat`'s resolver is the one that runs on the wire
+ — the index path parameter of every create/index call and the `_search` index
+ list — so `crates/xerj-api/tests/index_name_date_math_is_total.rs` now drives
+ it at the crate boundary; the same-file fix that missed the anchored branch
+ passed because nothing exercised the public entry point.
+
+- **A `_search` aggregation script containing one non-ASCII character aborted
+ the process.** `painless::tokenize` was not the only Painless-subset scanner
+ in the tree: `aggs.rs` holds two more with the identical byte-walk /
+ `&str`-slice shape — `lex_script` (`scripted_metric`) and `tokenize_script`
+ (`bucket_script` / `bucket_selector`) — and both took `&src[i..i + 2]` to test
+ for a two-character operator at a byte index that can sit inside a multi-byte
+ character, because every arm above it is ASCII-only.
+ `{"m":{"scripted_metric":{"map_script":"中"}}}` was `end byte index 2 is not a
+ char boundary`. Both also skipped whitespace with `is_whitespace()` on
+ `bytes[i] as char`, which is true for Latin-1 NEL and NBSP — bytes that in
+ valid UTF-8 are only ever *continuation* bytes. The two-character probe now
+ requires both bytes to be ASCII (every such operator is ASCII, so nothing is
+ lost) and whitespace is `is_ascii_whitespace`. A new `agg_script` fuzz target
+ covers all three entry points, which is how the next one gets found instead of
+ reported.
+
+- **An XML entity nobody declared no longer discards the text around it.**
+ Fallout from the `quick-xml` upgrade above, which is a breaking change to how
+ character data arrives: 0.38 stopped folding `&amp;` into the surrounding
+ `Event::Text` and began emitting each reference as its own
+ `Event::GeneralRef`. Entity handling is preserved across that change (a
+ reader that ignored the new event would have silently dropped every `&`, `<`,
+ `>` and `&#233;` while the words around them still arrived), and one case
+ gets better: 0.36's `BytesText::unescape()` failed the whole text node on an
+ entity no DTD declared for us, and `unwrap_or_default()` then discarded it —
+ now only the reference itself is dropped. Character data is also reassembled
+ from its fragments before being emitted, so an entity-bearing field stays one
+ value instead of becoming an array of pieces.
+
+ ## [1.0.0-rc.14] - 2026-08-10
+
+ ### Changed
+
+ - **One documented core and memory policy, and the worker knobs are live**
+ ([#240](https://github.com/xerj-org/xerj/issues/240)). XERJ had no single
+ answer to "how much of this machine may I take?", and the scattered answers it
+ did have disagreed with each other — and, on Darwin, with the OS.
+ `engine.flush_workers`, `engine.merge_workers` and `engine.search_workers`
+ were documented but inert; they now drive real pool widths, and a value of `0`
+ refuses startup instead of being silently reinterpreted. `--bulk-mb` above 24
+ and `--workers 0` are likewise refused rather than clamped, so a request the
+ engine will not honour fails loudly. Only hand-written configs are affected:
+ the shipped `xerj.default.toml` has no `[engine]` section.
+
+ - **One 16 MiB file no longer costs 65 GB of RSS**
+ ([#239](https://github.com/xerj-org/xerj/issues/239)). `split_sections` was
+ quadratic in both time and memory: a 16 MiB text file with no blank line
+ needed **65.6 GB of peak RSS and 16.3 s** to produce 16.8 MB of sections.
+ Now linear — the same input takes **8.3 ms and 35 MB**.
+
+ - **BREAKING — `server.bind_address` now defaults to `127.0.0.1`, and a
+ cleartext node refuses to publish itself to the network**
+ ([#228](https://github.com/xerj-org/xerj/issues/228)). The old default was
+ `0.0.0.0` while TLS is off by default, so an out-of-the-box node accepted its
+ admin API key over plain HTTP on every interface the host had — verified on a
+ stock boot, where `curl -H "Authorization: ApiKey …" http://<lan-ip>:9200/…`
+ answered `200` and the same URL over `https` had no listener to hand shake
+ with. Auth being on did not help: the credential *is* the thing on the wire.
+
+ A fresh node is now reachable from its own host and nowhere else. Exposing it
+ is two statements, not zero: set `server.bind_address` (or `--bind` /
+ `XERJ_BIND_ADDRESS`), and — while `tls.enabled = false` — also set
+ `server.allow_insecure_network_bind = true` (env
+ `XERJ_ALLOW_INSECURE_NETWORK_BIND`). Without the second, startup exits
+ non-zero before the data directory is created or a first-run admin key is
+ minted. `--insecure` does not evade it; it clears `tls.enabled`, which is
+ exactly what the check keys on. Same fail-closed shape as
+ [#229](https://github.com/xerj-org/xerj/issues/229), whose TLS-on gRPC check
+ is untouched and cannot be relaxed by this opt-out.
+
+ **Upgrading:** a deployment that relied on the old default becomes
+ unreachable from other hosts until it sets both; one that already wrote
+ `bind_address = "0.0.0.0"` without TLS now fails to start with a message
+ naming the setting that unblocks it. The shipped Docker image, compose file
+ and Helm chart set both, because a container's network namespace is the
+ boundary and its published port is where TLS belongs.
+
+ Two documentation claims that were false as written are now true: the CLI and
+ security pages both said `--insecure` "refuses to run with a non-loopback
+ bind", and it did not — a `--insecure` node on `0.0.0.0` accepted an
+ unauthenticated `PUT /leak/_doc/1` from the LAN (`201`). The startup banner
+ also prints the bind address on every listener line, so a loopback node and a
+ world-facing one no longer look identical.
+
+ A `bind_address` that is not an IP literal is rejected first, ahead of both
+ exposure checks, with the fault it actually has. Host names have never been
+ resolved — the pre-#228 code parsed `"{bind}:{port}"` as a socket address and
+ failed on them too — but the exposure predicates fail closed on anything they
+ cannot parse, so without that ordering `bind_address = "localhost"` was
+ refused with a message asserting that localhost "is not loopback" and would
+ "serve plain HTTP on a network-reachable interface", and pointed at an opt-out
+ that fixed nothing: setting it let the boot run on to the bind and fail there
+ instead, after the data directory, the `.xerj_*` system indices, the master
+ key and a printed first-run `admin.key` already existed.
+
+ ### Security
+
+ - **API key secrets are no longer stored in the clear**
+ ([#201](https://github.com/xerj-org/xerj/issues/201)). `api_keys.json` held
+ the secret verbatim. 0600 plus atomic rename is right for a secret a *process*
+ owns, but a file has more readers than a process: a backup, a snapshot, a
+ container layer, a support bundle, a decommissioned disk. Secrets are now
+ stored as a salted SHA-256 (`$ssha256$<salt>$<digest>`) and compared in
+ constant time, with existing plaintext records migrated on load. The fast hash
+ is deliberate: the secret is two v4 UUIDs — 244 bits of CSPRNG output, never
+ human-chosen — so offline guessing is out of reach whatever the hash costs,
+ while this comparison runs on *every authenticated request*, where an Argon2id
+ would add tens of milliseconds per request and hand out a free CPU-exhaustion
+ DoS. The same release makes `_security/_authenticate` report the key's real
+ identity instead of a hardcoded `superuser`, and gives the audit chain durable
+ storage.
+
+ - **The node refuses to start when TLS is on but gRPC would answer in cleartext
+ off-loopback** ([#229](https://github.com/xerj-org/xerj/issues/229)).
+ `tls.enabled = true` encrypted two of the three data-plane listeners; the
+ third never participated, because the gRPC server is built without tonic's
+ `tls` feature. Nothing surfaced it — gRPC clients connected and worked
+ identically either way — so an operator who enabled TLS and bound a network
+ interface got no error, no failed handshake and no symptom, while API keys and
+ document bodies crossed the network in the clear. Auth was never the gap;
+ confidentiality was. Boot now fails closed unless
+ `tls.allow_insecure_grpc_h2c` says otherwise.
+
+ ### Fixed
+
+ - **`multi_match` returns the same hit set before and after `_flush`**
+ ([#218](https://github.com/xerj-org/xerj/issues/218)). The memtable and
+ segment paths disagreed on multi-token semantics, so flushing an index could
+ change which documents a query matched — verified live on a one-document
+ index where an OR query returned 0 hits before `_flush` and 1 after, with ES
+ returning 1 throughout.
+
+ - **Dynamic string fields get Elasticsearch's default `.keyword` multi-field**
+ ([#209](https://github.com/xerj-org/xerj/issues/209)). Dynamic mapping
+ inferred every string as bare `text`, so `GET _mapping` never showed a
+ `.keyword` sub-field and `_field_caps` never listed one. Anyone arriving from
+ Elasticsearch — or any tool that reads `_field_caps` for field discovery,
+ Kibana included — could not see that `category.keyword` existed.
+
+ - **`min_score` is honoured at `size:0`, and an external scalar N keeps its
+ ghosts** ([#193](https://github.com/xerj-org/xerj/issues/193)). A
+ `size:0 + min_score` body never materialised a hit, so the threshold had
+ nothing to filter and `hits.total` reported the raw match count — measured at
+ **65** where the same query at `size:100` counted **5**. The tie-break
+ comparator item from that issue is deliberately not included here; it belongs
+ with [#191](https://github.com/xerj-org/xerj/issues/191).
+
+ - **Index blocks can be removed, `read_only_allow_delete` means what its name
+ says, and a blocked write answers 403 rather than 500.** Three compounding
+ defects: `PUT /_settings` wrote only the display-side map while enforcement
+ read the per-index `settings.json`, so a block was acknowledged, invisible to
+ `GET /_settings`, and — with `_block` registered PUT-only — removable only by
+ restarting with a hand-edited file. The one block that fires automatically did
+ the inverse of its name, and clients hitting any of it were told the server
+ had faulted.
+
+ - **`llms.txt` no longer tells agents six things the binary does not do.** The
+ "Running an index for a human" section is executed, not just read: an agent
+ follows it literally and then reports the result to its user. Corrected: the
+ job-size line is not the first line printed; `--dry-run` does not write
+ *nothing* (it creates and locks the state dir and appends a journal record);
+ a dry run reuses a frozen plan instead of re-costing the job, so it is not a
+ floor for the real run; exit `1` is the catch-all for every error, not
+ "endpoint unreachable"; `status` needs the same explicit `--state-dir` the run
+ used; and `--workers`/`--pdf-workers` bound the client-side extractor, not the
+ server (with `--pdf-workers` capped to 1–4).
+
+ - **A failed index can now be inspected, deleted and retried without stopping
+ the server** ([#206](https://github.com/xerj-org/xerj/issues/206)). An index
+ directory that refused to open at boot was recorded in a map that only
+ `Engine::health` ever read. Live-reproduced with a corrupt `snapshot.json`:
+ it was absent from `_cat/indices` and `_cluster/state`, `DELETE /{index}`
+ answered `404 index_not_found` and left the bytes on disk, and the only
+ recovery was to stop the server and edit the data directory by hand. A
+ failed index is now a real state: listed as `red` in `_cat/indices`, present
+ in `_cluster/state` as an `UNASSIGNED` primary with `ALLOCATION_FAILED` and
+ the verbatim open error, enumerated with its reason by
+ `GET /_cluster/indices/failed`, reopenable via
+ `POST /_cluster/indices/failed/{name}/_retry` once the cause is fixed, and
+ deletable through the ordinary `DELETE /{index}`. Searches, writes and
+ creates against it return `503 no_shard_available_action_exception` carrying
+ the reason instead of a `404` that claimed it did not exist, while the
+ metadata surfaces — `GET /{index}`, `HEAD /{index}` (the `indices.exists()`
+ every ES client calls), `_mapping`, `_settings`, `_cat/indices` — report it
+ as an index that exists, the way ES answers those from cluster metadata
+ rather than from a shard. Before this, `HEAD` said the name was free and the
+ following `PUT` said it was unavailable.
+
+ Two related defects in the same report are fixed with it. **Readiness no
+ longer hard-fails on a partly degraded node** — `/health/ready` returned
+ `503` for any red status, so one broken index pulled a pod holding 200
+ healthy ones out of service permanently; it now reports `200 ready
+ (degraded)` while the node can still serve something and `503` only when
+ every index it holds failed to open. And **`_cat/health` no longer prints a
+ hardcoded `green`** — the one health surface an existing ES dashboard points
+ at was the one that could not report a broken node; it and `_cluster/health`
+ now count an unopenable index as an unassigned primary, agreeing with
+ `/v1/health` and `/v1/cluster/health`.
+
+ Because `red` is now reachable on `_cluster/health` for the first time, the
+ two conditions that gate on it are consulted rather than assumed:
+ `wait_for_active_shards=all` and `wait_for_status=green|yellow` both answer
+ `408` with `timed_out: true` on a red node instead of `200 timed_out: false`.
+ `GET /_cluster/health?wait_for_status=green&timeout=30s` is the bootstrap gate
+ every docker healthcheck, CI wait loop and Kibana startup uses; a red node
+ used to sail straight through it. A green or yellow cluster is unaffected.
+
+ - **A `DELETE /{index}` that could not remove the bytes no longer strands the
+ index.** The open-index path pulled the handle out of the engine before
+ `remove_dir_all`, so a removal that failed (read-only mount, `EACCES`) freed
+ the name while the directory survived: no handle, no failed-index entry,
+ nothing on `_cat/indices`, and the next `DELETE` answered `404` — the same
+ dead end as [#206](https://github.com/xerj-org/xerj/issues/206), reached from
+ the other side. The handle is now restored on failure, so the index stays in
+ service and addressable and the operator can retry the delete once the cause
+ is fixed.
+
- **`autoindex` no longer leaves immortal catalog entries for skipped files**
([#238](https://github.com/xerj-org/xerj/issues/238)). A file added after the
resume plan was frozen is skipped and reported in the catalog — but that
diff --cc engine/README.md
index b0c7bb07d,fde49a58f..0ff189684
--- a/engine/README.md
+++ b/engine/README.md
@@@ -53,7 -53,7 +53,7 @@@ HTTP wire payloads, single-node, tmpfs
| Per-batch (5K) bulk p99 | **34.8 ms** | 80.3 ms | ES 2.3× |
| Data loss on restart | 0% | 0% | tied |
| GC pauses | 20–40 ms (JVM) | **0** (no GC) | **xerj** |
- | Config surface | 3,000+ knobs | **103**, all optional | **xerj** |
-| Config surface | 3,000+ knobs | **38** | **xerj** |
++| Config surface | 3,000+ knobs | **104**, all optional | **xerj** |
xerj wins on every operational and read-path metric. ES still wins
bulk-ingest throughput on warm Lucene `IndexWriter` (perf-backlog item
@@@ -81,9 -81,7 +81,9 @@@ The resulting binary is at `target/rele
## Configuration
-See [`xerj.default.toml`](xerj.default.toml) for all 38 settings with documentation.
+See [`xerj.default.toml`](xerj.default.toml) for the commonly-tuned settings, and
- [`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 103
++[`crates/xerj-common/src/config.rs`](crates/xerj-common/src/config.rs) for all 104
+— every one of them optional.
The minimal config to get started:
diff --cc engine/crates/xerj-common/src/config.rs
index ef0a33329,5ed2bac21..482b32af6
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@@ -1,14 -1,8 +1,14 @@@
//! xerj configuration system.
//!
- //! Configuration is intentionally minimal: **103 settings** versus
-//! Configuration is intentionally minimal: **38 settings** versus Elasticsearch's
-//! 3000+. Every option is named, documented, and has a sensible production-ready
-//! default. The format is TOML, loaded from a single file.
++//! Configuration is intentionally minimal: **104 settings** versus
+//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible
+//! production-ready default. The format is TOML, loaded from a single file.
+//!
+//! That number is measured, not asserted: `journey_zero_config` in
+//! `xerj-engine/tests/product_experience.rs` counts the leaf keys of a default
+//! `Config` and fails when it drifts. It used to say 38 here, 56 twenty lines
+//! down and 60 in the test — three figures, none of them the truth, because the
+//! test re-added a hardcoded sum instead of counting anything (#207).
//!
//! ## Quick start
//!
@@@ -92,10 -86,8 +92,10 @@@ pub struct Config
pub compat: CompatConfig,
}
- // 19 sub-configs, 103 leaf settings in total. Do not maintain that sum by hand
-// Total: 5+3+2+3+10+5+3+1+6+2+4+3+4+3+2 = 56 fields (incl. cors: 2, auth: 3,
-// logging: 2). `Default` is derived — every field is a sub-config that
++// 19 sub-configs, 104 leaf settings in total. Do not maintain that sum by hand
+// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a
+// serialised `Config::default()` and fails if this comment and the module
+// header stop matching. `Default` is derived: every field is a sub-config that
// implements `Default`, so the derive produces exactly the same all-defaults
// value the manual impl used to build by hand.
@@@ -329,8 -371,7 +392,8 @@@ impl Config
}
// ═════════════════════════════════════════════════════════════════════════════
- // Sub-configs (103 user-facing settings total; counted by
-// Sub-configs (38 user-facing settings total)
++// Sub-configs (104 user-facing settings total; counted by
+// `journey_zero_config`, not by hand)
// ═════════════════════════════════════════════════════════════════════════════
/// Network and data-directory settings.
@@@ -1670,305 -1697,6 +1787,305 @@@ mod tests
.expect("shipped xerj.default.toml must pass Config::validate");
}
+ /// Flatten a serialised config to `("merge.max_segment_mb", 8192)` pairs.
+ fn flatten(prefix: &str, value: &serde_json::Value, out: &mut Vec<(String, String)>) {
+ match value {
+ serde_json::Value::Object(fields) => {
+ for (name, v) in fields {
+ let path = if prefix.is_empty() {
+ name.clone()
+ } else {
+ format!("{prefix}.{name}")
+ };
+ flatten(&path, v, out);
+ }
+ }
+ other => out.push((prefix.to_string(), other.to_string())),
+ }
+ }
+
+ /// The shipped file opens with "documents … their production-ready
+ /// defaults", so every value it sets must BE the default — otherwise
+ /// `cp xerj.default.toml xerj.toml`, the documented first step, silently
+ /// changes the engine's behaviour and the file becomes a second, divergent
+ /// source of truth for every number it quotes.
+ ///
+ /// It had already drifted when #207 was filed: the file said
+ /// `max_segment_mb = 5120` against a real default of 8192, so copying it
+ /// shrank the largest mergeable segment by 37% without saying so.
+ #[test]
+ fn shipped_default_config_documents_the_real_defaults() {
+ let toml_src = include_str!("../../../xerj.default.toml");
+ let from_file = serde_json::to_value(Config::from_toml_str(toml_src).unwrap()).unwrap();
+ let defaults = serde_json::to_value(Config::default()).unwrap();
+
+ let (mut a, mut b) = (Vec::new(), Vec::new());
+ flatten("", &from_file, &mut a);
+ flatten("", &defaults, &mut b);
+
+ let drift: Vec<String> = a
+ .iter()
+ .zip(b.iter())
+ .filter(|((_, file), (_, code))| file != code)
+ .map(|((key, file), (_, code))| format!("{key}: file says {file}, code says {code}"))
+ .collect();
+ assert!(
+ drift.is_empty(),
+ "engine/xerj.default.toml claims to document the shipped defaults but \
+ disagrees with Config::default() on {} setting(s):\n {}",
+ drift.len(),
+ drift.join("\n ")
+ );
+
- // …and the file's own header quotes how many of the 103 it sets. That
++ // …and the file's own header quotes how many of the 104 it sets. That
+ // number was 38, then 56, and never once the truth (#207), so count the
+ // assignments instead of trusting the sentence.
+ let set_here = toml_src
+ .lines()
+ .filter(|l| {
+ let l = l.trim_start();
+ !l.starts_with('#')
+ && l.split_once('=')
+ .is_some_and(|(k, _)| !k.trim().is_empty() && !k.contains('['))
+ })
+ .count();
+ let total: usize = SETTINGS_BY_SECTION.iter().map(|(_, c)| c).sum();
+ let claim = format!("sets {set_here} of XERJ's {total} user-facing settings");
+ assert!(
+ toml_src.contains(&claim),
+ "xerj.default.toml's header must say {claim:?} — it sets {set_here} keys"
+ );
+ }
+
+ /// Undo the four HTML entities the docs pages actually use.
+ fn html_unescape(s: &str) -> String {
+ s.replace("&quot;", "\"")
+ .replace("&#39;", "'")
+ .replace("&lt;", "<")
+ .replace("&gt;", ">")
+ .replace("&amp;", "&")
+ }
+
+ /// Render a serialised default the way `config.html` writes it: strings
+ /// quoted, empty arrays as `[]`, numbers and bools bare.
+ fn as_documented(value: &serde_json::Value) -> String {
+ match value {
+ serde_json::Value::String(s) => format!("\"{s}\""),
+ serde_json::Value::Array(items) if items.is_empty() => "[]".to_string(),
+ other => other.to_string(),
+ }
+ }
+
+ /// The reference table on `landing/docs/config.html`, as
+ /// `(section, key, documented default)` in page order.
+ fn docs_default_table(page: &str) -> Vec<(String, String, String)> {
+ const GROUP: &str = "<div class=\"group\" id=\"";
+ const CELL_K: &str = "<div class=\"cell k\">";
+ const CELL_D: &str = "<div class=\"cell d\">";
+
+ let mut rows = Vec::new();
+ let mut section = String::new();
+ let mut rest = page;
+ loop {
+ let group_at = rest.find(GROUP);
+ let key_at = rest.find(CELL_K);
+ let take_group = match (group_at, key_at) {
+ (Some(g), Some(k)) => g < k,
+ (Some(_), None) => true,
+ _ => false,
+ };
+ if take_group {
+ let after = &rest[group_at.unwrap() + GROUP.len()..];
+ let (id, tail) = after.split_once('"').expect("group id must be quoted");
+ section = id.to_string();
+ rest = tail;
+ } else if let Some(k) = key_at {
+ let after = &rest[k + CELL_K.len()..];
+ let (key, tail) = after.split_once("</div>").expect("cell k must close");
+ let d = tail
+ .find(CELL_D)
+ .expect("every key cell has a default cell");
+ let (default, tail) = tail[d + CELL_D.len()..]
+ .split_once("</div>")
+ .expect("cell d must close");
+ rows.push((
+ section.clone(),
+ key.trim().to_string(),
+ html_unescape(default).trim().to_string(),
+ ));
+ rest = tail;
+ } else {
+ return rows;
+ }
+ }
+ }
+
+ /// Split `"batched" # durability knob` into value and comment, without
+ /// mistaking a `#` inside a quoted string for the comment marker.
+ fn split_value_and_comment(rhs: &str) -> (&str, &str) {
+ let rhs = rhs.trim();
+ let end = if let Some(inner) = rhs.strip_prefix('"') {
+ inner.find('"').map(|i| i + 2).unwrap_or(rhs.len())
+ } else {
+ rhs.find('#').unwrap_or(rhs.len())
+ };
+ let (value, comment) = rhs.split_at(end);
+ (value.trim(), comment.trim())
+ }
+
+ /// The docs site is the *other* copy of the defaults, and it drifted
+ /// exactly the way `xerj.default.toml` did (#207).
+ ///
+ /// `landing/docs/config.html` shipped `wal_max_size_mb = 512`,
+ /// `flush_size_mb = 256` and `default_quantization = "scalar8"` in its
+ /// copy-pasteable `[storage]` and `[vector]` blocks while its own DEFAULT
+ /// table — thirty lines above, on the same page — said 1024, 512 and
+ /// `"none"`. An operator pasting the `[vector]` block switched on 8-bit
+ /// quantization and its 1–2% recall loss without asking for it.
+ /// `shipped_default_config_documents_the_real_defaults` did not catch it
+ /// because it only reads the TOML, so read the page too:
+ ///
+ /// 1. every DEFAULT cell in the reference table must equal
+ /// `Config::default()`, and
+ /// 2. every assignment in an example block must equal the real default —
+ /// unless its comment says `not a default`, which is how the blocks
+ /// that exist to *turn something on* (TLS, cluster, embedding) declare
+ /// themselves to the reader as well as to this test.
+ #[test]
+ fn the_docs_site_config_page_agrees_with_the_real_defaults() {
+ let page = include_str!("../../../../landing/docs/config.html");
+ let defaults = serde_json::to_value(Config::default()).unwrap();
+ let table = docs_default_table(page);
+ assert!(
+ table.len() > 40,
+ "only parsed {} rows out of the config.html reference table — the page's \
+ markup changed and this guard is no longer reading it",
+ table.len()
+ );
+
+ let mut drift: Vec<String> = Vec::new();
+ for (section, key, documented) in &table {
+ match defaults.get(section).and_then(|s| s.get(key)) {
+ None => drift.push(format!(
+ "table: [{section}] {key} is documented but is not a key in Config"
+ )),
+ Some(actual) => {
+ let real = as_documented(actual);
+ if &real != documented {
+ drift.push(format!(
+ "table: [{section}] {key} says {documented}, code says {real}"
+ ));
+ }
+ }
+ }
+ }
+
+ for block in page.split("<pre class=\"code\">").skip(1) {
+ let block = html_unescape(block.split("</pre>").next().unwrap_or_default());
+ let mut section = String::new();
+ for line in block.lines() {
+ let line = line.trim();
+ if line.starts_with('#') {
+ continue;
+ }
+ if let Some(name) = line.strip_prefix('[').and_then(|l| l.strip_suffix(']')) {
+ section = name.to_string();
+ continue;
+ }
+ let Some((key, rhs)) = line.split_once('=') else {
+ continue;
+ };
+ let key = key.trim();
+ if section.is_empty()
+ || key.is_empty()
+ || !key
+ .bytes()
+ .all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'_')
+ {
+ continue;
+ }
+ let Some(actual) = defaults.get(&section).and_then(|s| s.get(key)) else {
+ drift.push(format!(
+ "example: [{section}] {key} is not a key in Config — a reader pasting \
+ this block gets `unknown field` and no server"
+ ));
+ continue;
+ };
+ let (value, comment) = split_value_and_comment(rhs);
+ let real = as_documented(actual);
+ if value != real && !comment.contains("not a default") {
+ drift.push(format!(
+ "example: [{section}] {key} = {value} contradicts the real default \
+ {real} and does not say so — add `# not a default ({real})` to the \
+ line if the example means it"
+ ));
+ }
+ }
+ }
+
+ assert!(
+ drift.is_empty(),
+ "landing/docs/config.html disagrees with Config::default() in {} place(s):\n {}",
+ drift.len(),
+ drift.join("\n ")
+ );
+ }
+
+ /// `merge.strategy = "log_structured"` used to parse, validate and then run
+ /// size-tiered merging anyway — nothing in the tree reads the field (#207).
+ /// An operator who picks a levelled policy for its read amplification must
+ /// hear that it does not exist, not get the other one silently.
+ #[test]
+ fn the_unimplemented_merge_strategy_is_rejected_not_silently_substituted() {
+ let err = Config::from_toml_str("[merge]\nstrategy = \"log_structured\"\n")
+ .expect_err("log_structured must be refused while it is unimplemented")
+ .to_string();
+ assert!(
+ err.contains("merge.strategy") && err.contains("not implemented"),
+ "the error must name the setting and say why: {err}"
+ );
+
+ Config::from_toml_str("[merge]\nstrategy = \"size_tiered\"\n")
+ .expect("the implemented policy must still be accepted");
+ }
+
+ /// The three merge knobs this build accepts without acting on. Silence was
+ /// the bug (#207): the operator throttling merges to protect query latency
+ /// got no throttle and no signal.
+ #[test]
+ fn dormant_merge_settings_are_named_only_when_an_operator_sets_them() {
+ assert!(
+ MergeConfig::default().dormant_overrides().is_empty(),
+ "an untouched default asks for nothing, so it must not warn"
+ );
+
+ for (toml, expected) in [
+ ("io_rate_mb_per_sec = 250", "merge.io_rate_mb_per_sec"),
+ ("min_segments = 4", "merge.min_segments"),
+ ("max_concurrent = 4", "merge.max_concurrent"),
+ ] {
+ let cfg = Config::from_toml_str(&format!("[merge]\n{toml}\n")).unwrap();
+ let named: Vec<&str> = cfg
+ .merge
+ .dormant_overrides()
+ .into_iter()
+ .map(|(key, _)| key)
+ .collect();
+ assert_eq!(
+ named,
+ vec![expected],
+ "setting {toml} must be reported, and nothing else"
+ );
+ }
+
+ // A setting that IS read by `run_merge_once` must never be reported.
+ let cfg = Config::from_toml_str("[merge]\nmax_segment_mb = 4096\n").unwrap();
+ assert!(
+ cfg.merge.dormant_overrides().is_empty(),
+ "max_segment_mb reaches the merge policy, so it is not dormant"
+ );
+ }
+
#[test]
fn onnx_throughput_controls_preserve_single_session_defaults() {
let cfg = Config::from_toml_str("[embedding]\n").unwrap();
@@@ -2035,87 -1763,34 +2152,87 @@@
);
}
+ /// Leaf keys in a serialised value — one per setting a user can set.
+ /// Sub-configs are grouping, not settings, so only leaves count.
+ fn count_settings(value: &serde_json::Value) -> usize {
+ match value {
+ serde_json::Value::Object(fields) => fields.values().map(count_settings).sum(),
+ _ => 1,
+ }
+ }
+
+ /// Settings per section, and the total. Every one of these numbers is
+ /// quoted in a doc comment above the matching struct (and the total in the
+ /// module header, in `xerj-common/src/lib.rs`, in `engine/README.md` and in
+ /// `xerj.default.toml`) — change the table and the prose together.
+ const SETTINGS_BY_SECTION: &[(&str, usize)] = &[
- ("server", 6),
++ ("server", 7),
+ ("auth", 3),
+ ("cors", 2),
+ ("tls", 4),
+ ("storage", 10),
+ ("merge", 8),
+ ("compression", 3),
+ ("fts", 1),
+ ("vector", 6),
+ ("logs", 2),
+ ("embedding", 19),
+ ("limits", 13),
+ ("indexing", 3),
+ ("engine", 4),
+ ("cluster", 5),
+ ("pit", 3),
+ ("search_context", 7),
+ ("logging", 2),
+ ("compat", 2),
+ ];
+
+ /// Count the settings by *counting them*.
+ ///
+ /// This test used to read
+ ///
+ /// ```ignore
+ /// let total = 6 + 3 + 4 + 10 + 5 + 3 + 1 + 6 + 2 + 4 + 12 + 3 + 2;
+ /// assert_eq!(total, 61);
+ /// ```
+ ///
+ /// — the identity `61 == 61`, which could not fail whatever `Config`
+ /// actually held. By the time #207 was filed it had drifted so far that the
+ /// tree quoted 38, 50, 56, 60 and 61 in five places, none of them right.
+ /// `Config` is `Serialize` and every field defaults, so the honest count is
+ /// the number of leaf keys in a default config.
#[test]
fn count_user_facing_settings() {
- // 50 user-facing settings:
- // server: 7 (rest_port, grpc_port, es_compat_port, data_dir,
- // bind_address, allow_insecure_network_bind, ← #228
- // trusted_proxies)
- // auth: 3 (enabled, admin_api_key, metrics_token) ← RC4-W4 item 4
- // tls: 4 (enabled, cert_path, key_path,
- // allow_insecure_grpc_h2c) ← issue #229
- // storage: 10 (wal_sync, wal_batch_ms, wal_max_size_mb, flush_size_mb,
- // flush_interval_secs, backend, s3_bucket, s3_prefix,
- // s3_region, local_cache_dir)
- // merge: 5 (strategy, min_segments, max_segment_mb, io_rate_mb_per_sec, max_concurrent)
- // compression: 3 (enabled, level, block_size_docs)
- // fts: 1 (default_analyzer)
- // vector: 6 (default_metric, hnsw_m, hnsw_ef_construction, hnsw_ef_search,
- // default_quantization, max_dimensions)
- // logs: 2 (retention_days, time_partition)
- // embedding: 4 (default_endpoint, default_model, batch_size, timeout_ms)
- // limits: 12 (query/concurrency/mapping/bulk-actions/body/result/mget/bucket
- // limits, snapshot repo allowlist, total memtable,
- // segment hydration cache, RSS, disk)
- // indexing: 3 (turbo_batch_size, turbo_parallel, turbo_fast_analyzer)
- // logging: 2 (format, access_log) ← RC4-W4 item 6
- // ─────────
- // total: 62 fields, minus 1 auto-generated (admin_api_key) = 61 meaningful user settings
- let total: usize = 7 + 3 + 4 + 10 + 5 + 3 + 1 + 6 + 2 + 4 + 12 + 3 + 2;
- assert_eq!(total, 62);
+ let cfg = serde_json::to_value(Config::default()).expect("Config serialises");
+ let sections = cfg.as_object().expect("Config is a table");
+
+ let measured: Vec<(String, usize)> = sections
+ .iter()
+ .map(|(name, v)| (name.clone(), count_settings(v)))
+ .collect();
+ let expected: Vec<(String, usize)> = SETTINGS_BY_SECTION
+ .iter()
+ .map(|(n, c)| ((*n).to_string(), *c))
+ .collect();
+ assert_eq!(
+ measured, expected,
+ "the per-section settings counts changed. That is fine — but each is \
+ quoted in the doc comment above its struct, so update both"
+ );
+
+ let total: usize = SETTINGS_BY_SECTION.iter().map(|(_, c)| c).sum();
+ assert_eq!(
+ total,
+ count_settings(&cfg),
+ "the section table must sum to the whole config"
+ );
+ assert_eq!(
- total, 103,
++ total, 104,
+ "the total settings count changed. It is quoted in this module's \
+ header, in xerj-common/src/lib.rs, in engine/README.md, in \
+ xerj.default.toml and in EXPECTED_SETTINGS in \
+ xerj-engine/tests/product_experience.rs — update all of them"
+ );
}
// ── gRPC h2c exposure: fail closed (issue #229) ──────────────────────────
diff --cc engine/crates/xerj-common/src/lib.rs
index 3afbbb258,a8a3d1765..5fbaa904b
--- a/engine/crates/xerj-common/src/lib.rs
+++ b/engine/crates/xerj-common/src/lib.rs
@@@ -7,13 -7,13 +7,13 @@@
//! ## Design philosophy
//!
//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes
- //! **103 settings**, each meaningful and production-tested. Every default
-//! exactly **38 settings**, each meaningful and production-tested. Every default
++//! **104 settings**, each meaningful and production-tested. Every default
//! is chosen so that a fresh deployment with zero configuration changes performs
//! well for the majority of workloads.
//!
//! ## Modules
//!
- //! - [`config`] — TOML-based configuration (103 settings)
-//! - [`config`] — TOML-based configuration (38 settings)
++//! - [`config`] — TOML-based configuration (104 settings)
//! - [`error`] — Unified error type ([`XerjError`])
//! - [`types`] — Core domain types (documents, fields, IDs)
//! - [`schema`] — Index schema management and mapping evolution
diff --cc engine/crates/xerj-engine/tests/product_experience.rs
index d3e6cb6fd,78e00a588..53c464f22
--- a/engine/crates/xerj-engine/tests/product_experience.rs
+++ b/engine/crates/xerj-engine/tests/product_experience.rs
@@@ -739,22 -730,6 +739,22 @@@ async fn journey_upgrade()
// Journey 8: "How many settings do I need to configure?"
// ═════════════════════════════════════════════════════════════════════════════
+/// Leaf keys in a serialised config — one per setting a user can set.
+///
+/// Sub-configs are grouping, not settings, so only leaves count. An empty map
+/// contributes nothing, which is correct: a map-valued setting with no entries
+/// is not a knob anybody has turned.
+fn count_settings(value: &serde_json::Value) -> usize {
+ match value {
+ serde_json::Value::Object(fields) => fields.values().map(count_settings).sum(),
+ _ => 1,
+ }
+}
+
+/// Settings in a default `Config`, measured by `count_settings`. Quoted in
+/// `xerj-common/src/config.rs`; keep the two in step.
- const EXPECTED_SETTINGS: usize = 103;
++const EXPECTED_SETTINGS: usize = 104;
+
#[tokio::test]
async fn journey_zero_config() {
// Start with ZERO configuration.
diff --cc engine/crates/xerj-server/src/main.rs
index 5f36d6b01,c2bb8398f..9c10fdfc1
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@@ -364,15 -432,42 +432,51 @@@ fn load_config(args: &CliArgs) -> Resul
cfg.server.data_dir = dir.clone();
}
+ // Merge settings this build accepts but does not act on. An operator who
+ // throttles merges to protect query latency was getting no throttle and no
+ // signal (#207); the signal is the minimum. See
+ // `MergeConfig::dormant_overrides` for why these are a warning rather than
+ // the hard startup error `storage.backend` gets.
+ for (key, effect) in cfg.merge.dormant_overrides() {
+ warn!("{key} is set but has no effect in this build: {effect}");
+ }
+
+ // Bind address override: `--bind` flag or `XERJ_BIND_ADDRESS` env (flag
+ // wins). Both exist because the default is loopback (#228): without a way
+ // to say "expose me" that is not a TOML file, a container image or a
+ // one-off `xerj -b 0.0.0.0` would have no path at all.
+ if let Some(addr) = args
+ .bind
+ .clone()
+ .or_else(|| std::env::var("XERJ_BIND_ADDRESS").ok())
+ {
+ let addr = addr.trim().to_string();
+ if addr.is_empty() {
+ anyhow::bail!("--bind / XERJ_BIND_ADDRESS is empty; pass an IP address");
+ }
+ info!("server.bind_address = {addr} (from CLI/env)");
+ cfg.server.bind_address = addr;
+ }
+
+ // The #228 opt-out, also settable from the environment so a container can
+ // declare its exposure without shipping a config file. Parsed strictly: a
+ // value we cannot read is refused rather than treated as `false`, because
+ // silently falling back to the safe value here means a boot that refuses
+ // with a message about a setting the operator believes they set.
+ if let Ok(raw) = std::env::var("XERJ_ALLOW_INSECURE_NETWORK_BIND") {
+ let v = raw.trim();
+ let parsed = match v.to_ascii_lowercase().as_str() {
+ "1" | "true" | "yes" | "on" => true,
+ "0" | "false" | "no" | "off" => false,
+ _ => anyhow::bail!(
+ "XERJ_ALLOW_INSECURE_NETWORK_BIND={raw:?} is not a boolean; \
+ use true or false"
+ ),
+ };
+ info!("server.allow_insecure_network_bind = {parsed} (from env)");
+ cfg.server.allow_insecure_network_bind = parsed;
+ }
+
if args.insecure {
warn!("--insecure: TLS and auth disabled");
cfg.tls.enabled = false;
diff --cc engine/tests/battle_sla.sh
index 8b2ca29b4,78a3e725e..e4c80a082
--- a/engine/tests/battle_sla.sh
+++ b/engine/tests/battle_sla.sh
@@@ -624,17 -624,11 +624,17 @@@ log "Scenario: count user-facing confi
log "ES comparison: elasticsearch.yml has 3000+ documented properties"
log ""
+# The shipped file sets the commonly tuned subset; the engine's full surface is
- # 103 settings, every one optional — measured by `count_user_facing_settings`
++# 104 settings, every one optional — measured by `count_user_facing_settings`
+# (xerj-common/src/config.rs), which counts leaf keys of a serialised
+# `Config::default()`. This bound is on the FILE, and it is deliberately not the
+# published headline: quoting a number nobody counted is what issue #207 was
+# filed about.
USER_SETTINGS=$(grep -v "^#\|^$\|^\[" "$DEFAULT_TOML" | grep "=" | wc -l || echo 0)
- info "Key=value lines in xerj.default.toml: $USER_SETTINGS (of 103 settings in total)"
-info "Total key=value lines in xerj.default.toml: $USER_SETTINGS"
-[ "$USER_SETTINGS" -le 50 ] \
- && pass "Config has $USER_SETTINGS settings (Elasticsearch has 3000+)" \
- || fail "Config has $USER_SETTINGS settings (expected ≤ 50)"
++info "Key=value lines in xerj.default.toml: $USER_SETTINGS (of 104 settings in total)"
+[ "$USER_SETTINGS" -le 60 ] \
- && pass "The shipped config sets $USER_SETTINGS keys; XERJ has 103 settings in total, all optional (Elasticsearch has 3000+)" \
++ && pass "The shipped config sets $USER_SETTINGS keys; XERJ has 104 settings in total, all optional (Elasticsearch has 3000+)" \
+ || fail "The shipped config sets $USER_SETTINGS keys (expected ≤ 60)"
# Verify every setting has a comment above it
COMMENT_RATIO=$(grep -c "^#" "$DEFAULT_TOML" || echo 0)
diff --cc engine/xerj.default.toml
index 9e47527ca,bc640720b..e499ce40e
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@@ -1,11 -1,8 +1,11 @@@
# xerj Default Configuration
# ============================================================
- # This file sets 53 of XERJ's 103 user-facing settings to their
-# This file documents all 38 user-facing settings with their
-# production-ready defaults. Uncomment and edit any section
-# you want to override.
++# This file sets 54 of XERJ's 104 user-facing settings to their
+# production-ready defaults, and describes the rest in comments.
+# Every value here IS the default, so copying this file changes
+# nothing until you edit it (a test enforces that). The complete
+# set — every one of them optional — is documented in
+# crates/xerj-common/src/config.rs.
#
# To use: copy this file to xerj.toml and start the server:
# xerj --config /path/to/xerj.toml
diff --cc landing/docs/config.html
index b466d71d5,e72c36821..6876a0ac9
--- a/landing/docs/config.html
+++ b/landing/docs/config.html
@@@ -124,7 -124,7 +124,7 @@@
<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p>
<h2>Full key table</h2>
- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"0.0.0.0"</div><div class="cell dsc">Interface to bind every listener to. Use "127.0.0.1" for dev or a private address in production.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div>
- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">256</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">"size_tiered" or "log_structured". Size-tiered for write-heavy, log-structured for mixed workloads.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Minimum live segments before a merge pass is scheduled. Must be ≥ 2.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">5120</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments larger than this are never merged.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Merge I/O throttle (MiB/s). Set 0 to disable — not recommended in production.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Concurrent merge workers per index. Bump to 2–4 on fast NVMe.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"scalar8"</div><div class="cell dsc">"none" · "scalar8" · "scalar4" · "binary". scalar8 = 4× RAM saving, 1–2% recall loss.</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div>
++ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div>
<h2 id="server-section">[server]</h2>
@@@ -133,8 -133,10 +133,10 @@@
rest_port = 8080 # native /v1/* API
es_compat_port = 9200 # ES wire-compatible API
grpc_port = 8081 # reserved
- bind_address = "0.0.0.0" # listen on all interfaces
-bind_address = "0.0.0.0" # default is "127.0.0.1" (loopback only)
-allow_insecure_network_bind = true # required for a non-loopback bind while TLS is off
-data_dir = "/var/lib/xerj" # absolute path is strongly recommended</pre>
++bind_address = "0.0.0.0" # not a default — the default is "127.0.0.1" (loopback only)
++allow_insecure_network_bind = true # not a default (false) — required for a non-loopback bind while TLS is off
+data_dir = "/var/lib/xerj" # not a default ("./data") — an absolute path is strongly recommended</pre>
+ <p>The default bind is loopback, so an unconfigured node is not reachable from the network. Exposing it while <code class="inline">tls.enabled = false</code> refuses to start unless <code class="inline">allow_insecure_network_bind</code> says the cleartext exposure is intended — enable TLS and neither line is needed.</p>
<h2 id="auth-section">[auth]</h2>
<p>Static API-key authentication. The first-run admin key is written to <code class="inline">&lt;data_dir&gt;/admin.key</code>; subsequent starts reuse it. Clients pass <code class="inline">Authorization: ApiKey &lt;key&gt;</code> on every request.</p>
diff --cc landing/openapi.json
index 1905c1a5c,b52f68a01..4fcf1d9a9
--- a/landing/openapi.json
+++ b/landing/openapi.json
@@@ -1,10 -1,10 +1,10 @@@
{
"openapi": "3.1.0",
"info": {
- "title": "XERJ REST API (Elasticsearch-compatible)",
- "version": "1.0",
+ "title": "XERJ REST API \u2014 agent core (partial spec)",
+ "version": "0.1-partial",
"summary": "Core index, ingest, search, and agent-memory endpoints for XERJ.",
- "description": "XERJ is an Elasticsearch-compatible search engine written in Rust. This document covers the core REST surface that backs the six canonical agent operations: index creation, document ingest (single and bulk), search (lexical, semantic, vector/kNN, and hybrid), and namespaced agent memory (store + recall). The search endpoint is a single verb whose behaviour is selected by the request body: a `semantic` query for meaning-based recall, a top-level `knn` clause for vector search, or a `hybrid` query for score fusion. Semantic and semantic-memory features use XERJ's built-in embedder and require no external embedding provider or API key. Unfiltered vector (kNN) search is served by a persisted HNSW graph (approximate) with exact rescoring of candidates — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976; num_candidates sets the beam width (floored at 800). Filtered kNN and other ineligible shapes (non-cosine, SQ8, indexes under 1,024 docs) run an exact brute-force scan (recall 1.00). SCOPE: this document is deliberately partial. It specifies the 7 routes behind those six operations, not XERJ's REST surface, which is roughly 210 routes wide. The version above describes this document, not the engine \u2014 read an endpoint's absence here as \"not written up yet\", never as \"not supported\". The complete wire contract is the one the ES-compat YAML conformance suite exercises (engine/tests/es-compat-yaml).",
- "description": "XERJ is an Elasticsearch-compatible search engine written in Rust. This document covers the core REST surface that backs the six canonical agent operations: index creation, document ingest (single and bulk), search (lexical, semantic, vector/kNN, and hybrid), and namespaced agent memory (store + recall). The search endpoint is a single verb whose behaviour is selected by the request body: a `semantic` query for meaning-based recall, a top-level `knn` clause for vector search, or a `hybrid` query for score fusion. Semantic and semantic-memory features use XERJ's built-in embedder and require no external embedding provider or API key. Unfiltered vector (kNN) search is served by a persisted HNSW graph (approximate) with exact rescoring of candidates — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976; num_candidates sets the beam width (floored at 800). Filtered kNN and other ineligible shapes (non-cosine, SQ8, indexes under 1,024 docs) run an exact brute-force scan (recall 1.00).",
++ "description": "XERJ is an Elasticsearch-compatible search engine written in Rust. This document covers the core REST surface that backs the six canonical agent operations: index creation, document ingest (single and bulk), search (lexical, semantic, vector/kNN, and hybrid), and namespaced agent memory (store + recall). The search endpoint is a single verb whose behaviour is selected by the request body: a `semantic` query for meaning-based recall, a top-level `knn` clause for vector search, or a `hybrid` query for score fusion. Semantic and semantic-memory features use XERJ's built-in embedder and require no external embedding provider or API key. Unfiltered vector (kNN) search is served by a persisted HNSW graph (approximate) with exact rescoring of candidates — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976; num_candidates sets the beam width (floored at 800). Filtered kNN and other ineligible shapes (non-cosine, SQ8, indexes under 1,024 docs) run an exact brute-force scan (recall 1.00). SCOPE: this document is deliberately partial. It specifies the 7 routes behind those six operations, not XERJ's REST surface, which is more than 200 routes wide. The version above describes this document, not the engine \u2014 read an endpoint's absence here as \"not written up yet\", never as \"not supported\". The complete wire contract is the one the ES-compat YAML conformance suite exercises (engine/tests/es-compat-yaml).",
"contact": { "name": "XERJ", "url": "https://xerj.org" },
"license": { "name": "Apache-2.0", "identifier": "Apache-2.0" }
},
diff --cc user-feedback/01-operational-complexity/cluster-management.md
index 41804b8a4,41cbf2d73..381a487d1
--- a/user-feedback/01-operational-complexity/cluster-management.md
+++ b/user-feedback/01-operational-complexity/cluster-management.md
@@@ -60,7 -60,7 +60,7 @@@ The #1 complaint across all review plat
- Single-node architecture in M1 eliminates all cluster coordination
- No master election, no shard allocation, no cluster state broadcast
- Single binary deployment: `./xerj --config config.toml`
- - 103 configuration settings, every one optional (ES has 3,000+)
-- < 50 configuration settings (ES has 3,000+)
++- 104 configuration settings, every one optional (ES has 3,000+)
- No dedicated specialists needed for basic operation
## Sources
diff --cc user-feedback/01-operational-complexity/community-horror-stories.md
index 30c00b2da,a69f8886e..16d9664ee
--- a/user-feedback/01-operational-complexity/community-horror-stories.md
+++ b/user-feedback/01-operational-complexity/community-horror-stories.md
@@@ -79,7 -79,7 +79,7 @@@ A team spent 14 hours manually relocati
Every one of these complaints traces to one or more of:
1. JVM/GC → Rust eliminates entirely
2. Cluster coordination → single-node M1 eliminates entirely
- 3. Configuration sprawl → 103 settings, all optional, with sensible defaults
-3. Configuration sprawl → <50 settings with sensible defaults
++3. Configuration sprawl → 104 settings, all optional, with sensible defaults
4. Shard management → no shards, automatic segment management
5. Upgrade complexity → replace binary + restart
diff --cc user-feedback/01-operational-complexity/learning-curve.md
index 4d07b7ca1,8fa374b7a..d556b0aa9
--- a/user-feedback/01-operational-complexity/learning-curve.md
+++ b/user-feedback/01-operational-complexity/learning-curve.md
@@@ -48,7 -48,7 +48,7 @@@ Appears in nearly every negative revie
- Minimal API surface: 10 REST endpoints, not hundreds
- Native REST API with intuitive JSON (not deeply nested DSL)
- ES-compat layer on port 9200 for teams already fluent in ES DSL
- - 103 config settings, every one optional, all with sensible defaults
-- < 50 config settings with sensible defaults
++- 104 config settings, every one optional, all with sensible defaults
- Single TOML config file (not elasticsearch.yml + jvm.options + log4j2.properties)
- No shard/replica/JVM concepts to learn for M1
diff --cc user-feedback/15-clustering-durability-expectations-2025-2026/03-user-expectations-2025-2026.md
index 48cee7116,15548f137..1385afe91
--- a/user-feedback/15-clustering-durability-expectations-2025-2026/03-user-expectations-2025-2026.md
+++ b/user-feedback/15-clustering-durability-expectations-2025-2026/03-user-expectations-2025-2026.md
@@@ -104,7 -104,7 +104,7 @@@ Based on all user feedback collected
5. **Instant write visibility** — no refresh delay
6. **AI-native** — text + vector + hybrid in one query
7. **WASM pipelines** — user-defined transforms at ingest speed
- 8. **103 settings, not 3,000** — every one optional; works out of the box
-8. **38 settings, not 3,000** — works out of the box
++8. **104 settings, not 3,000** — every one optional; works out of the box
9. **Serverless pricing** — pay per query, not per node
10. **Zero-downtime everything** — upgrades, scaling, schema changes
diff --cc user-feedback/ROADMAP-TO-GA.md
index aa7e7d4b8,54da8951b..dcce32bb9
--- a/user-feedback/ROADMAP-TO-GA.md
+++ b/user-feedback/ROADMAP-TO-GA.md
@@@ -111,10 -111,9 +111,10 @@@ destructive step (engine/crates/xerj-st
**Config sprawl — "3,000+ settings" (01-operational-complexity/learning-curve.md:18-20).** An empty
or absent config file is valid; every setting has a production default; a typo'd key fails startup
(`deny_unknown_fields`); schema-present-but-unwired features are rejected loudly at startup
- (config.rs:42-47, 94-108, 198-229). The true count is 103 serde-visible settings — measured by
-(config.rs:42-47, 94-108, 198-229). The true count is 102 serde-visible settings (field-by-field
-verified) — the published "38"/"<50" figures are wrong and are blocker 10. 102 vs 3,000+ is ~29x
-fewer (PROJECTED: 3000/102), told truthfully.
++(config.rs:42-47, 94-108, 198-229). The true count is 104 serde-visible settings — measured by
+`count_user_facing_settings`, which serialises `Config::default()` and counts leaf keys, not by
+hand. The published "38"/"<50" figures were wrong; they were corrected with blocker 10
- (2026-08-09, #207). 103 vs 3,000+ is ~29x fewer (PROJECTED: 3000/103), told truthfully.
++(2026-08-09, #207). 104 vs 3,000+ is ~29x fewer (PROJECTED: 3000/104), told truthfully.
**Shard sizing expertise (04-scaling-and-shards, cluster-management.md:11).** No up-front shard
decision exists: ES `number_of_shards` is stored and echoed for wire-compat but never used as a
@@@ -429,14 -428,7 +429,15 @@@ Files: engine/crates/xerj-api/src/es_co
migrating users' dashboards silently lose their breaker signal, and the engine's strongest anti-OOM
feature is experienced as a regression.
-### 10. Correct the settings-count claim and make the count drift-proof (small)
+### 10. Correct the settings-count claim and make the count drift-proof (small) — DONE 2026-08-09 (#207)
+
+> Landed as described: the count test now serialises `Config::default()` and counts leaf
- > keys (`journey_zero_config`), the measured total is **103**, and the 38/56/60/`<50`
++> keys (`journey_zero_config`), the measured total was **103** at the time (104 since
++> rc.14 added `server.allow_insecure_network_bind`), and the 38/56/60/`<50`
+> figures are corrected across config.rs, xerj-common/src/lib.rs, engine/README.md,
+> xerj.default.toml and the feedback responses. Per-sub-config annotations are corrected
+> too (limits 3 → 13, storage 5 → 10, cluster 4 → 5, embedding 4 → 19, merge 5 → 8,
+> tls 3 → 4, auth 2 → 3). The adversarial pass counted 102; the measured figure is 103.
**Pain.** The "<50 configuration settings (ES has 3,000+)" figure is published in the shipped
feedback responses (user-feedback/01-operational-complexity/cluster-management.md:63).
@@@ -450,8 -442,8 +451,8 @@@ verifier disputed it to blocking under
doing it before GA, not waiving it. This roadmap adopts the verifier's position.]
**Build.** Fix all inconsistent counts; replace the hand-written count test with introspection
(serialize `Config::default()` to `toml::Value`, count leaf keys) so the number can never drift;
- sweep public docs and the feedback responses for the stale "<50"/"38". 103 vs 3,000+ is still the
-sweep public docs and the feedback responses for the stale "<50"/"38". 102 vs 3,000+ is still the
-winning story, told truthfully. Files: engine/crates/xerj-common/src/config.rs, docs, user-feedback
++sweep public docs and the feedback responses for the stale "<50"/"38". 104 vs 3,000+ is still the
+winning story, told truthfully. [DONE 2026-08-09, #207 — the measured figure is 103.] Files: engine/crates/xerj-common/src/config.rs, docs, user-feedback
response sections.
**If shipped without.** A reviewer counts the TOML keys, finds 2.7x the claimed number, and every
other XERJ number takes the credibility hit.