Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into pr-1278

xerj-org8d84a71877a9
diff --cc landing/docs/config.html
index a785eac67,ae8c4089f..e2f77823c
--- a/landing/docs/config.html
+++ b/landing/docs/config.html
@@@ -221,10 -221,10 +221,10 @@@
<p>XERJ reads one TOML file. Every key has a production-ready default, so the smallest working config is an empty file. The table below is the full surface; the sections below it walk each group with a runnable example.</p>
- <p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p>
+ <p>The config file is read from <code class="inline">--config /path/to/xerj.toml</code> on the command line, else from the path in <code class="inline">XERJ_CONFIG</code>. Nothing else is searched: with neither set, the built-in defaults are used even when <code class="inline">./xerj.toml</code> or <code class="inline">/etc/xerj/xerj.toml</code> exists.</p>
<h2>Full key table</h2>
- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">gRPC API (<code class="inline">xerj.v1.XerjSearch</code>, the same engine calls as REST, same authorization). Cleartext h2c even when <code class="inline">[tls]</code> is on: off loopback with TLS enabled the node refuses to start unless <code class="inline">tls.allow_insecure_grpc_h2c = true</code>.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. 0 binds an OS-assigned port (logged at startup); it does not disable the listener.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, <strong>not</strong> RAM: the scan reads the full-precision vector from <code class="inline">_source</code> and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list, one "host:port=host:port" entry per node — the label must equal the address, or the node refuses to start (#1171). The local node identifies itself from the entry matching bind_address and cluster.port. Example: ["10.0.0.1:9300=10.0.0.1:9300","10.0.0.2:9300=10.0.0.2:9300","10.0.0.3:9300=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ&#39;s own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index&#39;s WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index. Must be at least 1.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target&#39;s http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. Range 1&ndash;86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node&#39;s disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap &mdash; the config file must not be a way around the bound.</div><div class="group" id="rerank">[rerank]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Allow the <code class="inline">rerank</code> search stage. It is the only search-time feature that sends document text off this node, and it does nothing until a key is set. <strong>false</strong> refuses every rerank request with HTTP 403, whatever the environment holds, and nothing is sent.</div><div class="cell k">api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Rerank provider key. Empty falls back to the TYPESAFE_API_KEY environment variable; a value here wins over the environment. Empty in both places means reranking is not configured and a rerank request gets HTTP 503. Write-only: GET /_xerj/rerank reports whether a key is set and where it came from, never its value. There is no per-request key.</div><div class="cell k">endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Where judged document text is POSTed. Empty falls back to TYPESAFE_ENDPOINT, then to https://api.typesafe.ai/v1/systemone. Must be an absolute http:// or https:// URL with no credentials in it (user:pass@host is refused at startup, whether it came from this file or from TYPESAFE_ENDPOINT): the endpoint is echoed by GET /_xerj/rerank.</div></div>
- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping. <strong>Not wired</strong>: nothing reads this key — a text field uses its own analyzer, else the index's <code class="inline">analysis.analyzer.default</code>, else standard. Setting it away from the default logs a warning at startup.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice. <strong>Not wired</strong>: similarity comes from the field mapping, and a mapping without one gets cosine whatever this says. Setting it away from the default logs a warning at startup.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, <strong>not</strong> RAM: the scan reads the full-precision vector from <code class="inline">_source</code> and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping). <strong>Not wired as a node-wide default</strong>: only the "binary" refusal reads it; quantization comes from each field's <code class="inline">index_options</code> in the mapping. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096. <strong>Not wired</strong>: validated (≥ 1), but no mapping or document is checked against it. Setting it away from the default logs a warning at startup.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Accepted but <strong>not wired</strong>: no retention job runs and nothing deletes log data in this build — delete old indices or documents yourself. Setting it away from the default logs a warning at startup.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning. <strong>Not wired</strong>: log indices are not partitioned by time. Setting it away from the default logs a warning at startup.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for each embedding request attempt (ms, at least 1). A timed-out attempt is retried up to 3 times with backoff before ingest fails with the error.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="cell k">max_result_window</div><div class="cell t">u64</div><div class="cell d">10000</div><div class="cell dsc">Largest <code class="inline">from + size</code> a search may ask for, for any index whose settings carry no <code class="inline">index.max_result_window</code> (that setting wins when present, above or below this). 1–2147483647. Page past it with <code class="inline">search_after</code> or a point-in-time cursor.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency. <strong>Not wired</strong>: a turbo request is indexed as one batch. Setting it away from the default logs a warning at startup.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging. <strong>Not wired</strong>: turbo ingest no longer pre-tokenises, so there is nothing to parallelise. Setting it away from the default logs a warning at startup.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed. <strong>Not wired</strong>: the turbo path ignores it; fields keep their analyzers. Setting it away from the default logs a warning at startup.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ&#39;s own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index&#39;s WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index. Must be at least 1.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target&#39;s http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. Range 1&ndash;86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node&#39;s disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap &mdash; the config file must not be a way around the bound.</div><div class="group" id="rerank">[rerank]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Allow the <code class="inline">rerank</code> search stage. It is the only search-time feature that sends document text off this node, and it does nothing until a key is set. <strong>false</strong> refuses every rerank request with HTTP 403, whatever the environment holds, and nothing is sent.</div><div class="cell k">api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Rerank provider key. Empty falls back to the TYPESAFE_API_KEY environment variable; a value here wins over the environment. Empty in both places means reranking is not configured and a rerank request gets HTTP 503. Write-only: GET /_xerj/rerank reports whether a key is set and where it came from, never its value. There is no per-request key.</div><div class="cell k">endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Where judged document text is POSTed. Empty falls back to TYPESAFE_ENDPOINT, then to https://api.typesafe.ai/v1/systemone. Must be an absolute http:// or https:// URL with no credentials in it (user:pass@host is refused at startup, whether it came from this file or from TYPESAFE_ENDPOINT): the endpoint is echoed by GET /_xerj/rerank.</div></div>
++ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">gRPC API (<code class="inline">xerj.v1.XerjSearch</code>, the same engine calls as REST, same authorization). Cleartext h2c even when <code class="inline">[tls]</code> is on: off loopback with TLS enabled the node refuses to start unless <code class="inline">tls.allow_insecure_grpc_h2c = true</code>.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. 0 binds an OS-assigned port (logged at startup); it does not disable the listener.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping. <strong>Not wired</strong>: nothing reads this key — a text field uses its own analyzer, else the index's <code class="inline">analysis.analyzer.default</code>, else standard. Setting it away from the default logs a warning at startup.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice. <strong>Not wired</strong>: similarity comes from the field mapping, and a mapping without one gets cosine whatever this says. Setting it away from the default logs a warning at startup.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, <strong>not</strong> RAM: the scan reads the full-precision vector from <code class="inline">_source</code> and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping). <strong>Not wired as a node-wide default</strong>: only the "binary" refusal reads it; quantization comes from each field's <code class="inline">index_options</code> in the mapping. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096. <strong>Not wired</strong>: validated (≥ 1), but no mapping or document is checked against it. Setting it away from the default logs a warning at startup.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Accepted but <strong>not wired</strong>: no retention job runs and nothing deletes log data in this build — delete old indices or documents yourself. Setting it away from the default logs a warning at startup.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning. <strong>Not wired</strong>: log indices are not partitioned by time. Setting it away from the default logs a warning at startup.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for each embedding request attempt (ms, at least 1). A timed-out attempt is retried up to 3 times with backoff before ingest fails with the error.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="cell k">max_result_window</div><div class="cell t">u64</div><div class="cell d">10000</div><div class="cell dsc">Largest <code class="inline">from + size</code> a search may ask for, for any index whose settings carry no <code class="inline">index.max_result_window</code> (that setting wins when present, above or below this). 1–2147483647. Page past it with <code class="inline">search_after</code> or a point-in-time cursor.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency. <strong>Not wired</strong>: a turbo request is indexed as one batch. Setting it away from the default logs a warning at startup.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging. <strong>Not wired</strong>: turbo ingest no longer pre-tokenises, so there is nothing to parallelise. Setting it away from the default logs a warning at startup.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed. <strong>Not wired</strong>: the turbo path ignores it; fields keep their analyzers. Setting it away from the default logs a warning at startup.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list, one "host:port=host:port" entry per node — the label must equal the address, or the node refuses to start (#1171). The local node identifies itself from the entry matching bind_address and cluster.port. Example: ["10.0.0.1:9300=10.0.0.1:9300","10.0.0.2:9300=10.0.0.2:9300","10.0.0.3:9300=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ&#39;s own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index&#39;s WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index. Must be at least 1.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target&#39;s http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. Range 1&ndash;86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node&#39;s disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap &mdash; the config file must not be a way around the bound.</div><div class="group" id="rerank">[rerank]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Allow the <code class="inline">rerank</code> search stage. It is the only search-time feature that sends document text off this node, and it does nothing until a key is set. <strong>false</strong> refuses every rerank request with HTTP 403, whatever the environment holds, and nothing is sent.</div><div class="cell k">api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Rerank provider key. Empty falls back to the TYPESAFE_API_KEY environment variable; a value here wins over the environment. Empty in both places means reranking is not configured and a rerank request gets HTTP 503. Write-only: GET /_xerj/rerank reports whether a key is set and where it came from, never its value. There is no per-request key.</div><div class="cell k">endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Where judged document text is POSTed. Empty falls back to TYPESAFE_ENDPOINT, then to https://api.typesafe.ai/v1/systemone. Must be an absolute http:// or https:// URL with no credentials in it (user:pass@host is refused at startup, whether it came from this file or from TYPESAFE_ENDPOINT): the endpoint is echoed by GET /_xerj/rerank.</div></div>
<h2 id="server-section">[server]</h2>
diff --cc landing/docs/troubleshooting.html
index b50b149b6,4b358511e..54d89a811
--- a/landing/docs/troubleshooting.html
+++ b/landing/docs/troubleshooting.html
@@@ -241,12 -241,12 +241,12 @@@ xerj_merge_duration_seconds_sum{index="
<p><strong>Check:</strong></p>
<pre class="code">$ curl -s http://127.0.0.1:8080/v1/metrics | grep memory_usage
xerj_memory_usage_bytes 14200000000 # 14 GB and climbing</pre>
- <p><strong>Fix:</strong> set <code class="inline">[vector] default_quantization = "scalar8"</code> (or opt heavy fields in per-mapping) so vectors are stored as 1-byte-per-dimension codes. Or lower <code class="inline">flush_size_mb</code> so memtables don't grow unbounded.</p>
+ <p><strong>Fix:</strong> lower <code class="inline">flush_size_mb</code> so memtables don't grow unbounded, and set an explicit <code class="inline">[limits] max_process_memory_mb</code> if the automatic cap does not suit the host. Quantization is not a memory fix today: <code class="inline">scalar8</code> (opted into per field with <code class="inline">index_options</code>) shrinks the kNN scoring working set, not resident memory (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>), and the node-wide <code class="inline">[vector] default_quantization</code> key is not wired.</p>
- <h2>Symptom: "WAL replay failed" on restart</h2>
- <p><strong>Likely cause:</strong> the server was killed mid-fsync (power loss, OOM kill). The tail WAL file is torn.</p>
- <p><strong>Check:</strong> first boot log — <code class="inline">"wal replay: truncating torn tail at offset N"</code> is benign (XERJ truncates the torn suffix and continues). A hard <code class="inline">"wal replay: checksum mismatch at offset N, refusing to start"</code> is not.</p>
- <p><strong>Fix:</strong> if truncation worked on its own, nothing to do — the last few seconds of writes are lost but the index is consistent. If it refused to start, run <code class="inline">xerj verify --data-dir /var/lib/xerj --repair-wal</code> which truncates the WAL at the last valid entry.</p>
+ <h2>Symptom: WAL warnings on restart after a crash</h2>
+ <p><strong>Likely cause:</strong> the server was killed mid-write (power loss, OOM kill, a full disk), leaving a torn or corrupt WAL file.</p>
+ <p><strong>Check:</strong> the first boot log. XERJ repairs the WAL on its own and never refuses to start over it: a torn tail is truncated (<code class="inline">WAL open: healed torn tail (crash/ENOSPC leftover) by truncation</code>), a file with mid-file corruption is frozen and appends move to a fresh generation (<code class="inline">WAL open: mid-file corruption detected</code>), and unreadable frames are skipped during replay (<code class="inline">skipping corrupt WAL entry during replay</code>).</p>
+ <p><strong>Fix:</strong> nothing to run — there is no manual WAL repair command. Writes in the torn or corrupt region are lost; everything before it is replayed. If these warnings appear without a crash or power loss, check the disk and file an issue.</p>
<h2>Symptom: "disk full" during a merge</h2>
<p><strong>Likely cause:</strong> no reservation for merge scratch space. A merge of two N-sized segments needs 2N free until the merge completes.</p>