AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(site): document the built-in neural embedder + correct stale tally & dead links (#492)
xerj-org3d371044beb1
diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex ca822b63e..2a2b2528c 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -404,6 +404,15 @@ timeout_ms = 5000[limits]+# Process-wide memory ceiling (MiB) that bounds every internal budget — segment+# hydration, query working sets, and the ingest circuit breaker. OMIT it (the+# default) to AUTO-size the cap from the machine in binary-GiB tiers+# (8 / 16 / 32 GiB): a laptop stays bounded, a large box climbs. Set an explicit+# MiB ceiling to raise headroom on a big host — e.g. when a large `xerj autoindex`+# trips `429 circuit_breaking_exception`. Set 0 to uncap to the whole machine+# (not recommended). Env override: XERJ_MAX_PROCESS_MEMORY_MB=auto|off|<MiB>.+# max_process_memory_mb = 0+# Maximum memory a single query may allocate for intermediate results (MiB).# Queries that exceed this limit are cancelled with a resource-exceeded error.# Increase for aggregation-heavy workloads; decrease to protect shared nodes.diff --git a/landing/case-studies.html b/landing/case-studies.htmlindex f6ffb9e29..d49059114 100644--- a/landing/case-studies.html+++ b/landing/case-studies.html@@ -246,8 +246,8 @@<div class="uc-copy"><span class="uc-num">05</span><div class="uc-title">SAME MODEL.<br>FEWER<br><span class="accent">TOKENS.</span></div>- <p class="uc-pitch">On code the model has not memorised, handing the <em>same</em> Claude Code a XERJ-retrieved reference makes it correct where memory fails — <strong>15/15 vs 1/15</strong> — at 1.5–2.7× fewer output tokens than native Claude Code grepping the source, across five languages. Red is answering from memory; yellow is XERJ-assisted. On memorised code, retrieval is honest overhead — and we say so.</p>- <div class="uc-metric">15/15 vs 1/15 SOLVED · 6.5× CHEAPER THAN MEMORY</div>+ <p class="uc-pitch">On code the model has not memorised, handing the <em>same</em> Claude Code a XERJ-retrieved reference makes it correct where memory fails — <strong>21/21 vs 1/21</strong> — at 1.5–2.7× fewer output tokens than native Claude Code grepping the source, across five languages. Red is answering from memory; yellow is XERJ-assisted. On memorised code, retrieval is honest overhead — and we say so.</p>+ <div class="uc-metric">21/21 vs 1/21 SOLVED · 6.5× CHEAPER THAN MEMORY</div><div class="uc-persona">AI-AGENT BUILDERS · PLATFORM & DX TEAMS · SYNTHETIC CORPUS</div><a href="/case-studies/reference-coding.html" class="uc-cta">FULL CASE STUDY →</a></div>diff --git a/landing/case-studies/reference-coding.html b/landing/case-studies/reference-coding.htmlindex e1c41f6c2..116307ded 100644--- a/landing/case-studies/reference-coding.html+++ b/landing/case-studies/reference-coding.html@@ -316,7 +316,7 @@ Do this proactively for the rest of the session without me asking again.</pre><h2 class="scene" style="font-size:var(--fs-56);">POINT IT AT <span class="accent">YOUR</span> CODE.</h2><div style="margin-top:var(--sp-6); display:flex; gap:14px; justify-content:center; flex-wrap:wrap;"><a class="btn btn-primary" href="https://github.com/xerj-org/xerj/blob/main/docs/case-studies/reference-coding/CASE_STUDY.md" rel="external">READ THE FULL CASE STUDY →</a>- <a class="btn" href="/get/index.html">GET XERJ</a>+ <a class="btn" href="/docs/install.html">GET XERJ</a></div></section>diff --git a/landing/demo/index.html b/landing/demo/index.htmlindex d493d5aea..85ea51719 100644--- a/landing/demo/index.html+++ b/landing/demo/index.html@@ -283,7 +283,6 @@ footer .links { display: flex; gap: 28px; flex-wrap: wrap; }<a href="#step-4"><span class="num">04</span>Open Xerj Console</a><a href="#step-5"><span class="num">05</span>Find attackers</a><a href="#step-6"><span class="num">06</span>Attack timeline</a>- <a href="#step-6b"><span class="num">06b</span>AI dashboards</a><a href="#step-7"><span class="num">07</span>TCO numbers</a><a href="#step-8"><span class="num">08</span>Wire-protocol</a></nav>@@ -445,7 +444,7 @@ Ready: /home/claude/ai/xerj/engine/demo-data/ssh_big.ndjson</span></pre><code>@timestamp · host · proc · pid · event · user · src_ip · src_port · message</code>.Event tags (<code>auth_failure_invalid_user</code>, <code>possible_break_in</code>,<code>auth_success</code>, …) come from a small regex parser- (<a href="https://github.com/xerj-org/xerj/blob/main/engine/demo-data/build_corpus.py"><code>build_corpus.py</code></a>).+ (<code>build_corpus.py</code>).</p><div class="se-says">diff --git a/landing/docs/config.html b/landing/docs/config.htmlindex de21d36e3..ede231fdc 100644--- a/landing/docs/config.html+++ b/landing/docs/config.html@@ -124,7 +124,7 @@<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p><h2>Full key table</h2>- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, <strong>not</strong> RAM: the scan reads the full-precision vector from <code class="inline">_source</code> and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ's own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index's WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index. Must be at least 1.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target's http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. Range 1–86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node's disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap — the config file must not be a way around the bound.</div></div>+ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, <strong>not</strong> RAM: the scan reads the full-precision vector from <code class="inline">_source</code> and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="cell k">max_process_memory_mb</div><div class="cell t">u64</div><div class="cell d">auto</div><div class="cell dsc">Process-wide memory ceiling (MiB) bounding every internal budget — segment hydration, query working sets, the ingest circuit breaker. Omitted (the default) it AUTO-sizes from the machine in binary-GiB tiers (8 / 16 / 32 GiB): a laptop stays bounded, a big box climbs. An explicit MiB value forces a fixed ceiling (still min'd with the machine) — raise it if a large <code class="inline">xerj autoindex</code> trips <code class="inline">429 circuit_breaking_exception</code>. <code class="inline">0</code> uncaps to the whole machine. Env: <code class="inline">XERJ_MAX_PROCESS_MEMORY_MB=auto|off|<MiB></code>.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ's own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index's WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index. Must be at least 1.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target's http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. Range 1–86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node's disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap — the config file must not be a way around the bound.</div></div><h2 id="server-section">[server]</h2>@@ -218,11 +218,12 @@ timeout_ms = 5000# default_model = "nomic-embed-text"</pre><h2 id="limits-section">[limits]</h2>- <p>Hard caps to protect the server from runaway queries and mapping explosions. Lower these on shared nodes, raise <code class="inline">max_query_memory_mb</code> for aggregation-heavy workloads.</p>+ <p>Hard caps to protect the server from runaway queries and mapping explosions. Lower these on shared nodes, raise <code class="inline">max_query_memory_mb</code> for aggregation-heavy workloads. Leave <code class="inline">max_process_memory_mb</code> out to auto-size the process-wide cap from the machine (8 / 16 / 32 GiB tiers); set an explicit MiB ceiling only to raise headroom on a large host — for example if a big <code class="inline">xerj autoindex</code> trips <code class="inline">429 circuit_breaking_exception</code>.</p><pre class="code">[limits]max_query_memory_mb = 512max_concurrent_searches = 64-max_fields_per_index = 500</pre>+max_fields_per_index = 500+# max_process_memory_mb = 0 # omit → AUTO (8/16/32 GiB by machine); 0 = whole machine</pre><h2 id="indexing-section">[indexing]</h2><p>Turbo mode knobs. Turbo is opt-in per-request via <code class="inline">POST /v1/indices/:name/turbo-ingest</code> or the <code class="inline">X-Turbo: true</code> header on <code class="inline">_bulk</code>; these settings only apply when turbo is active.</p>diff --git a/landing/docs/recipes/semantic-search-rag.html b/landing/docs/recipes/semantic-search-rag.htmlindex b8d5b051f..f262590af 100644--- a/landing/docs/recipes/semantic-search-rag.html+++ b/landing/docs/recipes/semantic-search-rag.html@@ -208,20 +208,32 @@ RESULT: 5/5 semantic retrievals returned the right doc at rank 1</pre>match top -> NO HITSsemantic top -> kb-2 Refund policy</pre>- <h2>Honest limitation: the built-in embedder is <em>lexical</em>, not neural</h2>- <p>The zero-config embedder is a <strong>lexical</strong> model — it maps overlapping and related vocabulary into vector space. It shines when the question and the answer share <em>some</em> terms (a synonym here, a rephrase there), which covers a lot of real support/FAQ traffic. It is <strong>not</strong> a neural sentence embedder and won't capture deep semantics with no lexical bridge.</p>+ <h2>Three embedding backends: lexical (default), built-in neural, or bring-your-own</h2>+ <p>The <code class="inline">semantic_text</code> mapping and the <code class="inline">semantic</code> query above are <strong>identical</strong> across all three backends. You choose the backend once, at the server, with <code class="inline">--embed-mode</code> (or <code class="inline">embedding.mode</code> in config, or <code class="inline">XERJ_EMBED_MODE</code>):</p>+ <ul>+ <li><strong><code class="inline">lexical</code> (default)</strong> — built-in feature-hash embedder. Offline, deterministic, zero download. Maps overlapping and related vocabulary into vector space.</li>+ <li><strong><code class="inline">neural</code></strong> — built-in BERT sentence encoder (<code class="inline">all-MiniLM-L6-v2</code>) running in-process via <code class="inline">candle</code>. Pure Rust, no Python, no external service. The model auto-downloads (~90 MB) once on first use, or point <code class="inline">embedding.local_model_dir</code> at a pre-staged copy for a fully <strong>air-gapped</strong> install.</li>+ <li><strong><code class="inline">proxy</code></strong> — bring your own model: XERJ calls any OpenAI-compatible <code class="inline">/v1/embeddings</code> service (OpenAI, Cohere, a local vLLM). Set <code class="inline">embedding.default_endpoint</code> in config, or point a field at it in the mapping via <code class="inline">inference_endpoint</code> / <code class="inline">inference_id</code>.</li>+ </ul>- <p>Concretely, from this same KB:</p>+ <h3>The default is lexical — know where it stops</h3>+ <p>The zero-config embedder is a <strong>lexical</strong> model. It shines when the question and the answer share <em>some</em> terms (a synonym here, a rephrase there), which covers a lot of real support/FAQ traffic. It is <strong>not</strong> a neural sentence embedder and won't capture deep semantics with no lexical bridge. Concretely, from this same KB:</p><pre class="code">Q: 'how do I rotate my API key'- semantic top -> kb-5 Rate limits (expected kb-1)</pre>-- <p>The doc is <em>titled</em> "Rotating an API key," but only its <strong>body</strong> is embedded, and the body never uses the word "rotate" — so a pure "rotate" paraphrase misses. Two honest takeaways:</p>+ semantic top -> kb-5 Rate limits (expected kb-1) # --embed-mode lexical</pre>+ <p>The doc is <em>titled</em> "Rotating an API key," but only its <strong>body</strong> is embedded, and the body never uses the word "rotate" — so a pure "rotate" paraphrase misses in lexical mode.</p>++ <h3>Upgrade to real neural semantics — still one binary, still in place</h3>+<pre class="code"># Built-in neural embedder — ships in the binary, no external service.+# Downloads all-MiniLM-L6-v2 (~90 MB) once on first use, then runs in-process.+xerj --insecure --data-dir ./data --embed-mode neural</pre>+ <p>With <code class="inline">--embed-mode neural</code>, that "rotate" paraphrase lands on <code class="inline">kb-1</code>: the BERT encoder captures that <em>rotate ≈ regenerate ≈ replace</em> even with no shared term. The mapping and query don't change — only the server flag does.</p>+ <p>Two honest takeaways that apply in every mode:</p><ul><li><strong>Embed the text that carries the meaning.</strong> If titles matter, index them into the <code class="inline">semantic_text</code> field too (or add a second <code class="inline">semantic_text</code> field and combine with <code class="inline">hybrid</code>), so the vocabulary the user might use is actually in the vector.</li>- <li><strong>For production-grade semantics, wire an external embeddings endpoint.</strong> XERJ will call any OpenAI-compatible <code class="inline">/v1/embeddings</code> service. Set <code class="inline">embedding.default_endpoint</code> in config, or point a field at it in the mapping via <code class="inline">inference_endpoint</code> / <code class="inline">inference_id</code>. The <code class="inline">semantic</code> query and the ingest path are identical — you just get neural-quality vectors instead of the built-in lexical ones. (Anomaly-style continuous re-embedding isn't a thing here; embedding happens at ingest time.)</li>+ <li><strong>Bring your own model when you need a specific one.</strong> <code class="inline">--embed-mode proxy</code> with <code class="inline">embedding.default_endpoint</code> reaches any OpenAI-compatible endpoint. Embedding happens at ingest time (there is no continuous re-embedding).</li></ul>- <p>Everything above is <code class="inline">semantic_text</code> + the built-in embedder — no external service was configured for this recipe.</p>+ <p>Everything shown above the "Upgrade" step is <code class="inline">semantic_text</code> + the built-in lexical embedder — no external service was configured for this recipe.</p><hr>diff --git a/landing/docs/vectors.html b/landing/docs/vectors.htmlindex ce9539ef2..5c88c0ca6 100644--- a/landing/docs/vectors.html+++ b/landing/docs/vectors.html@@ -121,6 +121,18 @@<p>XERJ stores your vectors in full at <strong>ingest</strong> time and maintains a <strong>persisted HNSW graph</strong> alongside them. An unfiltered <code class="inline">knn</code> on a full-precision cosine field (≥1,024 docs) is served by an approximate HNSW beam search whose candidates are <strong>exact-rescored</strong>, so returned scores match the exact path bit-for-bit — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937). <code class="inline">num_candidates</code> sets the beam width (floored at 800 to match Elasticsearch's per-segment candidate semantics). A <code class="inline">filter</code>, non-cosine similarity, SQ8 quantization, a small index, or a stale graph falls back to the <strong>exact brute-force</strong> scan: every candidate is scored and the true nearest neighbours returned (recall 1.00; latency scales with vectors scanned). Distance metric and quantization are set in config. See <a href="#knn-query">KNN query</a> below.</p>+ <h2 id="embedders">Embedding backends — how vectors get made</h2>+ <p>Everything above assumes a vector already exists on the doc. XERJ can produce that vector for you from a <code class="inline">semantic_text</code> field, and you pick <em>how</em> once at the server with <code class="inline">--embed-mode</code> (or <code class="inline">embedding.mode</code> in config, or <code class="inline">XERJ_EMBED_MODE</code>). The mapping and the <code class="inline">semantic</code> query are identical across all three:</p>+ <div class="cfg" style="grid-template-columns: 1fr 1.2fr 3fr;">+ <div class="head">MODE</div><div class="head">SHIPS / NEEDS</div><div class="head">WHAT IT IS</div>+ <div class="cell k">lexical</div><div class="cell t">built-in · default</div><div class="cell dsc">Feature-hash embedder. Offline, deterministic, zero download. Maps overlapping and related vocabulary into vector space — strong when question and answer share terms, not a neural sentence encoder.</div>+ <div class="cell k">neural</div><div class="cell t">built-in · ~90 MB on first use</div><div class="cell dsc">BERT sentence encoder (<code class="inline">all-MiniLM-L6-v2</code>) running in-process via <code class="inline">candle</code> — pure Rust, no Python, no external service. The model auto-downloads (~90 MB) once, or point <code class="inline">embedding.local_model_dir</code> at a pre-staged copy for a fully <strong>air-gapped</strong> install. Captures meaning with no shared term (<em>rotate ≈ regenerate</em>).</div>+ <div class="cell k">proxy</div><div class="cell t">external endpoint</div><div class="cell dsc">Bring your own model: XERJ calls any OpenAI-compatible <code class="inline">/v1/embeddings</code> (OpenAI, Cohere, a local vLLM). Set <code class="inline">embedding.default_endpoint</code>. Embedding happens at ingest time.</div>+ </div>+ <p>Turn on the built-in neural embedder with a single server flag — no mapping or query change:</p>+<pre class="code">xerj --insecure --data-dir ./data --embed-mode neural</pre>+ <p>See the <a href="/docs/recipes/semantic-search-rag.html">Semantic search & RAG recipe</a> for the end-to-end walkthrough.</p>+<h2 id="hnsw">hnsw_* config keys — accepted, no effect on search</h2><div class="cfg" style="grid-template-columns: 1.5fr 1fr 1fr 2.5fr;"><div class="head">KEY</div>diff --git a/landing/resources/index.html b/landing/resources/index.htmlindex 1324b8085..353894f2b 100644--- a/landing/resources/index.html+++ b/landing/resources/index.html@@ -181,22 +181,6 @@</div></section>-<!-- BUNDLE -->-<section class="scene-section">- <div class="kicker"><span>BUNDLE</span><span class="dash">·</span><span>THE WHOLE KIT</span></div>- <h2 class="scene">TWELVE PDFS.<br><span class="accent">ONE ZIP.</span></h2>-- <p class="wide">- All 12 briefs above packaged in three folders (anchors ·- use-cases · industries) for evaluation teams. ~2.4 MB total.- </p>-- <a href="/resources/xerj-sales-kit.zip" class="cta">- <span>DOWNLOAD THE FULL KIT · ZIP</span>- <span class="arrow">→</span>- </a>-</section>-<footer class="footer"><span>XERJ.AI · RESOURCES · V1</span><span>TWELVE BRIEFS · THREE LAYERS</span>