Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

config: name every accepted-but-unread key at startup (DORMANT), and stop documenting them as working

Tom Villanida9c74299818
diff --git a/engine/README.md b/engine/README.md
index dd4fe452e..7fe7e1c7b 100644
--- a/engine/README.md
+++ b/engine/README.md
@@ -378,7 +378,8 @@ limit of 4,096. This accommodates current and next-generation embedding models:
| GPT-4o embeddings | 3,072 |
| Custom / future models | ≤ 16,384 |
-Set `max_dimensions` in `xerj.toml` (default is 16,384).
+There is no enforced dimension cap: `[vector] max_dimensions` (default 16,384) is accepted
+but not wired in this build, and setting it logs a warning at startup.
## Embedding Token Limits
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 0c58f6e9e..b1287cf27 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -112,6 +112,21 @@ pub struct Config {
// value the manual impl used to build by hand.
impl Config {
+ /// Every setting this build accepts but does not act on, across all
+ /// sections, in the [`MergeConfig::dormant_overrides`] shape — named only
+ /// when the operator moved one off its default. The server logs each one
+ /// at WARN on startup (the accepted-and-ignored class, #204).
+ pub fn dormant_overrides(&self) -> Vec<(&'static str, &'static str)> {
+ let mut out = self.storage.dormant_overrides();
+ out.extend(self.merge.dormant_overrides());
+ out.extend(self.compression.dormant_overrides());
+ out.extend(self.fts.dormant_overrides());
+ out.extend(self.vector.dormant_overrides());
+ out.extend(self.logs.dormant_overrides());
+ out.extend(self.indexing.dormant_overrides());
+ out
+ }
+
/// Load configuration from a TOML file.
///
/// Missing keys fall back to their `Default` values, so a minimal config
@@ -904,6 +919,23 @@ impl Default for StorageConfig {
}
}
+impl StorageConfig {
+ /// Storage settings this build accepts but does not act on, in the
+ /// [`MergeConfig::dormant_overrides`] shape. `local_cache_dir` has been
+ /// dormant since #965 (see its field docs) without being reported.
+ pub fn dormant_overrides(&self) -> Vec<(&'static str, &'static str)> {
+ let mut out = Vec::new();
+ if self.local_cache_dir != StorageConfig::default().local_cache_dir {
+ out.push((
+ "storage.local_cache_dir",
+ "nothing reads it on the index path — segment families \
+ materialize in data_dir/segments (#965)",
+ ));
+ }
+ out
+ }
+}
+
/// Storage backend selector.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
@@ -1217,6 +1249,12 @@ pub struct FtsConfig {
///
/// Built-in analyzers: `"standard"`, `"whitespace"`, `"simple"`, `"english"`.
/// Custom analyzers are defined at index creation time.
+ ///
+ /// **DORMANT — accepted, but nothing reads it.** A `text` field without
+ /// an analyzer of its own uses the index's `analysis.analyzer.default`
+ /// when the index declares one (#991), and `standard` otherwise; this
+ /// node-wide key never enters that choice. Reported at startup when set
+ /// (see [`FtsConfig::dormant_overrides`]).
pub default_analyzer: String,
}
@@ -1228,6 +1266,23 @@ impl Default for FtsConfig {
}
}
+impl FtsConfig {
+ /// Full-text settings this build accepts but does not act on, in the
+ /// [`MergeConfig::dormant_overrides`] shape: named only when the operator
+ /// moved one off its default.
+ pub fn dormant_overrides(&self) -> Vec<(&'static str, &'static str)> {
+ let mut out = Vec::new();
+ if self.default_analyzer != FtsConfig::default().default_analyzer {
+ out.push((
+ "fts.default_analyzer",
+ "nothing reads it — a text field uses its own analyzer, the \
+ index's analysis.analyzer.default, or standard",
+ ));
+ }
+ out
+ }
+}
+
// ─────────────────────────────────────────────────────────────────────────────
/// Vector search (HNSW) settings.
@@ -1237,17 +1292,28 @@ impl Default for FtsConfig {
#[serde(default, deny_unknown_fields)]
pub struct VectorConfig {
/// Similarity metric: `"cosine"`, `"dot_product"`, or `"euclidean"` (default: `"cosine"`).
+ ///
+ /// **DORMANT — accepted, but nothing reads it.** A `dense_vector` field
+ /// takes its `similarity` from the mapping, and a mapping without one
+ /// gets `cosine` (the ES default) whatever this says.
pub default_metric: VectorMetric,
/// HNSW `M` parameter — edges per node per layer (default: `16`).
///
/// Higher values improve recall at the cost of memory and build time.
+ ///
+ /// **DORMANT — validated, but nothing reads it.** The graph builds with
+ /// a fixed `M = 16` (`HnswParams::new`).
pub hnsw_m: usize,
/// HNSW `ef_construction` — search width during index build (default: `200`).
+ ///
+ /// **DORMANT — validated, but nothing reads it.** The graph builds with
+ /// a fixed `ef_construction = 200` (`HnswParams::new`).
pub hnsw_ef_construction: usize,
/// HNSW `ef` — search width at query time (default: `100`).
///
- /// Can be overridden per query. Must be ≥ the number of neighbours
- /// requested (`k`).
+ /// **DORMANT — validated, but nothing reads it.** The query-time beam
+ /// width comes from the request's `num_candidates` (1.5 × `k` when
+ /// omitted), floored at `HNSW_EF_FLOOR`.
pub hnsw_ef_search: usize,
/// Default quantization: `"none"` (default) or `"scalar8"`. `"binary"` is
/// **not implemented in this build** and is rejected at startup.
@@ -1262,20 +1328,79 @@ pub struct VectorConfig {
/// originals, so this shrinks the scoring working set, not total resident
/// memory. When the code store cannot serve (open-time walk not yet
/// converged, a publication race, coverage broken by a wrong-dimension
- /// write), queries fall back to the exact `_source` scan. Typically
- /// opted into per field via `index_options.type: int8_hnsw` on the
- /// mapping; this global default applies the same scheme index-wide.
+ /// write), queries fall back to the exact `_source` scan. Opted into
+ /// per field via `index_options.type: int8_hnsw` (or `int8_flat`) on
+ /// the mapping.
/// - `"binary"` — 1-bit binary quantization (~32× memory reduction) — NOT
/// YET IMPLEMENTED (no `BinaryQuantizer` exists).
///
/// Honouring `binary` would silently store full-precision vectors while
/// claiming a saving, so only `none` and `scalar8` are accepted (see
/// `Config::validate`).
+ ///
+ /// **DORMANT — only the `binary` refusal reads it.** Quantization comes
+ /// from each field's `index_options`; setting `"scalar8"` here does not
+ /// quantize a field whose mapping does not ask for it.
pub default_quantization: VectorQuantization,
/// Maximum supported vector dimensionality (default: `16384`).
+ ///
+ /// **DORMANT — validated (must be ≥ 1), but nothing reads it.** No
+ /// mapping or document is checked against it.
pub max_dimensions: usize,
}
+impl VectorConfig {
+ /// Vector settings this build accepts but does not act on, in the
+ /// [`MergeConfig::dormant_overrides`] shape: named only when the operator
+ /// moved one off its default.
+ pub fn dormant_overrides(&self) -> Vec<(&'static str, &'static str)> {
+ let d = VectorConfig::default();
+ let mut out = Vec::new();
+ if self.default_metric != d.default_metric {
+ out.push((
+ "vector.default_metric",
+ "nothing reads it — similarity comes from the field mapping, \
+ cosine when the mapping omits it",
+ ));
+ }
+ if self.hnsw_m != d.hnsw_m {
+ out.push((
+ "vector.hnsw_m",
+ "nothing reads it — the HNSW graph builds with a fixed M = 16",
+ ));
+ }
+ if self.hnsw_ef_construction != d.hnsw_ef_construction {
+ out.push((
+ "vector.hnsw_ef_construction",
+ "nothing reads it — the HNSW graph builds with a fixed \
+ ef_construction = 200",
+ ));
+ }
+ if self.hnsw_ef_search != d.hnsw_ef_search {
+ out.push((
+ "vector.hnsw_ef_search",
+ "nothing reads it — the query-time beam width comes from the \
+ request's num_candidates",
+ ));
+ }
+ if self.default_quantization != d.default_quantization {
+ out.push((
+ "vector.default_quantization",
+ "nothing reads it — quantization comes from each field's \
+ index_options in the mapping",
+ ));
+ }
+ if self.max_dimensions != d.max_dimensions {
+ out.push((
+ "vector.max_dimensions",
+ "nothing reads it — mappings and documents are not checked \
+ against it",
+ ));
+ }
+ out
+ }
+}
+
impl Default for VectorConfig {
fn default() -> Self {
Self {
@@ -1322,10 +1447,17 @@ pub enum VectorQuantization {
#[serde(default, deny_unknown_fields)]
pub struct LogsConfig {
/// How long to retain log data before automatic deletion (default: `90` days).
+ ///
+ /// **DORMANT — accepted, but nothing reads it, and nothing deletes log
+ /// data.** `xerj_logs::RetentionPolicy` exists but is constructed only by
+ /// that crate's own tests; no server or engine code uses `xerj-logs`.
pub retention_days: u32,
/// Time-based partition granularity (default: `"1h"`).
///
/// Supported values: `"1m"`, `"5m"`, `"15m"`, `"1h"`, `"6h"`, `"1d"`.
+ ///
+ /// **DORMANT — accepted, but nothing reads it.** Log indices are not
+ /// partitioned by time.
pub time_partition: String,
}
@@ -1338,6 +1470,34 @@ impl Default for LogsConfig {
}
}
+impl LogsConfig {
+ /// `[logs]` settings this build accepts but does not act on, in the
+ /// [`MergeConfig::dormant_overrides`] shape.
+ ///
+ /// Unlike the merge and compression knobs, the cost of ignoring these is
+ /// data, not latency: an operator who sets `retention_days = 30` to keep a
+ /// log volume bounded gets unbounded growth. They stay accepted so a
+ /// config the project handed out still boots, and are named at startup.
+ pub fn dormant_overrides(&self) -> Vec<(&'static str, &'static str)> {
+ let d = LogsConfig::default();
+ let mut out = Vec::new();
+ if self.retention_days != d.retention_days {
+ out.push((
+ "logs.retention_days",
+ "nothing deletes log data in this build — no retention job \
+ runs; delete old indices or documents yourself",
+ ));
+ }
+ if self.time_partition != d.time_partition {
+ out.push((
+ "logs.time_partition",
+ "nothing reads it — log indices are not partitioned by time",
+ ));
+ }
+ out
+ }
+}
+
// ─────────────────────────────────────────────────────────────────────────────
/// Embedding backend settings.
@@ -1650,10 +1810,18 @@ pub struct IndexingConfig {
///
/// Larger batches amortise WAL and fsync overhead but increase per-batch
/// latency. Values between 500 and 5000 work well for most workloads.
+ ///
+ /// **DORMANT — accepted, but nothing reads it.** A turbo request is
+ /// indexed as one batch, whatever its size.
pub turbo_batch_size: usize,
/// Enable parallel tokenisation via Rayon in turbo mode (default: `true`).
///
/// Disable only for debugging or on single-core machines.
+ ///
+ /// **DORMANT — read, but a no-op.** Since M5.9 the turbo pipeline does
+ /// not pre-tokenise (the FTS index is built from stored fields at merge
+ /// time), so the `parallel` argument it feeds is unused — see the
+ /// `turbo_ingest` module docs.
pub turbo_parallel: bool,
/// Skip stemming and stopword removal in turbo mode for maximum speed (default: `false`).
///
@@ -1661,6 +1829,9 @@ pub struct IndexingConfig {
/// normally be processed by the configured `fts.default_analyzer`. Search
/// recall may be reduced (e.g. "running" won't match "run"), but ingest
/// throughput increases significantly.
+ ///
+ /// **DORMANT — read, but a no-op.** The turbo endpoint passes it to
+ /// `index_batch_turbo`, whose `_fast_analyzer` parameter is unused.
pub turbo_fast_analyzer: bool,
}
@@ -1674,6 +1845,36 @@ impl Default for IndexingConfig {
}
}
+impl IndexingConfig {
+ /// Turbo settings this build accepts but does not act on, in the
+ /// [`MergeConfig::dormant_overrides`] shape.
+ pub fn dormant_overrides(&self) -> Vec<(&'static str, &'static str)> {
+ let d = IndexingConfig::default();
+ let mut out = Vec::new();
+ if self.turbo_batch_size != d.turbo_batch_size {
+ out.push((
+ "indexing.turbo_batch_size",
+ "nothing reads it — a turbo request is indexed as one batch",
+ ));
+ }
+ if self.turbo_parallel != d.turbo_parallel {
+ out.push((
+ "indexing.turbo_parallel",
+ "turbo ingest no longer pre-tokenises, so there is nothing to \
+ parallelise",
+ ));
+ }
+ if self.turbo_fast_analyzer != d.turbo_fast_analyzer {
+ out.push((
+ "indexing.turbo_fast_analyzer",
+ "the turbo path ignores it — fields keep their configured \
+ analyzers",
+ ));
+ }
+ out
+ }
+}
+
// ─────────────────────────────────────────────────────────────────────────────
/// Engine parallelism settings for vertical scaling.
@@ -3092,6 +3293,98 @@ mod tests {
);
}
+ /// The rest of the accepted-and-ignored keys (#204): each one was
+ /// documented as working, validated where it has a range, and read by
+ /// nothing — measured on rc.89, `fts.default_analyzer = "whitespace"`,
+ /// `vector.default_metric = "euclidean"`, `vector.max_dimensions = 4` and
+ /// a 90→30 day `logs.retention_days` all changed nothing, with no warning.
+ /// `Config::dormant_overrides` must name each one, only when it is set,
+ /// and must stay quiet on the shipped default file.
+ #[test]
+ fn every_dormant_setting_is_named_only_when_an_operator_sets_it() {
+ assert!(
+ Config::default().dormant_overrides().is_empty(),
+ "an untouched default asks for nothing, so it must not warn"
+ );
+ let shipped = Config::from_toml_str(include_str!("../../../xerj.default.toml")).unwrap();
+ assert!(
+ shipped.dormant_overrides().is_empty(),
+ "`cp xerj.default.toml xerj.toml` must boot without warnings: {:?}",
+ shipped.dormant_overrides()
+ );
+
+ for (toml, expected) in [
+ (
+ "[fts]\ndefault_analyzer = \"whitespace\"",
+ "fts.default_analyzer",
+ ),
+ (
+ "[vector]\ndefault_metric = \"euclidean\"",
+ "vector.default_metric",
+ ),
+ ("[vector]\nhnsw_m = 32", "vector.hnsw_m"),
+ (
+ "[vector]\nhnsw_ef_construction = 400",
+ "vector.hnsw_ef_construction",
+ ),
+ ("[vector]\nhnsw_ef_search = 200", "vector.hnsw_ef_search"),
+ (
+ "[vector]\ndefault_quantization = \"scalar8\"",
+ "vector.default_quantization",
+ ),
+ ("[vector]\nmax_dimensions = 4", "vector.max_dimensions"),
+ ("[logs]\nretention_days = 30", "logs.retention_days"),
+ ("[logs]\ntime_partition = \"1d\"", "logs.time_partition"),
+ (
+ "[indexing]\nturbo_batch_size = 2000",
+ "indexing.turbo_batch_size",
+ ),
+ (
+ "[indexing]\nturbo_parallel = false",
+ "indexing.turbo_parallel",
+ ),
+ (
+ "[indexing]\nturbo_fast_analyzer = true",
+ "indexing.turbo_fast_analyzer",
+ ),
+ (
+ "[storage]\nlocal_cache_dir = \"/mnt/nvme\"",
+ "storage.local_cache_dir",
+ ),
+ // The two older sections now reach the same aggregate.
+ ("[merge]\nmin_segments = 4", "merge.min_segments"),
+ (
+ "[compression]\nblock_size_docs = 512",
+ "compression.block_size_docs",
+ ),
+ ] {
+ let cfg = Config::from_toml_str(&format!("{toml}\n")).unwrap();
+ let named: Vec<&str> = cfg
+ .dormant_overrides()
+ .into_iter()
+ .map(|(key, _)| key)
+ .collect();
+ assert_eq!(
+ named,
+ vec![expected],
+ "setting {toml:?} must be reported, and nothing else"
+ );
+ }
+
+ // Neighbours that ARE read must never be reported.
+ for toml in [
+ "[storage]\nwal_sync = \"sync\"",
+ "[embedding]\nbatch_size = 16",
+ "[limits]\nmax_fields_per_index = 50",
+ ] {
+ let cfg = Config::from_toml_str(&format!("{toml}\n")).unwrap();
+ assert!(
+ cfg.dormant_overrides().is_empty(),
+ "{toml:?} is wired, so it is not dormant"
+ );
+ }
+ }
+
/// The documented 16–4096 range was never enforced: #318's repro node
/// booted happily on `block_size_docs = 999999`.
#[test]
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index 09f6149ee..2d0be9a0d 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -571,23 +571,6 @@ fn load_config(args: &CliArgs) -> Result<Config> {
anyhow::bail!("config error: {reason}");
}
- // Merge settings this build accepts but does not act on. An operator who
- // throttles merges to protect query latency was getting no throttle and no
- // signal (#207); the signal is the minimum. See
- // `MergeConfig::dormant_overrides` for why these are a warning rather than
- // the hard startup error `storage.backend` gets.
- for (key, effect) in cfg.merge.dormant_overrides() {
- warn!("{key} is set but has no effect in this build: {effect}");
- }
-
- // Same treatment for `[compression]` (#318): `level` is wired into the
- // merge re-encode, `enabled` and `block_size_docs` are not, and an
- // operator who set all three to trade CPU for $/GB previously got three
- // no-ops in silence.
- for (key, effect) in cfg.compression.dormant_overrides() {
- warn!("{key} is set but has no effect in this build: {effect}");
- }
-
// Bind address override: `--bind` flag or `XERJ_BIND_ADDRESS` env (flag
// wins). Both exist because the default is loopback (#228): without a way
// to say "expose me" that is not a TOML file, a container image or a
@@ -998,6 +981,23 @@ async fn build_tls_config(cfg: &Config) -> Result<Option<RustlsConfig>> {
// Observability
// ─────────────────────────────────────────────────────────────────────────────
+/// Settings this build accepts but does not act on. An operator who throttles
+/// merges to protect query latency was getting no throttle and no signal
+/// (#207); the signal is the minimum. See `MergeConfig::dormant_overrides` for
+/// why these are a warning rather than the hard startup error `storage.backend`
+/// gets. `[compression]` (#318) and the `[fts]` / `[vector]` / `[logs]` /
+/// `[indexing]` / `storage.local_cache_dir` keys that nothing reads get the
+/// same treatment — `Config::dormant_overrides` collects every section.
+///
+/// Call it after `init_tracing`. It used to run inside `load_config`, which
+/// runs before the subscriber exists, so every one of these warnings was
+/// dropped and an operator saw nothing.
+fn warn_dormant_overrides(cfg: &Config) {
+ for (key, effect) in cfg.dormant_overrides() {
+ warn!("{key} is set but has no effect in this build: {effect}");
+ }
+}
+
fn init_tracing(logging: &xerj_common::config::LoggingConfig) {
let mut filter = EnvFilter::try_from_env("XERJ_LOG")
.or_else(|_| EnvFilter::try_from_env("RUST_LOG"))
@@ -1350,6 +1350,7 @@ async fn run_cli_index(cmd: IndexCmdArgs) -> Result<()> {
let mut cfg = load_config(&fake_cli)?;
// Tracing after config so the [logging] format applies (RC4-W4 item 6).
init_tracing(&cfg.logging);
+ warn_dormant_overrides(&cfg);
cfg.tls.enabled = false;
cfg.auth.enabled = false;
cfg.cluster.enabled = false;
@@ -2311,6 +2312,7 @@ async fn async_main() -> Result<()> {
init_tracing(&cfg.logging);
info!("xerj v{} starting", env!("CARGO_PKG_VERSION"));
+ warn_dormant_overrides(&cfg);
// 3a. `server.bind_address` must be an IP literal — and must be rejected
// *here*, ahead of the two exposure checks that follow.
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index e8f49ec05..ef8326360 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -329,6 +329,10 @@ block_size_docs = 128
#
# Custom analyzers (with tokenizer + filter chains) can be defined at index
# creation time via the mappings API.
+#
+# NOT WIRED in this build: nothing reads this key. A text field uses its own
+# analyzer, else the index's analysis.analyzer.default, else "standard".
+# Setting it away from the default logs a warning at startup.
default_analyzer = "standard"
# ── Vector Search / HNSW (6 settings) ─────────────────────────────────────────
@@ -341,8 +345,16 @@ default_analyzer = "standard"
# Best for text embeddings and semantic search.
# "dot_product" — Raw dot product. Fastest; requires unit-norm vectors.
# "euclidean" — L2 (Euclidean) distance. Best for image/audio features.
+#
+# NOT WIRED in this build: a dense_vector field takes `similarity` from its
+# mapping, and a mapping without one gets "cosine" whatever this says.
default_metric = "cosine"
+# The three hnsw_* keys are validated but NOT WIRED in this build: the graph
+# builds with a fixed M = 16 and ef_construction = 200, and the query-time
+# beam width comes from the request's num_candidates (floored at 800).
+# Setting any of them away from the default logs a warning at startup.
+
# HNSW M parameter — number of bi-directional edges per node per layer.
# Higher M → better recall at the cost of more memory and slower index builds.
# Typical range: 8 – 64. Default: 16.
@@ -355,7 +367,7 @@ hnsw_ef_construction = 200
# HNSW ef — search beam width at query time.
# Higher values → better recall at the cost of slower queries.
-# Can be overridden per-query. Must be >= the number of neighbors requested (k).
+# Must be >= the number of neighbors requested (k).
# Typical range: 50 – 1000. Default: 100.
hnsw_ef_search = 100
@@ -374,11 +386,16 @@ hnsw_ef_search = 100
# vectors while claiming a 32× saving. 4-bit ("scalar4") quantization and
# automatic quantization downgrade are not implemented either and have no
# settings here.
+#
+# NOT WIRED as a node-wide default: only the "binary" refusal reads this key.
+# Quantization comes from each field's mapping (index_options.type: int8_hnsw
+# or int8_flat); "scalar8" here does not quantize any field.
default_quantization = "none"
# Maximum supported vector dimensionality.
-# Vectors with more dimensions than this are rejected at ingest time.
-# xerj supports up to 16,384 dimensions — 4× Elasticsearch's limit of 4,096.
+# NOT WIRED in this build: validated (must be >= 1), but no mapping or
+# document is checked against it. Setting it away from the default logs a
+# warning at startup.
# Common values: 384 (MiniLM), 768 (BERT), 1536 (OpenAI ada-002), 3072 (GPT-4o).
max_dimensions = 16384
@@ -386,17 +403,18 @@ max_dimensions = 16384
[logs]
+# NOT WIRED in this build: nothing reads either key, no retention job runs,
+# and log indices are not partitioned by time. Log data is kept until you
+# delete it. Setting either key away from its default logs a warning at
+# startup.
+
# How many days to retain log index data before automatic deletion.
# Set to 0 to disable automatic deletion (retain forever).
retention_days = 90
# Time-based partition granularity for log indices.
-# Log data is split into partitions of this width to enable efficient
-# time-range pruning during retention enforcement.
#
# Valid values: "1m", "5m", "15m", "1h", "6h", "1d"
-# Use "1h" or "6h" for high-volume streaming logs.
-# Use "1d" for lower-volume application logs.
time_partition = "1h"
# ── External Embedding Service (4 of 19 settings) ─────────────────────────────
@@ -499,7 +517,12 @@ max_segment_hydration_cache_mb = 0
# - POST /v1/indices/{name}/turbo-ingest (dedicated endpoint)
# - X-Turbo: true header on standard _bulk requests
#
-# These settings control the behaviour when turbo mode is activated.
+# NOT WIRED in this build: none of the three keys below changes anything.
+# A turbo request is indexed as one batch, turbo ingest no longer
+# pre-tokenises (so there is nothing to parallelise), and the fast-analyzer
+# flag reaches a parameter the turbo path ignores. The standard _bulk path
+# takes the batched route for every request, header or not. Setting any key
+# away from its default logs a warning at startup.
[indexing]
diff --git a/landing/docs/analyzers.html b/landing/docs/analyzers.html
index ee1f5c817..c825a019b 100644
--- a/landing/docs/analyzers.html
+++ b/landing/docs/analyzers.html
@@ -219,7 +219,7 @@
<h1>Analyzers</h1>
- <p>An analyzer turns raw text into a token stream. Every text field has exactly one analyzer chosen at index-create time. XERJ ships four built-ins. <code class="inline">[fts] default_analyzer</code> picks which one applies unless a field overrides it.</p>
+ <p>An analyzer turns raw text into a token stream. Every text field has exactly one analyzer chosen at index-create time. XERJ ships four built-ins. A field without an analyzer of its own uses the index's <code class="inline">analysis.analyzer.default</code> when the index declares one, and <code class="inline">standard</code> otherwise. (<code class="inline">[fts] default_analyzer</code> in the node config is accepted but not wired in this build; setting it logs a warning at startup.)</p>
<h2>standard</h2>
<p>The default. Unicode-aware segmentation, lowercasing, stop-word list for English. Correct for most log and document workloads. Equivalent to ES's <code class="inline">standard</code> analyzer.</p>
diff --git a/landing/docs/config.html b/landing/docs/config.html
index af5b7c2bf..1f734f9e4 100644
--- a/landing/docs/config.html
+++ b/landing/docs/config.html
@@ -224,7 +224,7 @@
<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p>
<h2>Full key table</h2>
- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, <strong>not</strong> RAM: the scan reads the full-precision vector from <code class="inline">_source</code> and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ&#39;s own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index&#39;s WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index. Must be at least 1.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target&#39;s http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. Range 1&ndash;86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node&#39;s disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap &mdash; the config file must not be a way around the bound.</div><div class="group" id="rerank">[rerank]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Allow the <code class="inline">rerank</code> search stage. It is the only search-time feature that sends document text off this node, and it does nothing until a key is set. <strong>false</strong> refuses every rerank request with HTTP 403, whatever the environment holds, and nothing is sent.</div><div class="cell k">api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Rerank provider key. Empty falls back to the TYPESAFE_API_KEY environment variable; a value here wins over the environment. Empty in both places means reranking is not configured and a rerank request gets HTTP 503. Write-only: GET /_xerj/rerank reports whether a key is set and where it came from, never its value. There is no per-request key.</div><div class="cell k">endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Where judged document text is POSTed. Empty falls back to TYPESAFE_ENDPOINT, then to https://api.typesafe.ai/v1/systemone. Must be an absolute http:// or https:// URL with no credentials in it (user:pass@host is refused at startup, whether it came from this file or from TYPESAFE_ENDPOINT): the endpoint is echoed by GET /_xerj/rerank.</div></div>
+ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping. <strong>Not wired</strong>: nothing reads this key — a text field uses its own analyzer, else the index's <code class="inline">analysis.analyzer.default</code>, else standard. Setting it away from the default logs a warning at startup.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice. <strong>Not wired</strong>: similarity comes from the field mapping, and a mapping without one gets cosine whatever this says. Setting it away from the default logs a warning at startup.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, <strong>not</strong> RAM: the scan reads the full-precision vector from <code class="inline">_source</code> and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping). <strong>Not wired as a node-wide default</strong>: only the "binary" refusal reads it; quantization comes from each field's <code class="inline">index_options</code> in the mapping. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096. <strong>Not wired</strong>: validated (≥ 1), but no mapping or document is checked against it. Setting it away from the default logs a warning at startup.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Accepted but <strong>not wired</strong>: no retention job runs and nothing deletes log data in this build — delete old indices or documents yourself. Setting it away from the default logs a warning at startup.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning. <strong>Not wired</strong>: log indices are not partitioned by time. Setting it away from the default logs a warning at startup.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency. <strong>Not wired</strong>: a turbo request is indexed as one batch. Setting it away from the default logs a warning at startup.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging. <strong>Not wired</strong>: turbo ingest no longer pre-tokenises, so there is nothing to parallelise. Setting it away from the default logs a warning at startup.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed. <strong>Not wired</strong>: the turbo path ignores it; fields keep their analyzers. Setting it away from the default logs a warning at startup.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div><div class="group" id="wal_tap">[wal_tap]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.</div><div class="cell k">target_url</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.</div><div class="cell k">target_auth</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.</div><div class="cell k">indices</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ&#39;s own .xerj* system indices are never shipped whatever this says.</div><div class="cell k">poll_interval_ms</div><div class="cell t">u64</div><div class="cell d">500</div><div class="cell dsc">How often each allowlisted index&#39;s WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.</div><div class="cell k">max_batch_docs</div><div class="cell t">usize</div><div class="cell d">1000</div><div class="cell dsc">Maximum WAL entries read and shipped per poll, per index. Must be at least 1.</div><div class="cell k">max_batch_bytes</div><div class="cell t">usize</div><div class="cell d">5242880</div><div class="cell dsc">Maximum _bulk body size (bytes). Bounds one HTTP request against the target&#39;s http.max_content_length — a larger batch is split across requests, never truncated.</div><div class="cell k">request_timeout_secs</div><div class="cell t">u64</div><div class="cell d">30</div><div class="cell dsc">Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.</div><div class="cell k">max_retry_backoff_secs</div><div class="cell t">u64</div><div class="cell d">60</div><div class="cell dsc">Ceiling on the exponential, jittered retry backoff when the target is down. Range 1&ndash;86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.</div><div class="cell k">min_retained_generations</div><div class="cell t">u64</div><div class="cell d">0</div><div class="cell dsc">Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node&#39;s disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap &mdash; the config file must not be a way around the bound.</div><div class="group" id="rerank">[rerank]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Allow the <code class="inline">rerank</code> search stage. It is the only search-time feature that sends document text off this node, and it does nothing until a key is set. <strong>false</strong> refuses every rerank request with HTTP 403, whatever the environment holds, and nothing is sent.</div><div class="cell k">api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Rerank provider key. Empty falls back to the TYPESAFE_API_KEY environment variable; a value here wins over the environment. Empty in both places means reranking is not configured and a rerank request gets HTTP 503. Write-only: GET /_xerj/rerank reports whether a key is set and where it came from, never its value. There is no per-request key.</div><div class="cell k">endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Where judged document text is POSTed. Empty falls back to TYPESAFE_ENDPOINT, then to https://api.typesafe.ai/v1/systemone. Must be an absolute http:// or https:// URL with no credentials in it (user:pass@host is refused at startup, whether it came from this file or from TYPESAFE_ENDPOINT): the endpoint is echoed by GET /_xerj/rerank.</div></div>
<h2 id="server-section">[server]</h2>
@@ -283,26 +283,26 @@ level = "balanced" # Zstd L1 / L3 / L6, applied at MERGE (flush is
block_size_docs = 128 # accepted, not wired — 16-4096, range-checked at startup</pre>
<h2 id="fts-section">[fts]</h2>
- <p>Default analyzer applied to untyped <code class="inline">text</code> fields. Override per-field in the mapping when creating an index. See <a href="/docs/analyzers">Analyzers</a> for the built-ins.</p>
+ <p>This key is accepted but <strong>not wired</strong>: a <code class="inline">text</code> field without an analyzer of its own uses the index's <code class="inline">analysis.analyzer.default</code> when the index declares one, and <code class="inline">standard</code> otherwise. Set the analyzer per field or per index in the mapping when creating an index. See <a href="/docs/analyzers">Analyzers</a> for the built-ins.</p>
<pre class="code">[fts]
-default_analyzer = "standard" # unicode words + lowercase</pre>
+default_analyzer = "standard" # accepted, not wired — see the table</pre>
<h2 id="vector-section">[vector]</h2>
<p>Vector search settings. Unfiltered kNN is served by a <strong>persisted HNSW graph with exact rescoring</strong> (measured recall@10 1.00 on the official bench query); filtered kNN and SQ8 fields run the exact scan. The <code class="inline">hnsw_*</code> keys are accepted and validated but not yet wired: the graph builds with fixed M=16 / ef_construction=200, and the query-time beam width comes from the request's <code class="inline">num_candidates</code> (floored at 800). Quantization trades recall for scoring precision, not for memory — see <code class="inline">default_quantization</code> below and <a href="https://github.com/xerj-org/xerj/issues/392">#392</a>.</p>
<pre class="code">[vector]
-default_metric = "cosine"
+default_metric = "cosine" # accepted, not wired — similarity comes from the mapping
hnsw_m = 16 # accepted for compat; kNN serving is exact
hnsw_ef_construction = 200 # accepted for compat; no effect on results
hnsw_ef_search = 100 # accepted for compat; no effect on results
default_quantization = "none" # "scalar8" = int8 precision, 1–2% recall loss.
# NOT a RAM saving today — see issue #392
-max_dimensions = 16384</pre>
+max_dimensions = 16384 # accepted, not wired — nothing checks dims against it</pre>
<h2 id="logs-section">[logs]</h2>
- <p>Time-series retention. Log indices are sliced into partitions of <code class="inline">time_partition</code> width so retention prunes are O(partitions), not O(documents).</p>
+ <p>Time-series retention is <strong>not implemented in this build</strong>: both keys are accepted and nothing reads them, no retention job runs, and log data is kept until you delete it. Setting either away from its default logs a warning at startup.</p>
<pre class="code">[logs]
-retention_days = 30 # not a default (90) — keep 30 days
-time_partition = "1h" # 1-hour partitions</pre>
+retention_days = 90 # accepted, not wired — nothing deletes log data
+time_partition = "1h" # accepted, not wired</pre>
<h2 id="embedding-section">[embedding]</h2>
<p>Delegates vector generation to an OpenAI-compatible endpoint. Leave <code class="inline">default_endpoint</code> empty if clients provide vectors themselves. Token limits are model-specific; the chunker in the <code class="inline">ai</code> crate splits long documents to fit the model's window.</p>
@@ -326,11 +326,11 @@ max_fields_per_index = 500
# max_process_memory_mb = 0 # omit → AUTO (8/16/32 GiB by machine); 0 = whole machine</pre>
<h2 id="indexing-section">[indexing]</h2>
- <p>Turbo mode knobs. Turbo is opt-in per-request via <code class="inline">POST /v1/indices/:name/turbo-ingest</code> or the <code class="inline">X-Turbo: true</code> header on <code class="inline">_bulk</code>; these settings only apply when turbo is active.</p>
+ <p>Turbo mode knobs. Turbo is opt-in per-request via <code class="inline">POST /v1/indices/:name/turbo-ingest</code> or the <code class="inline">X-Turbo: true</code> header on <code class="inline">_bulk</code>; none of the three settings below changes anything in this build (each logs a warning at startup when set away from its default).</p>
<pre class="code">[indexing]
-turbo_batch_size = 2000 # not a default (1000)
-turbo_parallel = true
-turbo_fast_analyzer = false # true only if recall doesn't matter</pre>
+turbo_batch_size = 1000 # accepted, not wired
+turbo_parallel = true # accepted, not wired
+turbo_fast_analyzer = false # accepted, not wired</pre>
<h2 id="cluster-section">[cluster]</h2>
<p>Multi-node mode. Default is off — single-node doesn't need a consensus layer. When enabled, the embedded Raft implementation replicates <strong>metadata only</strong> (schemas, shard assignments, node roster). See <a href="/docs/clustering">Clustering</a> for the full story.</p>
diff --git a/landing/docs/migration-from-es.html b/landing/docs/migration-from-es.html
index 4cb4fd29a..74591107b 100644
--- a/landing/docs/migration-from-es.html
+++ b/landing/docs/migration-from-es.html
@@ -248,7 +248,7 @@
<li><strong>Cross-cluster search.</strong> Single-cluster only in v0.1.</li>
<li><strong>Machine-learning jobs.</strong> Not a feature.</li>
<li><strong>Watcher / alerting framework.</strong> Use the <code class="inline">/v1/explain-plan</code> endpoint + an external rule runner.</li>
- <li><strong>ILM (Index Lifecycle Management).</strong> Retention is controlled per-index via <code class="inline">[logs] retention_days</code>.</li>
+ <li><strong>ILM (Index Lifecycle Management).</strong> Not executed, and there is no retention job: <code class="inline">[logs] retention_days</code> is accepted but not wired in this build. Delete old indices, or old documents with <code class="inline">_delete_by_query</code>, on your own schedule.</li>
</ul>
<h2 id="known-semantic-differences">Known semantic differences</h2>
diff --git a/landing/docs/playbooks/log-analytics.html b/landing/docs/playbooks/log-analytics.html
index 3d1b2d8e1..ce96e5310 100644
--- a/landing/docs/playbooks/log-analytics.html
+++ b/landing/docs/playbooks/log-analytics.html
@@ -4,7 +4,7 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>XERJ.ai · Docs · Playbook · Log analytics</title>
-<meta name="description" content="Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — schema, ingest command, queries, and retention without ILM.">
+<meta name="description" content="Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — the schema, the ingest command and the queries to copy.">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Big+Shoulders+Display:wght@400;700;800;900&family=Inter:wght@400;500;600;700&family=IBM+Plex+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">
@@ -15,7 +15,7 @@
<meta property="og:site_name" content="XERJ.ai">
<meta property="og:type" content="article">
<meta property="og:title" content="XERJ.ai · Docs · Playbook · Log analytics">
-<meta property="og:description" content="Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — schema, ingest command, queries, and retention without ILM.">
+<meta property="og:description" content="Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — the schema, the ingest command and the queries to copy.">
<meta property="og:url" content="https://xerj.org/docs/playbooks/log-analytics">
<meta property="og:image" content="https://xerj.org/og/xerj-card.png">
<meta property="og:image:width" content="1200">
@@ -24,7 +24,7 @@
<meta property="og:locale" content="en_US">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="XERJ.ai · Docs · Playbook · Log analytics">
-<meta name="twitter:description" content="Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — schema, ingest command, queries, and retention without ILM.">
+<meta name="twitter:description" content="Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — the schema, the ingest command and the queries to copy.">
<meta name="twitter:image" content="https://xerj.org/og/xerj-card.png">
<script type="application/ld+json">
{
@@ -58,7 +58,7 @@
"@context": "https://schema.org",
"@type": "TechArticle",
"headline": "XERJ.ai · Docs · Playbook · Log analytics",
- "description": "Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — schema, ingest command, queries, and retention without ILM.",
+ "description": "Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — the schema, the ingest command and the queries to copy.",
"url": "https://xerj.org/docs/playbooks/log-analytics",
"image": "https://xerj.org/og/xerj-card.png",
"inLanguage": "en",
@@ -219,7 +219,7 @@
<h1>Playbook · Log analytics</h1>
- <p>Ingest Nginx, JSON, syslog, or OTLP at line rate. Query hot logs in milliseconds. Cold-drop to cheap storage with ILM-free retention. This is the workload XERJ was originally built for.</p>
+ <p>Ingest Nginx, JSON, syslog, or OTLP at line rate. Query hot logs in milliseconds. Keep the volume bounded by dropping old indices yourself (see Retention). This is the workload XERJ was originally built for.</p>
<h2>Schema</h2>
<pre class="code">$ curl -sX PUT http://localhost:8080/v1/indices/logs \
@@ -277,10 +277,10 @@
}</pre>
<h2>Retention</h2>
- <p>Per-index, in the config. No ILM policy DSL required.</p>
-<pre class="code">[logs]
-retention_days = 30
-time_partition = "1h"</pre>
+ <p>Automatic retention is <strong>not implemented in this build</strong>. The <code class="inline">[logs] retention_days</code> and <code class="inline">time_partition</code> keys are accepted and nothing reads them (setting either logs a warning at startup), so log data is kept until you delete it. Write to one index per day or week and delete the indices that fall outside your window, or run <code class="inline">_delete_by_query</code> with a <code class="inline">range</code> on the timestamp field.</p>
+<pre class="code">DELETE /logs-2026.09.01
+POST /logs/_delete_by_query
+{ "query": { "range": { "@timestamp": { "lt": "now-30d" } } } }</pre>
<h2>Dashboard</h2>
<p>Open the playground and pick <a href="/playground">LOGS · OVERVIEW</a> — nine panels with click-to-filter TopN, 1 ms date_histogram, heatmap, and drill-through to the <a href="/playground">SYSTEM · OVERVIEW</a> dashboard.</p>
diff --git a/landing/docs/troubleshooting.html b/landing/docs/troubleshooting.html
index 827012ca6..4c3983008 100644
--- a/landing/docs/troubleshooting.html
+++ b/landing/docs/troubleshooting.html
@@ -237,11 +237,11 @@ xerj_merge_duration_seconds_sum{index="logs"} 1847.2</pre>
<p><strong>Fix:</strong> if it's memory — raise <code class="inline">[limits] max_query_memory_mb</code> (512 → 1024 or 2048 for aggregation-heavy workloads). If it's concurrency — raise <code class="inline">max_concurrent_searches</code>, but check that the host actually has headroom first.</p>
<h2>Symptom: node RAM climbs forever</h2>
- <p><strong>Likely cause:</strong> vector fields growing without quantization, or too many indices with large <code class="inline">flush_size_mb</code>.</p>
+ <p><strong>Likely cause:</strong> large vector fields, or too many indices with large <code class="inline">flush_size_mb</code>.</p>
<p><strong>Check:</strong></p>
<pre class="code">$ curl -s http://127.0.0.1:8080/v1/metrics | grep memory_usage
xerj_memory_usage_bytes 14200000000 # 14 GB and climbing</pre>
- <p><strong>Fix:</strong> set <code class="inline">[vector] default_quantization = "scalar8"</code> (or opt heavy fields in per-mapping) so vectors are stored as 1-byte-per-dimension codes. Or lower <code class="inline">flush_size_mb</code> so memtables don't grow unbounded.</p>
+ <p><strong>Fix:</strong> lower <code class="inline">flush_size_mb</code> so memtables don't grow unbounded, and set an explicit <code class="inline">[limits] max_process_memory_mb</code> if the automatic cap does not suit the host. Quantization is not a memory fix today: <code class="inline">scalar8</code> (opted into per field with <code class="inline">index_options</code>) shrinks the kNN scoring working set, not resident memory (<a href="https://github.com/xerj-org/xerj/issues/392">#392</a>), and the node-wide <code class="inline">[vector] default_quantization</code> key is not wired.</p>
<h2>Symptom: "WAL replay failed" on restart</h2>
<p><strong>Likely cause:</strong> the server was killed mid-fsync (power loss, OOM kill). The tail WAL file is torn.</p>
diff --git a/landing/docs/vectors.html b/landing/docs/vectors.html
index fcb36c60c..e4c4afc38 100644
--- a/landing/docs/vectors.html
+++ b/landing/docs/vectors.html
@@ -258,11 +258,11 @@
<div class="cell k">default_metric</div>
<div class="cell t">enum</div>
<div class="cell d">"cosine"</div>
- <div class="cell dsc">"cosine" · "dot_product" · "euclidean".</div>
+ <div class="cell dsc">"cosine" · "dot_product" · "euclidean". Accepted but not wired: similarity comes from the field mapping, and a mapping without one gets cosine.</div>
</div>
<h2 id="quantization">Quantization</h2>
- <p><strong>Quantization is off by default</strong> — <code class="inline">[vector] default_quantization = "none"</code>, full-precision f32. Opt in per field with <code class="inline">index_options.type: int8_hnsw</code> (or <code class="inline">int8_flat</code>) on the mapping, or index-wide by setting the key.</p>
+ <p><strong>Quantization is off by default</strong> — <code class="inline">[vector] default_quantization = "none"</code>, full-precision f32. Opt in per field with <code class="inline">index_options.type: int8_hnsw</code> (or <code class="inline">int8_flat</code>) on the mapping. The node-wide key is not wired as a default: only its <code class="inline">"binary"</code> refusal reads it, so <code class="inline">"scalar8"</code> there quantizes nothing.</p>
<ul>
<li><strong>none</strong> — full-precision f32. <em>The default.</em> Use when recall matters more than RAM.</li>
<li><strong>scalar8</strong> — 8-bit per dimension, 1–2% recall loss. The one quantizer the serving path reads. It buys <strong>precision, not memory</strong>: the scan reads the full-precision vector from <code class="inline">_source</code> and quantizes it per query, so nothing shrinks in RAM, and a <code class="inline">scalar8</code> <code class="inline">_score</code> (and the order it produces) depends on the candidate set — see the <a href="/docs/recipes/vector-quantization">quantization recipe</a> and <a href="https://github.com/xerj-org/xerj/issues/392">#392</a>.</li>
diff --git a/scripts/seo/pagedata.py b/scripts/seo/pagedata.py
index f4c7b19ed..f59317c06 100755
--- a/scripts/seo/pagedata.py
+++ b/scripts/seo/pagedata.py
@@ -437,7 +437,7 @@ PAGES: dict[str, dict[str, str]] = {
),
"docs/playbooks/log-analytics.html": dict(
label="Playbook · Log analytics", kind="techarticle",
- description="Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — schema, ingest command, queries, and retention without ILM.",
+ description="Ingest Nginx, JSON, syslog or OTLP logs at line rate and query them in milliseconds — the schema, the ingest command and the queries to copy.",
),
"docs/playbooks/observability.html": dict(
label="Playbook · Observability", kind="techarticle",