AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(compression): wire compression.level into the merge encoder, and say so where it still isn't
xerj-orgde73a2c56398
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 7ced3a626..da57ced40 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -182,6 +182,20 @@ impl Config {return Err(XerjError::config("merge.min_segments must be >= 2"));}+ // Compression: block_size_docs is documented as 16–4096 and was+ // accepted at any value (#318 got `999999` past startup in silence).+ // Range-check it even though the knob is dormant: an out-of-range+ // value is a typo the operator wants to hear about now, and the check+ // must not start passing quietly if the knob is ever wired.+ if !CompressionConfig::BLOCK_SIZE_DOCS_RANGE.contains(&self.compression.block_size_docs) {+ return Err(XerjError::config(format!(+ "compression.block_size_docs must be in {}..={}, got {}",+ CompressionConfig::BLOCK_SIZE_DOCS_RANGE.start(),+ CompressionConfig::BLOCK_SIZE_DOCS_RANGE.end(),+ self.compression.block_size_docs+ )));+ }+// Vector: hnsw_ef_construction >= hnsw_mif self.vector.hnsw_ef_construction < self.vector.hnsw_m {return Err(XerjError::config(@@ -908,14 +922,28 @@ pub enum MergeStrategy {#[serde(default, deny_unknown_fields)]pub struct CompressionConfig {/// Enable block compression for stored fields (default: `true`).+ ///+ /// **Dormant.** Every durable artifact XERJ writes is a self-describing+ /// compressed envelope (`ZBS2` stored, `ZFM4` `.meta`, `ZPS1` `.post`,+ /// the zstd-flagged `.dv` columns); "off" would mean emitting the legacy+ /// uncompressed layouts, which only exist as read-side compatibility+ /// paths. Setting `false` warns at startup — see+ /// [`CompressionConfig::dormant_overrides`].pub enabled: bool,/// Compression level: `"fast"`, `"balanced"`, or `"best"` (default: `"balanced"`).///- /// Uses LZ4 for `"fast"` and Zstandard for `"balanced"` / `"best"`.+ /// Selects the Zstandard effort level used to re-encode a segment's+ /// durable artifacts **at merge time**. See [`CompressionLevel`] for the+ /// level each name maps to, and why flush ignores this knob.pub level: CompressionLevel,/// Number of documents per compressed block (default: `128`).///- /// Larger blocks compress better but increase random read amplification.+ /// **Dormant.** Kept because it has shipped in `xerj.default.toml` since+ /// v0.1, but XERJ's stored codec is columnar over the whole segment+ /// section rather than blocked by document count, so there is no+ /// doc-block for this to size. Validated against its documented 16–4096+ /// range and warned about when moved off the default — see+ /// [`CompressionConfig::dormant_overrides`].pub block_size_docs: u32,}@@ -929,18 +957,113 @@ impl Default for CompressionConfig {}}-/// Compression level.+impl CompressionConfig {+ /// The documented range for `block_size_docs`, quoted in+ /// `xerj.default.toml` and on `landing/docs/config.html`.+ pub const BLOCK_SIZE_DOCS_RANGE: std::ops::RangeInclusive<u32> = 16..=4096;++ /// Compression settings this build accepts but does not act on, in the+ /// same `("compression.key", "what actually happens")` shape as+ /// [`MergeConfig::dormant_overrides`], and reported the same way — only+ /// when the operator has moved one off its default.+ ///+ /// Issue #318 found the whole `[compression]` section inert: two servers+ /// differing only in `enabled` / `level` / `block_size_docs` produced+ /// byte-identical segments (`.seg` 261,481 / `.dv` 10,679 / `.post`+ /// 192,287 on a 3,000-doc corpus), with no error and no warning, while+ /// the docs described all three as live. `level` is now wired into the+ /// merge re-encode; these two are not, and stay accepted-but-warned for+ /// the reason `merge.io_rate_mb_per_sec` does — both ship non-default in+ /// no config the project itself hands out, and the cost of ignoring them+ /// is disk footprint, not data.+ pub fn dormant_overrides(&self) -> Vec<(&'static str, &'static str)> {+ let d = CompressionConfig::default();+ let mut out = Vec::new();+ if self.enabled != d.enabled {+ out.push((+ "compression.enabled",+ "durable artifacts are always compressed in this build — the \+ stored (ZBS2), .meta (ZFM4), .post (ZPS1) and .dv envelopes \+ have no uncompressed write path, only legacy read support",+ ));+ }+ if self.block_size_docs != d.block_size_docs {+ out.push((+ "compression.block_size_docs",+ "nothing reads it — the stored codec is columnar over the \+ whole segment section, not blocked by document count",+ ));+ }+ out+ }+}++/// Compression level — the Zstandard effort applied when a segment's durable+/// artifacts are re-encoded at **merge**.+///+/// Three things about this enum are deliberate and were all decided against+/// what the docs used to promise (issue #318):+///+/// 1. **Merge only.** Flush stays pinned at [`CompressionLevel::Balanced`]'s+/// level 3 regardless of this setting. Raising the flush level to 19 is+/// exactly the regression recorded in+/// `engine/reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`+/// — 1.55 M docs/s peak collapsed to 21 K docs/s with 75 % of documents+/// rejected, because a ~50 ms segment flush became 5–10 s and tripped+/// back-pressure. Merge is off the ingest critical path; flush is not.+/// 2. **All three names are Zstandard**, not "LZ4 for fast". Every durable+/// envelope XERJ writes (`ZBS2`, `ZFM4`, `ZPS1`, zstd-flagged `.dv`) is+/// zstd-framed, and its magic is what the reader dispatches on; making+/// one level switch algorithm would mean writing the legacy LZ4 layouts+/// that survive only as read-compatibility paths. `"fast"` is therefore+/// zstd's own fastest level, not a different codec.+/// 3. **`"best"` is level 6, not 19.** The RFC #148 thread measured L6 at+/// −14.4 % stored / −14.3 % `.dv` / −15.4 % `.meta`, while L9 doubles the+/// decode window from 2.00 to 4.00 MiB and L19 takes it to 8.00 MiB — a+/// cost every streaming point-get pays on read, forever, for a ratio gain+/// that measurement did not show. Lucene and Elasticsearch land in the+/// same place from the other direction: ES's `best_compression` stored+/// fields format tops out at zstd **3** with a bigger block, not at a+/// high effort level (`Zstd814StoredFieldsFormat.java:38-46`, read for+/// approach only — that code is AGPL/SSPL/Elastic-2.0 and none of it is+/// copied here).+///+/// Decoding never depends on the level a segment was written at, so mixed+/// levels across segments — the normal state after changing this setting —+/// need no format flag and no migration. That is why XERJ does not need+/// Lucene's `Lucene90StoredFieldsFormat.MODE_KEY` segment attribute+/// (`Lucene90StoredFieldsFormat.java:113-137`, Apache-2.0): Lucene must+/// record the mode because BEST_SPEED and BEST_COMPRESSION are different+/// algorithms (LZ4 vs Deflate), whereas here they differ only in effort.#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]#[serde(rename_all = "snake_case")]pub enum CompressionLevel {- /// LZ4 — maximum throughput, moderate ratio.+ /// Zstandard level 1 — maximum throughput, moderate ratio.Fast,- /// Zstandard level 3 — good ratio with low CPU overhead.+ /// Zstandard level 3 — good ratio with low CPU overhead. Also the level+ /// every flush uses, whatever this setting says.Balanced,- /// Zstandard level 19 — maximum ratio, higher CPU cost.+ /// Zstandard level 6 — best measured ratio per byte of decode window.Best,}+impl CompressionLevel {+ /// The Zstandard level this name maps to.+ ///+ /// The single source of truth for the mapping: the encoder sites in+ /// `xerj-storage` and `xerj-fts` take an `i32` level, and+ /// `xerj-compress`'s codec factory routes through here too, so the+ /// meaning of "best" cannot drift between the config surface and the+ /// codecs.+ pub const fn zstd_level(self) -> i32 {+ match self {+ Self::Fast => 1,+ Self::Balanced => 3,+ Self::Best => 6,+ }+ }+}+// ─────────────────────────────────────────────────────────────────────────────/// Full-text search settings.@@ -2126,6 +2249,94 @@ mod tests {);}+ /// #318 — the `[compression]` section reached no encoder at all: two+ /// nodes differing only in `enabled` / `level` / `block_size_docs` wrote+ /// byte-identical segments, with no error and no warning, while the docs+ /// called all three live. `level` is wired now (see the engine-side merge+ /// test); the other two are dormant, and dormant must be *audible*.+ #[test]+ fn dormant_compression_settings_are_named_only_when_an_operator_sets_them() {+ assert!(+ CompressionConfig::default().dormant_overrides().is_empty(),+ "an untouched default asks for nothing, so it must not warn"+ );++ for (toml, expected) in [+ ("enabled = false", "compression.enabled"),+ ("block_size_docs = 512", "compression.block_size_docs"),+ ] {+ let cfg = Config::from_toml_str(&format!("[compression]\n{toml}\n")).unwrap();+ let named: Vec<&str> = cfg+ .compression+ .dormant_overrides()+ .into_iter()+ .map(|(key, _)| key)+ .collect();+ assert_eq!(+ named,+ vec![expected],+ "setting {toml} must be reported, and nothing else"+ );+ }++ // `level` now reaches the merge re-encode, so it must NOT be reported+ // — a warning on a knob that works is the same lie in the other+ // direction.+ let cfg = Config::from_toml_str("[compression]\nlevel = \"best\"\n").unwrap();+ assert!(+ cfg.compression.dormant_overrides().is_empty(),+ "compression.level reaches the merge encoder, so it is not dormant"+ );+ }++ /// The documented 16–4096 range was never enforced: #318's repro node+ /// booted happily on `block_size_docs = 999999`.+ #[test]+ fn out_of_range_block_size_docs_is_refused_at_startup() {+ for bad in [15u32, 4097, 999_999, 0] {+ // Rejected on the path a real boot takes — `from_toml_str`+ // validates, so #318's `block_size_docs = 999999` node would not+ // have started.+ let err = Config::from_toml_str(&format!("[compression]\nblock_size_docs = {bad}\n"))+ .expect_err("{bad} is outside the documented 16..=4096 range");+ assert!(+ err.to_string().contains("compression.block_size_docs"),+ "the error must name the setting, got: {err}"+ );++ // And directly, so the check cannot be lost by a refactor that+ // moves parsing off `validate`.+ let mut cfg = Config::default();+ cfg.compression.block_size_docs = bad;+ assert!(cfg.validate().is_err(), "{bad} must fail validate()");+ }++ for ok in [16u32, 128, 4096] {+ Config::from_toml_str(&format!("[compression]\nblock_size_docs = {ok}\n"))+ .unwrap_or_else(|e| panic!("block_size_docs = {ok} is in range, got: {e}"));+ }+ }++ /// The zstd level behind each name. Pinned because these numbers are+ /// quoted in `xerj.default.toml` and on `landing/docs/compression.html`,+ /// and because `"best"` deliberately means 6 rather than the 19 the enum+ /// used to document — see the doc comment for the measurements.+ #[test]+ fn compression_level_names_map_to_the_documented_zstd_levels() {+ assert_eq!(CompressionLevel::Fast.zstd_level(), 1);+ assert_eq!(CompressionLevel::Balanced.zstd_level(), 3);+ assert_eq!(CompressionLevel::Best.zstd_level(), 6);++ // The default must stay byte-for-byte what flush already writes, so+ // that adopting this build changes nothing for an operator who never+ // touched the section.+ assert_eq!(+ CompressionConfig::default().level.zstd_level(),+ 3,+ "the default level must equal the pinned flush level"+ );+ }+#[test]fn onnx_throughput_controls_preserve_single_session_defaults() {let cfg = Config::from_toml_str("[embedding]\n").unwrap();diff --git a/engine/crates/xerj-compress/src/codec.rs b/engine/crates/xerj-compress/src/codec.rsindex 11cbe8052..57f032054 100644--- a/engine/crates/xerj-compress/src/codec.rs+++ b/engine/crates/xerj-compress/src/codec.rs@@ -1,5 +1,6 @@//! Compression codecs: LZ4, Zstd, and passthrough.+use xerj_common::config::CompressionLevel as ConfigLevel;use xerj_common::XerjError;/// Result alias for codec operations.@@ -27,7 +28,15 @@ pub trait Codec: Send + Sync {// Compression level// ─────────────────────────────────────────────────────────────────────────────-/// Selects the compression algorithm and effort level.+/// Selects the compression algorithm and effort level for the block codecs in+/// this crate.+///+/// Not the operator-facing type: `compression.level` in the config file+/// deserialises into [`xerj_common::config::CompressionLevel`], which has no+/// `None` variant and lives in a crate below this one in the dependency graph.+/// The two stay in step because the zstd effort each name means is defined+/// once, in that enum's `zstd_level`, and [`ZstdCodec::balanced`] /+/// [`ZstdCodec::best`] read it from there.#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]pub enum CompressionLevel {/// No compression — raw bytes pass through unchanged.@@ -35,9 +44,9 @@ pub enum CompressionLevel {/// LZ4: very fast compression with moderate ratio.#[default]Fast,- /// Zstd level 3: balanced speed vs. ratio.+ /// Zstd at the `"balanced"` effort level.Balanced,- /// Zstd level 19: maximum compression ratio.+ /// Zstd at the `"best"` effort level.Best,}@@ -118,7 +127,8 @@ impl Codec for Lz4Codec {/// Zstd-based codec wrapping the [`zstd`] crate.#[derive(Debug, Clone)]pub struct ZstdCodec {- /// Zstd compression level (1–22; defaults to 3 for balanced, 19 for best).+ /// Zstd compression level, from+ /// [`xerj_common::config::CompressionLevel::zstd_level`].level: i32,}@@ -127,14 +137,20 @@ impl ZstdCodec {Self { level }}- /// Balanced Zstd (level 3).+ /// Balanced Zstd — the level `compression.level = "balanced"` selects.pub fn balanced() -> Self {- Self::new(3)+ Self::new(ConfigLevel::Balanced.zstd_level())}- /// Maximum Zstd compression (level 19).+ /// Maximum-ratio Zstd — the level `compression.level = "best"` selects.+ ///+ /// This used to be a hardcoded 19 while the config surface's own enum+ /// documented 19 too and neither reached an encoder (#318). Both now read+ /// the same table, so "best" cannot mean one thing to an operator and+ /// another to a codec; see [`ConfigLevel`] for why the measured answer is+ /// 6 rather than 19.pub fn best() -> Self {- Self::new(19)+ Self::new(ConfigLevel::Best.zstd_level())}}diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 2187c51d7..7dee9b176 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -2106,6 +2106,176 @@ mod merge_publication_transaction_tests {(engine, index, component, segments_dir)}+ /// Deterministic, compressible-but-not-trivial documents: a fixed word+ /// list drawn by an LCG, so every run of this test encodes the same bytes+ /// and any size difference is attributable to the codec alone.+ fn compression_fixture_docs(count: usize) -> Vec<serde_json::Value> {+ const WORDS: [&str; 16] = [+ "segment",+ "merge",+ "posting",+ "term",+ "dictionary",+ "zstandard",+ "columnar",+ "envelope",+ "flush",+ "durable",+ "compression",+ "level",+ "operator",+ "ratio",+ "decode",+ "window",+ ];+ let mut state: u64 = 0x2026_0318;+ let mut next = || {+ state = state+ .wrapping_mul(6364136223846793005)+ .wrapping_add(1442695040888963407);+ (state >> 33) as usize+ };+ (0..count)+ .map(|i| {+ let body: Vec<&str> = (0..40).map(|_| WORDS[next() % WORDS.len()]).collect();+ serde_json::json!({ "body": format!("document {i} {}", body.join(" ")) })+ })+ .collect()+ }++ /// Ingest the same corpus into a fresh index at `level`, flush two+ /// segments, merge them, and return `extension → total bytes` for the+ /// pre-merge (flushed) and post-merge segment families.+ async fn artifact_bytes_at_level(+ dir: &TempDir,+ level: xerj_common::config::CompressionLevel,+ ) -> (+ std::collections::BTreeMap<String, u64>,+ std::collections::BTreeMap<String, u64>,+ ) {+ let mut cfg = config(dir);+ cfg.compression.level = level;+ // One ingest shard, so each flush publishes exactly one segment and+ // the merge below is a deterministic 2 → 1. With the default 16+ // shards a flush yields 16 tiny segments, most below `V2_MIN_DOCS`,+ // which is the wrong shape for measuring a codec.+ cfg.engine.ingest_shards = 1;+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("body", FieldType::Text))+ .unwrap();+ let engine = Engine::new(cfg).unwrap();+ engine.create_index("compression", schema).unwrap();+ let index = engine.get_index("compression").unwrap();+ index.abort_background_tasks();++ // Two flushes → two segments to merge. Each is comfortably over+ // `V2_MIN_DOCS` (128), so the stored section takes the columnar v2+ // path rather than the tiny-segment v1 fallback.+ let docs = compression_fixture_docs(600);+ for (half, chunk) in docs.chunks(300).enumerate() {+ for (i, doc) in chunk.iter().enumerate() {+ index+ .index_document(Some(format!("{half}-{i}")), doc.clone())+ .await+ .unwrap();+ }+ index.flush().await.unwrap();+ }+ let segments_dir = index.data_dir.join("segments");+ let flushed = artifact_bytes_by_extension(&segments_dir);++ assert_eq!(index.store.snapshot().segments.len(), 2);+ assert_eq!(index.run_merge_once().await.unwrap(), 1);+ let snapshot = index.store.snapshot();+ assert_eq!(snapshot.segments.len(), 1);+ let merged_id = snapshot.segments[0].id.to_string();++ let merged = artifact_bytes_by_extension_for(&segments_dir, &merged_id);+ (flushed, merged)+ }++ fn artifact_bytes_by_extension(+ segments_dir: &std::path::Path,+ ) -> std::collections::BTreeMap<String, u64> {+ artifact_bytes_by_extension_for(segments_dir, "")+ }++ /// Sum on-disk bytes per file extension, restricted to files whose name+ /// starts with `id_prefix` (pass `""` for every segment in the dir).+ /// Keyed by extension because merged segment IDs are fresh UUIDs, so the+ /// two runs being compared never share a filename.+ fn artifact_bytes_by_extension_for(+ segments_dir: &std::path::Path,+ id_prefix: &str,+ ) -> std::collections::BTreeMap<String, u64> {+ let mut out = std::collections::BTreeMap::new();+ for entry in std::fs::read_dir(segments_dir).unwrap() {+ let entry = entry.unwrap();+ let name = entry.file_name().to_string_lossy().into_owned();+ if !name.starts_with(id_prefix) {+ continue;+ }+ let Some(ext) = name.rsplit('.').next() else {+ continue;+ };+ // Only the compressed durable artifacts. `.fst` and `.norms` are+ // not zstd envelopes, and the marker files are fixed-size.+ if !matches!(ext, "seg" | "dv" | "post" | "meta") {+ continue;+ }+ *out.entry(ext.to_string()).or_insert(0) += entry.metadata().unwrap().len();+ }+ out+ }++ /// #318 — `[compression]` was accepted and ignored. Two servers on the+ /// same build with the same corpus, differing only in this section,+ /// produced byte-identical indices: `.seg` 261,481 / `.dv` 10,679 /+ /// `.post` 192,287 in both, no error, no warning, while the docs called+ /// the settings live.+ ///+ /// The knob is honoured at MERGE, so this asserts both halves of that+ /// contract: the merged artifacts must respond to `compression.level`,+ /// and the flushed ones must not — raising the flush level is the ingest+ /// collapse recorded in+ /// `reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`, and a+ /// well-meaning "make it consistent" change would reintroduce it.+ #[tokio::test]+ async fn compression_level_reaches_the_merge_encoder_but_never_the_flush_path() {+ use xerj_common::config::CompressionLevel;++ let fast_dir = TempDir::new().unwrap();+ let best_dir = TempDir::new().unwrap();+ let (fast_flushed, fast_merged) =+ artifact_bytes_at_level(&fast_dir, CompressionLevel::Fast).await;+ let (best_flushed, best_merged) =+ artifact_bytes_at_level(&best_dir, CompressionLevel::Best).await;++ assert_eq!(+ fast_flushed, best_flushed,+ "flush must ignore compression.level entirely — it is the \+ back-pressure-critical path (zstd19 ingest regression)"+ );++ // Every compressed artifact family the merge re-encodes must respond.+ // Before the fix each of these was equal, which is the bug.+ for ext in ["seg", "dv", "post", "meta"] {+ let fast = fast_merged.get(ext).copied().unwrap_or_default();+ let best = best_merged.get(ext).copied().unwrap_or_default();+ assert!(+ fast > 0 && best > 0,+ ".{ext} artifacts must exist to compare"+ );+ assert!(+ best < fast,+ ".{ext}: compression.level = \"best\" must produce a smaller \+ merged artifact than \"fast\" (got best={best}, fast={fast}); \+ equal bytes mean the setting reached no encoder"+ );+ }+ }+#[tokio::test]async fn merging_v1_raw_unsafe_fields_advances_marker_before_encoded_output() {let dir = TempDir::new().unwrap();@@ -5290,6 +5460,15 @@ pub struct Index {/// `Config.merge` at index construction; reads are cheap and merge/// runs hold the snapshot for the duration of one batch.merge_config: xerj_common::config::MergeConfig,+ /// Snapshot of `Config.compression`, read by `merge_pass_locked` to pick+ /// the zstd effort for the merged segment's durable artifacts. Only the+ /// merge path consults it: flush stays pinned at the level its own+ /// constants document, because raising it there is the ingest collapse in+ /// `reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`.+ ///+ /// Before #318 this section reached no encoder at all — two nodes+ /// differing only in `[compression]` wrote byte-identical segments.+ compression_config: xerj_common::config::CompressionConfig,/// Retained so a semantic field added after index creation can pin the/// same embedding identity before its first document is accepted.embedding_config: xerj_common::config::EmbeddingConfig,@@ -5851,6 +6030,7 @@ impl Index {)),merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),merge_config: config.merge.clone(),+ compression_config: config.compression.clone(),embedding_config: config.embedding.clone(),max_fields_per_index: config.limits.max_fields_per_index,embedder: Arc::new(RwLock::new(effective_embedder)),@@ -6204,6 +6384,7 @@ impl Index {)),merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),merge_config: config.merge.clone(),+ compression_config: config.compression.clone(),embedding_config: config.embedding.clone(),max_fields_per_index: config.limits.max_fields_per_index,embedder: Arc::new(RwLock::new(effective_embedder)),@@ -8479,6 +8660,12 @@ impl Index {let excluded_fts_fields_for_task = excluded_fts_fields.clone();let segments_dir_for_task = segments_dir.clone();let dv_skip_for_merge = dv_skip.clone();+ // #318 — the operator's `compression.level`, resolved once per+ // merge task. Merge is the only writer that honours it: it runs+ // on the nice-+15 SMALL pool, off the ingest critical path, which+ // is precisely why raising the effort here is safe and raising it+ // on flush was not.+ let merge_zstd_level = self.compression_config.level.zstd_level();let batch_for_task = batch;let metas_for_task = metas;let failed_for_task = Arc::clone(&failed_batches);@@ -8813,7 +9000,10 @@ impl Index {return None;}};- let encoded = xerj_storage::stored_codec::encode_stored_v2(&merged_json_buf);+ let encoded = xerj_storage::stored_codec::encode_stored_v2_at_level(+ &merged_json_buf,+ merge_zstd_level,+ );drop(merged_json_buf);if let Err(e) = writer.add_section(SectionType::Stored, &encoded) {tracing::error!("merge ABORTED: failed to add section: {e}");@@ -8874,7 +9064,8 @@ impl Index {&segments_dir_for_task,merged_meta.id.as_str(),Arc::clone(®istry_for_task),- );+ )+ .with_zstd_level(merge_zstd_level);for (field_name, cfg) in &field_configs_for_task {fts_writer.configure_field(field_name.clone(), cfg.clone());}@@ -8952,10 +9143,11 @@ impl Index {&dv_skip_for_merge,);if !columns.is_empty() {- if let Err(e) = write_doc_values_sidecar(+ if let Err(e) = write_doc_values_sidecar_at_level(&segments_dir_for_task,merged_meta.id.as_str(),&columns,+ merge_zstd_level,) {tracing::warn!("merge: doc-values write failed: {e}");}@@ -18778,9 +18970,26 @@ fn write_doc_values_sidecar(segments_dir: &std::path::Path,segment_id: &str,columns: &std::collections::BTreeMap<String, xerj_storage::doc_values::Column>,+) -> std::io::Result<()> {+ write_doc_values_sidecar_at_level(+ segments_dir,+ segment_id,+ columns,+ xerj_storage::doc_values::DV_ZSTD_LEVEL,+ )+}++/// [`write_doc_values_sidecar`] at a caller-chosen zstd level — the merge+/// path's entry point for the operator's `compression.level` (#318). The+/// flush path keeps the pinned level; see `DV_ZSTD_LEVEL`.+fn write_doc_values_sidecar_at_level(+ segments_dir: &std::path::Path,+ segment_id: &str,+ columns: &std::collections::BTreeMap<String, xerj_storage::doc_values::Column>,+ level: i32,) -> std::io::Result<()> {let path = segments_dir.join(format!("{segment_id}.dv"));- let bytes = xerj_storage::doc_values::encode_columns(columns);+ let bytes = xerj_storage::doc_values::encode_columns_at_level(columns, level);xerj_common::fsio::write_file_durable(&path, &bytes)}diff --git a/engine/crates/xerj-fts/src/index.rs b/engine/crates/xerj-fts/src/index.rsindex 696bdc374..02a0c083d 100644--- a/engine/crates/xerj-fts/src/index.rs+++ b/engine/crates/xerj-fts/src/index.rs@@ -320,7 +320,12 @@ const POST_MAGIC_ZSTD: &[u8; 4] = b"ZPS1";/// (merge-dominated long-term storage barely changes; only the/// freshest tier-0 segments are larger before they merge). See/// `engine/reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`.-const ZSTD_DURABLE_LEVEL: i32 = 3;+///+/// This is the **flush** level and the writer default. A merge-time caller+/// raises it via [`FtsIndexWriter::with_zstd_level`] to honour the operator's+/// `compression.level` (#318) — merge is off the ingest critical path, which+/// is exactly the distinction the paragraph above is about.+pub const ZSTD_DURABLE_LEVEL: i32 = 3;const ZFM3_RECORD_LEN: usize = 4 + 8 + 8 + 4; // 24 bytes: df, ttf, off, len/// Header byte length for a ZFM3 file (magic + total_docs + total_field_length@@ -357,6 +362,7 @@ fn encode_field_meta_v4(has_positions: bool,sorted_terms: &[String],term_postings: &HashMap<String, TermPostings>,+ zstd_level: i32,) -> Result<Vec<u8>> {let num_terms = sorted_terms.len();// Build the records section in the same byte layout that ZFM3@@ -380,7 +386,7 @@ fn encode_field_meta_v4(}let uncompressed_len = records.len() as u32;let compressed =- zstd::bulk::compress(&records, ZSTD_DURABLE_LEVEL).with_context(|| "ZFM4 zstd compress")?;+ zstd::bulk::compress(&records, zstd_level).with_context(|| "ZFM4 zstd compress")?;let mut out: Vec<u8> = Vec::with_capacity(ZFM3_HEADER_LEN + 4 + 4 + compressed.len());out.extend_from_slice(META_MAGIC_V4);out.write_u64::<LittleEndian>(stats.total_docs).unwrap();@@ -555,6 +561,9 @@ pub struct FtsIndexWriter {/// Per-field: (config, postings_writer, field_stats, norms)fields: HashMap<String, FieldData>,encoded_filename_layout_published: bool,+ /// Zstd effort for the `.meta` (ZFM4) and `.post` (ZPS1) envelopes.+ /// [`ZSTD_DURABLE_LEVEL`] unless [`Self::with_zstd_level`] raised it.+ zstd_level: i32,}struct FieldData {@@ -578,9 +587,23 @@ impl FtsIndexWriter {registry,fields: HashMap::new(),encoded_filename_layout_published: false,+ zstd_level: ZSTD_DURABLE_LEVEL,}}+ /// Re-encode this segment's `.meta` / `.post` at `level` instead of the+ /// flush default [`ZSTD_DURABLE_LEVEL`].+ ///+ /// For merge callers only — flush must not raise the level, and+ /// [`ZSTD_DURABLE_LEVEL`] records the measured reason. Both envelopes+ /// stay byte-format-identical across levels (zstd decode does not consult+ /// the level a payload was written at), so this changes only how hard the+ /// encoder works and how large the result is.+ pub fn with_zstd_level(mut self, level: i32) -> Self {+ self.zstd_level = level;+ self+ }+/// Register a field with its indexing configuration./// Must be called before `add_document` uses this field.pub fn configure_field(&mut self, field: impl Into<String>, config: FieldIndexConfig) {@@ -847,6 +870,7 @@ impl FtsIndexWriter {let segment_dir = self.segment_dir.clone();let segment_id = self.segment_id.clone();+ let zstd_level = self.zstd_level;// Drain fields into a Vec so we can parallelise the iterator.// Cloning `stats` before consuming `field_data` — `stats` goes into@@ -866,8 +890,14 @@ impl FtsIndexWriter {let results: Vec<Result<(String, FieldStats)>> = fields.into_par_iter().map(|(field_name, stats, field_data)| {- Self::write_field_static(&segment_dir, &segment_id, &field_name, field_data)- .with_context(|| format!("writing field '{}'", field_name))?;+ Self::write_field_static(+ &segment_dir,+ &segment_id,+ &field_name,+ field_data,+ zstd_level,+ )+ .with_context(|| format!("writing field '{}'", field_name))?;Ok((field_name, stats))}).collect();@@ -887,6 +917,7 @@ impl FtsIndexWriter {segment_id: &str,field_name: &str,field_data: FieldData,+ zstd_level: i32,) -> Result<()> {// NOTE: `PathBuf::with_extension` replaces the final `.ext` in the path,// so using `segment_dir.join("segment_id.field_name")` followed by@@ -959,7 +990,7 @@ impl FtsIndexWriter {Vec::new()} else {let uncompressed_len = post_data.len() as u32;- let compressed = zstd::bulk::compress(&post_data, ZSTD_DURABLE_LEVEL)+ let compressed = zstd::bulk::compress(&post_data, zstd_level).with_context(|| "ZPS1 zstd compress")?;let mut out = Vec::with_capacity(4 + 4 + compressed.len());out.extend_from_slice(POST_MAGIC_ZSTD);@@ -1030,6 +1061,7 @@ impl FtsIndexWriter {has_positions,&sorted_terms,&term_postings,+ zstd_level,)?;xerj_common::fsio::write_file_durable(&meta_path, &meta_bytes).with_context(|| format!("writing meta to {:?}", meta_path))?;diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rsindex 0f13c4e02..af88bd1c5 100644--- a/engine/crates/xerj-server/src/main.rs+++ b/engine/crates/xerj-server/src/main.rs@@ -441,6 +441,14 @@ fn load_config(args: &CliArgs) -> Result<Config> {warn!("{key} is set but has no effect in this build: {effect}");}+ // Same treatment for `[compression]` (#318): `level` is wired into the+ // merge re-encode, `enabled` and `block_size_docs` are not, and an+ // operator who set all three to trade CPU for $/GB previously got three+ // no-ops in silence.+ for (key, effect) in cfg.compression.dormant_overrides() {+ warn!("{key} is set but has no effect in this build: {effect}");+ }+// Bind address override: `--bind` flag or `XERJ_BIND_ADDRESS` env (flag// wins). Both exist because the default is loopback (#228): without a way// to say "expose me" that is not a TOML file, a container image or adiff --git a/engine/crates/xerj-storage/src/doc_values.rs b/engine/crates/xerj-storage/src/doc_values.rsindex 35cc6d43c..61e817824 100644--- a/engine/crates/xerj-storage/src/doc_values.rs+++ b/engine/crates/xerj-storage/src/doc_values.rs@@ -626,7 +626,24 @@ fn roaring_retained_bytes(bitmap: &RoaringBitmap) -> u64 {const KIND_FLAG_ZSTD: u8 = 0x80;const KIND_MASK: u8 = 0x7F;+/// Zstd level the flush path encodes `.dv` columns at. Pinned for the reason+/// on `stored_codec::STORED_ZSTD_LEVEL`: flush is the back-pressure-critical+/// bound on sustained ingest, and this used to be one of three stacked+/// level-19 encoders in a single flush.+pub const DV_ZSTD_LEVEL: i32 = 3;++/// Encode at [`DV_ZSTD_LEVEL`]; merge calls [`encode_columns_at_level`] to+/// honour `compression.level` (#318).pub fn encode_columns(columns: &BTreeMap<String, Column>) -> Vec<u8> {+ encode_columns_at_level(columns, DV_ZSTD_LEVEL)+}++/// [`encode_columns`] at a caller-chosen zstd effort level.+///+/// Level is a write-side choice only: the reader dispatches on the+/// `KIND_FLAG_ZSTD` bit and zstd decode is level-independent, so `.dv` files+/// written at different levels coexist in one index with no format change.+pub fn encode_columns_at_level(columns: &BTreeMap<String, Column>, level: i32) -> Vec<u8> {let mut out = Vec::new();out.write_u32::<LittleEndian>(MAGIC).unwrap();out.write_u32::<LittleEndian>(columns.len() as u32).unwrap();@@ -639,16 +656,19 @@ pub fn encode_columns(columns: &BTreeMap<String, Column>) -> Vec<u8> {Column::Numeric(n) => n.encode(),Column::Keyword(k) => k.encode(),};- // Zstd level 3 — fast encode (~250 MB/s/core). Reverted from- // level 19 because the flush path needs to keep up with sustained- // ingest (1 M+ docs/s) and the level-19 pass collapsed throughput- // by 70-100× under continuous load. See+ // `level` is DV_ZSTD_LEVEL (3) on flush — fast encode+ // (~250 MB/s/core), reverted from level 19 because the flush path+ // needs to keep up with sustained ingest (1 M+ docs/s) and the+ // level-19 pass collapsed throughput by 70-100× under continuous+ // load. See// `engine/reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`.+ // Merge passes the operator's `compression.level` instead, which is+ // off the ingest critical path.// Legacy LZ4-framed column payloads are still auto-detected by// the reader via the high bit on the kind byte; level-19-encoded// segments from older builds remain readable (zstd decompress is// level-independent).- let zstd_payload = match zstd::encode_all(&payload[..], 3) {+ let zstd_payload = match zstd::encode_all(&payload[..], level) {Ok(c) if c.len() < payload.len() => c,// If compression doesn't help (rare tiny columns), keep the// zstd path anyway — decompression still works and avoids adiff --git a/engine/crates/xerj-storage/src/stored_codec.rs b/engine/crates/xerj-storage/src/stored_codec.rsindex 0ddbe2487..920411060 100644--- a/engine/crates/xerj-storage/src/stored_codec.rs+++ b/engine/crates/xerj-storage/src/stored_codec.rs@@ -165,7 +165,12 @@ const CROSS_DEP_MAX_SRC_CARDINALITY: usize = 256;/// the on-flush overhead is only paid by the freshest tier-0 segments/// before they merge anyway). See/// `engine/reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`.-const STORED_ZSTD_LEVEL: i32 = 3;+///+/// This is now the level used **on flush**, which the operator cannot raise —+/// that is the whole content of the paragraph above. Merge re-encodes at+/// whatever `compression.level` resolves to (#318); see+/// [`encode_stored_v2_at_level`].+pub const STORED_ZSTD_LEVEL: i32 = 3;#[repr(u8)]#[derive(Copy, Clone, Debug, PartialEq)]@@ -196,7 +201,22 @@ impl ColCodec {/// On tiny segments (< `V2_MIN_DOCS` docs) or on any shape that doesn't/// fit the column assumptions, the function silently falls back to the/// v1 LZ4 encoder — the caller never has to care which was used.+///+/// Encodes at [`STORED_ZSTD_LEVEL`]; merge calls+/// [`encode_stored_v2_at_level`] instead to honour `compression.level`.pub fn encode_stored_v2(stored_docs_json: &[u8]) -> Vec<u8> {+ encode_stored_v2_at_level(stored_docs_json, STORED_ZSTD_LEVEL)+}++/// [`encode_stored_v2`] at a caller-chosen zstd effort level.+///+/// Only the merge path passes anything but [`STORED_ZSTD_LEVEL`] — flush is+/// back-pressure-critical and stays pinned (see that constant). The output is+/// the same ZBS2 format at any level: zstd decode does not need to know the+/// level a payload was written at, so segments written before and after an+/// operator changes `compression.level` interleave with no migration and no+/// format flag.+pub fn encode_stored_v2_at_level(stored_docs_json: &[u8], level: i32) -> Vec<u8> {// Parse the JSON array of documents. Failure → v1 fallback.let docs: Vec<serde_json::Value> = match serde_json::from_slice(stored_docs_json) {Ok(v) => v,@@ -250,7 +270,13 @@ pub fn encode_stored_v2(stored_docs_json: &[u8]) -> Vec<u8> {}}- encode_v2_columns(num_docs, &col_order, &columns, Some(stored_docs_json))+ encode_v2_columns(+ num_docs,+ &col_order,+ &columns,+ Some(stored_docs_json),+ level,+ )}/// P2.2 — columnar V2 encoder fed directly from already-parsed values.@@ -394,7 +420,16 @@ fn encode_stored_v2_from_values_inner();}- encode_v2_columns(num_docs, &col_order, &columns, stored_docs_json)+ // Flush-only entry point — pinned to the flush level, never the+ // operator's (#318): raising the level here is the ingest regression+ // documented on `STORED_ZSTD_LEVEL`.+ encode_v2_columns(+ num_docs,+ &col_order,+ &columns,+ stored_docs_json,+ STORED_ZSTD_LEVEL,+ )}/// Shared tail of the two V2 entry points: per-column codec selection +@@ -409,6 +444,7 @@ fn encode_v2_columns(col_order: &[String],columns: &[Vec<&serde_json::Value>],stored_docs_json: Option<&[u8]>,+ level: i32,) -> Vec<u8> {// THROWAWAY prof (XERJ_PROF): encode-phase breakdown.let prof = std::env::var_os("XERJ_PROF").is_some();@@ -497,7 +533,7 @@ fn encode_v2_columns(continue;}if let Some(src_ix) = cross_dep_src[cix] {- let (payload, ok) = encode_cross_dep(col, src_ix, &dict_encoded);+ let (payload, ok) = encode_cross_dep(col, src_ix, &dict_encoded, level);if ok {col_payloads.push((cname.clone(), ColCodec::CrossDep as u8, payload));continue;@@ -505,15 +541,15 @@ fn encode_v2_columns(}if let Some((entries, ids)) = &dict_encoded[cix] {if entries.len() <= DICT_MAX_CARDINALITY && all_scalar_dict_entries(entries) {- let payload = encode_dict_bitpack(entries, ids);+ let payload = encode_dict_bitpack(entries, ids, level);col_payloads.push((cname.clone(), ColCodec::DictBitpack as u8, payload));continue;}}// Fallback: zstd over JSON-array of the column's values.let col_json = serde_json::to_vec(col).unwrap_or_default();- let zstd_payload = zstd::encode_all(Cursor::new(&col_json), STORED_ZSTD_LEVEL)- .unwrap_or_else(|_| col_json.clone());+ let zstd_payload =+ zstd::encode_all(Cursor::new(&col_json), level).unwrap_or_else(|_| col_json.clone());// Choose RAW_JSON vs LZ4_JSON by size.let lz4_payload = lz4_flex::compress_prepend_size(&col_json);if lz4_payload.len() + 1 < zstd_payload.len() {@@ -2041,7 +2077,7 @@ fn decode_constant(payload: &[u8], num_docs: usize) -> Result<Vec<serde_json::VaOk(values)}-fn encode_dict_bitpack(entries: &[serde_json::Value], ids: &[u32]) -> Vec<u8> {+fn encode_dict_bitpack(entries: &[serde_json::Value], ids: &[u32], level: i32) -> Vec<u8> {let dict_count = entries.len();// Reserve one extra id for null → dict_count (which might be past the// packed range, so we allocate `bit_width` large enough for dict_count@@ -2059,8 +2095,7 @@ fn encode_dict_bitpack(entries: &[serde_json::Value], ids: &[u32]) -> Vec<u8> {// Dict entries as zstd(json array).let dict_json = serde_json::to_vec(entries).unwrap_or_default();- let dict_zstd =- zstd::encode_all(Cursor::new(&dict_json), STORED_ZSTD_LEVEL).unwrap_or(dict_json.clone());+ let dict_zstd = zstd::encode_all(Cursor::new(&dict_json), level).unwrap_or(dict_json.clone());out.write_u32::<LittleEndian>(dict_zstd.len() as u32).unwrap();out.extend_from_slice(&dict_zstd);@@ -2069,8 +2104,7 @@ fn encode_dict_bitpack(entries: &[serde_json::Value], ids: &[u32]) -> Vec<u8> {let packed = bitpack_u32(ids, bit_width);// zstd over the bit-packed stream — gives another 20-40 % on log// data because repeated ids cluster.- let packed_zstd =- zstd::encode_all(Cursor::new(&packed), STORED_ZSTD_LEVEL).unwrap_or(packed.clone());+ let packed_zstd = zstd::encode_all(Cursor::new(&packed), level).unwrap_or(packed.clone());out.write_u32::<LittleEndian>(ids.len() as u32).unwrap();out.write_u32::<LittleEndian>(packed_zstd.len() as u32).unwrap();@@ -2166,6 +2200,7 @@ fn encode_cross_dep<B: std::borrow::Borrow<serde_json::Value>>(target_col: &[B],src_ix: usize,dict_encoded: &[Option<(Vec<serde_json::Value>, Vec<u32>)>],+ level: i32,) -> (Vec<u8>, bool) {let (_src_entries, src_ids) = match &dict_encoded[src_ix] {Some(e) => e,@@ -2234,7 +2269,7 @@ fn encode_cross_dep<B: std::borrow::Borrow<serde_json::Value>>(prev_ord = *ord;}// zstd the whole thing.- let compressed = zstd::encode_all(Cursor::new(&out), STORED_ZSTD_LEVEL).unwrap_or(out.clone());+ let compressed = zstd::encode_all(Cursor::new(&out), level).unwrap_or(out.clone());// Save the smaller of raw vs zstd.let mut final_out = Vec::with_capacity(1 + compressed.len().max(out.len()));if compressed.len() + 1 < out.len() {diff --git a/engine/crates/xerj-storage/src/stored_codec/row_hydration.rs b/engine/crates/xerj-storage/src/stored_codec/row_hydration.rsindex b1d0295eb..2a7df97d2 100644--- a/engine/crates/xerj-storage/src/stored_codec/row_hydration.rs+++ b/engine/crates/xerj-storage/src/stored_codec/row_hydration.rs@@ -871,7 +871,7 @@ mod tests {("__id",ColCodec::DictBitpack,- encode_dict_bitpack(&values, &ids),+ encode_dict_bitpack(&values, &ids, STORED_ZSTD_LEVEL),),("__seq_no",@@ -902,7 +902,7 @@ mod tests {assert_eq!(unpack_id(&packed[..packed.len() - 1], width, 2), None);}- let mut corrupt = encode_dict_bitpack(&[json!("only")], &[0, 0, 0]);+ let mut corrupt = encode_dict_bitpack(&[json!("only")], &[0, 0, 0], STORED_ZSTD_LEVEL);// Rebuild the packed stream with id=3, which is outside {entry,null}.let dict_len = u32::from_le_bytes(corrupt[5..9].try_into().unwrap()) as usize;let packed_offset = 9 + dict_len + 8;@@ -926,7 +926,7 @@ mod tests {);assert!(decode_stored_v2_rows(&bad, &[0]).is_err());- let mut padding = encode_dict_bitpack(&[json!("only")], &[0, 0, 0]);+ let mut padding = encode_dict_bitpack(&[json!("only")], &[0, 0, 0], STORED_ZSTD_LEVEL);let dict_len = u32::from_le_bytes(padding[5..9].try_into().unwrap()) as usize;let packed_length_offset = 9 + dict_len + 4;let packed_offset = packed_length_offset + 4;@@ -1226,7 +1226,7 @@ mod tests {("__id",ColCodec::DictBitpack,- encode_dict_bitpack(&[json!("a"), json!("b")], &[0, 1, 0, 1]),+ encode_dict_bitpack(&[json!("a"), json!("b")], &[0, 1, 0, 1], STORED_ZSTD_LEVEL),),("__seq_no", ColCodec::CrossDep, dependent),],diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex f4f667f7a..6de9bf0b8 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -230,23 +230,36 @@ max_merge_count = 16[compression]-# Enable block compression for stored document fields.-# Disabling compression increases disk usage but reduces CPU overhead on read.+# ACCEPTED BUT NOT WIRED. Every durable artifact xerj writes is a compressed+# envelope — stored (ZBS2), .meta (ZFM4), .post (ZPS1), .dv — and none of them+# has an uncompressed write path; the legacy uncompressed layouts survive only+# so old segments stay readable. Setting this to false logs a warning at+# startup and changes nothing on disk.enabled = true-# Compression algorithm and level.+# Zstandard effort level, applied when segments are re-encoded AT MERGE.#-# "fast" — LZ4. Maximum throughput (~500 MB/s), moderate compression ratio.-# Best for hot indexes accessed frequently.-# "balanced" — Zstandard level 3. Good ratio (~3–4×) with low CPU overhead.-# Recommended for most workloads.-# "best" — Zstandard level 19. Maximum ratio (~5–6×) at higher CPU cost.-# Best for cold storage or archival indexes.+# "fast" — Zstd level 1. Fastest encode, loosest ratio.+# "balanced" — Zstd level 3. Recommended for most workloads.+# "best" — Zstd level 6. Tightest measured ratio per byte of decode+# window. Best for cold storage or archival indexes.+#+# Flush ignores this and always writes level 3. Flush is the back-pressure-+# critical bound on sustained ingest: raising its level to 19 once collapsed+# throughput from 1.55M docs/s to 21k docs/s with 75% of documents rejected+# (reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md). Merge runs+# off that path and produces the long-lived segments where footprint matters.+#+# A change therefore takes effect as segments merge, not at restart. To apply+# it to data already on disk, re-merge: POST /{index}/_forcemerge. Segments+# written at different levels coexist — zstd decode does not consult the level+# a payload was written at, so no migration or reindex is needed.level = "balanced"-# Number of documents grouped into a single compressed block.-# Larger blocks compress better (more context) but increase read amplification-# when fetching individual documents. Range: 16 – 4096.+# ACCEPTED BUT NOT WIRED. The stored codec is columnar over the whole segment+# section rather than blocked by document count, so there is no doc-block for+# this to size. Still range-checked (16 – 4096) and warned about when moved off+# the default, so a typo fails at startup instead of being ignored.block_size_docs = 128# ── Full-Text Search (1 setting) ──────────────────────────────────────────────diff --git a/landing/docs/compression.html b/landing/docs/compression.htmlindex 34157e50f..a0f8dcbf0 100644--- a/landing/docs/compression.html+++ b/landing/docs/compression.html@@ -122,13 +122,15 @@<p>XERJ uses two layers of compression. <strong>Column encodings</strong> are chosen per-field at write time based on the column's shape. <strong>Block compression</strong> runs over the encoded column data. The two together give XERJ a smaller on-disk footprint than Elasticsearch — measured 1.61× smaller vs ES 8.13.4 on the reproducible telemetry-corpus scorecard (the exact ratio varies by data shape).</p><h2>Block compression</h2>- <p>Three levels, controlled by <code class="inline">[compression] level</code>:</p>+ <p>Three Zstandard effort levels, controlled by <code class="inline">[compression] level</code>:</p><ul>- <li><strong>fast</strong> — LZ4. ~500 MB/s throughput. Use for write-heavy paths.</li>- <li><strong>balanced</strong> — Zstd L3. Default. ~3-4× ratio.</li>- <li><strong>best</strong> — Zstd L19. ~5-6× ratio. Use for cold indices you query rarely.</li>+ <li><strong>fast</strong> — Zstd L1. Fastest encode, loosest ratio.</li>+ <li><strong>balanced</strong> — Zstd L3. Default.</li>+ <li><strong>best</strong> — Zstd L6. Tightest measured ratio per byte of decode window. Use for cold indices you query rarely.</li></ul>- <p><code class="inline">block_size_docs</code> (16-4096, default 128) controls the compression block size.</p>+ <p><strong>The level applies at merge, not at flush.</strong> Freshly flushed segments are always written at L3: flush is the back-pressure-critical bound on ingest, and raising its level once collapsed sustained throughput from 1.55M docs/s to 21k docs/s with 75% of documents rejected. Merges run off that path, and they are what produces the long-lived segments where footprint actually matters. So a level change takes effect as segments merge, not the instant you restart — and re-merging an existing index (<code class="inline">POST /{index}/_forcemerge</code>) applies it to data already on disk.</p>+ <p>Levels are a write-side choice only. Zstd decoding does not depend on the level a payload was written at, so segments written at different levels coexist in one index with no migration and no reindex.</p>+ <p><code class="inline">enabled</code> and <code class="inline">block_size_docs</code> are <strong>accepted but not wired</strong>. Every durable artifact XERJ writes is a compressed envelope with no uncompressed write path, so <code class="inline">enabled = false</code> does nothing; and the stored codec is columnar over the whole segment section rather than blocked by document count, so there is no doc-block for <code class="inline">block_size_docs</code> (16-4096, default 128) to size. Setting either off its default logs a warning at startup. Both were silently ignored before <a href="https://github.com/xerj-org/xerj/issues/318">#318</a>.</p><h2>Column encodings</h2><p>Picked automatically at ingest time from this set:</p>diff --git a/landing/docs/config.html b/landing/docs/config.htmlindex 87495f675..ef4a7e18a 100644--- a/landing/docs/config.html+++ b/landing/docs/config.html@@ -124,7 +124,7 @@<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p><h2>Full key table</h2>- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div>+ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div><h2 id="server-section">[server]</h2>@@ -176,11 +176,11 @@ min_merge_count = 4 # the real per-tier merge triggermax_merge_count = 16 # caps peak merge RAM</pre><h2 id="compression-section">[compression]</h2>- <p>See <a href="/docs/compression.html">Compression</a> for the encoding catalog. This section picks the outer <em>block</em> codec only — the inner per-column encodings are chosen automatically at write time.</p>+ <p>See <a href="/docs/compression.html">Compression</a> for the encoding catalog. This section picks the outer <em>block</em> codec effort only — the inner per-column encodings are chosen automatically at write time.</p><pre class="code">[compression]-enabled = true-level = "balanced" # LZ4 / Zstd L3 / Zstd L19-block_size_docs = 128</pre>+enabled = true # accepted, not wired — no uncompressed write path exists+level = "balanced" # Zstd L1 / L3 / L6, applied at MERGE (flush is always L3)+block_size_docs = 128 # accepted, not wired — 16-4096, range-checked at startup</pre><h2 id="fts-section">[fts]</h2><p>Default analyzer applied to untyped <code class="inline">text</code> fields. Override per-field in the mapping when creating an index. See <a href="/docs/analyzers.html">Analyzers</a> for the built-ins.</p>diff --git a/landing/docs/index.html b/landing/docs/index.htmlindex 1352d1295..ecbb4e4fc 100644--- a/landing/docs/index.html+++ b/landing/docs/index.html@@ -252,7 +252,7 @@<span class="label">ENGINE</span><a href="/docs/ingest.html">Ingest pipelines<span class="d">Turbo, logs, syslog, OTLP — in-process</span></a><a href="/docs/storage.html">Storage & WAL<span class="d">WAL + immutable segments · three files per segment</span></a>- <a href="/docs/compression.html">Compression<span class="d">LZ4 · Zstd L3 · Zstd L19 · 9 encodings</span></a>+ <a href="/docs/compression.html">Compression<span class="d">Zstd L1 · L3 · L6 · 9 encodings</span></a><a href="/docs/clustering.html">Clustering<span class="d">Embedded Raft · metadata replication · region split/merge</span></a><a href="/docs/metrics.html">Metrics<span class="d">Prometheus text · counters · histograms · gauges</span></a><a href="/docs/security.html">Auth & TLS<span class="d">API-key auth · PEM certs</span></a>