Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(compression): wire compression.level into the merge encoder, and say so where it still isn't

xerj-orgde73a2c56398
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 7ced3a626..da57ced40 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -182,6 +182,20 @@ impl Config {
return Err(XerjError::config("merge.min_segments must be >= 2"));
}
+ // Compression: block_size_docs is documented as 16–4096 and was
+ // accepted at any value (#318 got `999999` past startup in silence).
+ // Range-check it even though the knob is dormant: an out-of-range
+ // value is a typo the operator wants to hear about now, and the check
+ // must not start passing quietly if the knob is ever wired.
+ if !CompressionConfig::BLOCK_SIZE_DOCS_RANGE.contains(&self.compression.block_size_docs) {
+ return Err(XerjError::config(format!(
+ "compression.block_size_docs must be in {}..={}, got {}",
+ CompressionConfig::BLOCK_SIZE_DOCS_RANGE.start(),
+ CompressionConfig::BLOCK_SIZE_DOCS_RANGE.end(),
+ self.compression.block_size_docs
+ )));
+ }
+
// Vector: hnsw_ef_construction >= hnsw_m
if self.vector.hnsw_ef_construction < self.vector.hnsw_m {
return Err(XerjError::config(
@@ -908,14 +922,28 @@ pub enum MergeStrategy {
#[serde(default, deny_unknown_fields)]
pub struct CompressionConfig {
/// Enable block compression for stored fields (default: `true`).
+ ///
+ /// **Dormant.** Every durable artifact XERJ writes is a self-describing
+ /// compressed envelope (`ZBS2` stored, `ZFM4` `.meta`, `ZPS1` `.post`,
+ /// the zstd-flagged `.dv` columns); "off" would mean emitting the legacy
+ /// uncompressed layouts, which only exist as read-side compatibility
+ /// paths. Setting `false` warns at startup — see
+ /// [`CompressionConfig::dormant_overrides`].
pub enabled: bool,
/// Compression level: `"fast"`, `"balanced"`, or `"best"` (default: `"balanced"`).
///
- /// Uses LZ4 for `"fast"` and Zstandard for `"balanced"` / `"best"`.
+ /// Selects the Zstandard effort level used to re-encode a segment's
+ /// durable artifacts **at merge time**. See [`CompressionLevel`] for the
+ /// level each name maps to, and why flush ignores this knob.
pub level: CompressionLevel,
/// Number of documents per compressed block (default: `128`).
///
- /// Larger blocks compress better but increase random read amplification.
+ /// **Dormant.** Kept because it has shipped in `xerj.default.toml` since
+ /// v0.1, but XERJ's stored codec is columnar over the whole segment
+ /// section rather than blocked by document count, so there is no
+ /// doc-block for this to size. Validated against its documented 16–4096
+ /// range and warned about when moved off the default — see
+ /// [`CompressionConfig::dormant_overrides`].
pub block_size_docs: u32,
}
@@ -929,18 +957,113 @@ impl Default for CompressionConfig {
}
}
-/// Compression level.
+impl CompressionConfig {
+ /// The documented range for `block_size_docs`, quoted in
+ /// `xerj.default.toml` and on `landing/docs/config.html`.
+ pub const BLOCK_SIZE_DOCS_RANGE: std::ops::RangeInclusive<u32> = 16..=4096;
+
+ /// Compression settings this build accepts but does not act on, in the
+ /// same `("compression.key", "what actually happens")` shape as
+ /// [`MergeConfig::dormant_overrides`], and reported the same way — only
+ /// when the operator has moved one off its default.
+ ///
+ /// Issue #318 found the whole `[compression]` section inert: two servers
+ /// differing only in `enabled` / `level` / `block_size_docs` produced
+ /// byte-identical segments (`.seg` 261,481 / `.dv` 10,679 / `.post`
+ /// 192,287 on a 3,000-doc corpus), with no error and no warning, while
+ /// the docs described all three as live. `level` is now wired into the
+ /// merge re-encode; these two are not, and stay accepted-but-warned for
+ /// the reason `merge.io_rate_mb_per_sec` does — both ship non-default in
+ /// no config the project itself hands out, and the cost of ignoring them
+ /// is disk footprint, not data.
+ pub fn dormant_overrides(&self) -> Vec<(&'static str, &'static str)> {
+ let d = CompressionConfig::default();
+ let mut out = Vec::new();
+ if self.enabled != d.enabled {
+ out.push((
+ "compression.enabled",
+ "durable artifacts are always compressed in this build — the \
+ stored (ZBS2), .meta (ZFM4), .post (ZPS1) and .dv envelopes \
+ have no uncompressed write path, only legacy read support",
+ ));
+ }
+ if self.block_size_docs != d.block_size_docs {
+ out.push((
+ "compression.block_size_docs",
+ "nothing reads it — the stored codec is columnar over the \
+ whole segment section, not blocked by document count",
+ ));
+ }
+ out
+ }
+}
+
+/// Compression level — the Zstandard effort applied when a segment's durable
+/// artifacts are re-encoded at **merge**.
+///
+/// Three things about this enum are deliberate and were all decided against
+/// what the docs used to promise (issue #318):
+///
+/// 1. **Merge only.** Flush stays pinned at [`CompressionLevel::Balanced`]'s
+/// level 3 regardless of this setting. Raising the flush level to 19 is
+/// exactly the regression recorded in
+/// `engine/reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`
+/// — 1.55 M docs/s peak collapsed to 21 K docs/s with 75 % of documents
+/// rejected, because a ~50 ms segment flush became 5–10 s and tripped
+/// back-pressure. Merge is off the ingest critical path; flush is not.
+/// 2. **All three names are Zstandard**, not "LZ4 for fast". Every durable
+/// envelope XERJ writes (`ZBS2`, `ZFM4`, `ZPS1`, zstd-flagged `.dv`) is
+/// zstd-framed, and its magic is what the reader dispatches on; making
+/// one level switch algorithm would mean writing the legacy LZ4 layouts
+/// that survive only as read-compatibility paths. `"fast"` is therefore
+/// zstd's own fastest level, not a different codec.
+/// 3. **`"best"` is level 6, not 19.** The RFC #148 thread measured L6 at
+/// −14.4 % stored / −14.3 % `.dv` / −15.4 % `.meta`, while L9 doubles the
+/// decode window from 2.00 to 4.00 MiB and L19 takes it to 8.00 MiB — a
+/// cost every streaming point-get pays on read, forever, for a ratio gain
+/// that measurement did not show. Lucene and Elasticsearch land in the
+/// same place from the other direction: ES's `best_compression` stored
+/// fields format tops out at zstd **3** with a bigger block, not at a
+/// high effort level (`Zstd814StoredFieldsFormat.java:38-46`, read for
+/// approach only — that code is AGPL/SSPL/Elastic-2.0 and none of it is
+/// copied here).
+///
+/// Decoding never depends on the level a segment was written at, so mixed
+/// levels across segments — the normal state after changing this setting —
+/// need no format flag and no migration. That is why XERJ does not need
+/// Lucene's `Lucene90StoredFieldsFormat.MODE_KEY` segment attribute
+/// (`Lucene90StoredFieldsFormat.java:113-137`, Apache-2.0): Lucene must
+/// record the mode because BEST_SPEED and BEST_COMPRESSION are different
+/// algorithms (LZ4 vs Deflate), whereas here they differ only in effort.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum CompressionLevel {
- /// LZ4 — maximum throughput, moderate ratio.
+ /// Zstandard level 1 — maximum throughput, moderate ratio.
Fast,
- /// Zstandard level 3 — good ratio with low CPU overhead.
+ /// Zstandard level 3 — good ratio with low CPU overhead. Also the level
+ /// every flush uses, whatever this setting says.
Balanced,
- /// Zstandard level 19 — maximum ratio, higher CPU cost.
+ /// Zstandard level 6 — best measured ratio per byte of decode window.
Best,
}
+impl CompressionLevel {
+ /// The Zstandard level this name maps to.
+ ///
+ /// The single source of truth for the mapping: the encoder sites in
+ /// `xerj-storage` and `xerj-fts` take an `i32` level, and
+ /// `xerj-compress`'s codec factory routes through here too, so the
+ /// meaning of "best" cannot drift between the config surface and the
+ /// codecs.
+ pub const fn zstd_level(self) -> i32 {
+ match self {
+ Self::Fast => 1,
+ Self::Balanced => 3,
+ Self::Best => 6,
+ }
+ }
+}
+
// ─────────────────────────────────────────────────────────────────────────────
/// Full-text search settings.
@@ -2126,6 +2249,94 @@ mod tests {
);
}
+ /// #318 — the `[compression]` section reached no encoder at all: two
+ /// nodes differing only in `enabled` / `level` / `block_size_docs` wrote
+ /// byte-identical segments, with no error and no warning, while the docs
+ /// called all three live. `level` is wired now (see the engine-side merge
+ /// test); the other two are dormant, and dormant must be *audible*.
+ #[test]
+ fn dormant_compression_settings_are_named_only_when_an_operator_sets_them() {
+ assert!(
+ CompressionConfig::default().dormant_overrides().is_empty(),
+ "an untouched default asks for nothing, so it must not warn"
+ );
+
+ for (toml, expected) in [
+ ("enabled = false", "compression.enabled"),
+ ("block_size_docs = 512", "compression.block_size_docs"),
+ ] {
+ let cfg = Config::from_toml_str(&format!("[compression]\n{toml}\n")).unwrap();
+ let named: Vec<&str> = cfg
+ .compression
+ .dormant_overrides()
+ .into_iter()
+ .map(|(key, _)| key)
+ .collect();
+ assert_eq!(
+ named,
+ vec![expected],
+ "setting {toml} must be reported, and nothing else"
+ );
+ }
+
+ // `level` now reaches the merge re-encode, so it must NOT be reported
+ // — a warning on a knob that works is the same lie in the other
+ // direction.
+ let cfg = Config::from_toml_str("[compression]\nlevel = \"best\"\n").unwrap();
+ assert!(
+ cfg.compression.dormant_overrides().is_empty(),
+ "compression.level reaches the merge encoder, so it is not dormant"
+ );
+ }
+
+ /// The documented 16–4096 range was never enforced: #318's repro node
+ /// booted happily on `block_size_docs = 999999`.
+ #[test]
+ fn out_of_range_block_size_docs_is_refused_at_startup() {
+ for bad in [15u32, 4097, 999_999, 0] {
+ // Rejected on the path a real boot takes — `from_toml_str`
+ // validates, so #318's `block_size_docs = 999999` node would not
+ // have started.
+ let err = Config::from_toml_str(&format!("[compression]\nblock_size_docs = {bad}\n"))
+ .expect_err("{bad} is outside the documented 16..=4096 range");
+ assert!(
+ err.to_string().contains("compression.block_size_docs"),
+ "the error must name the setting, got: {err}"
+ );
+
+ // And directly, so the check cannot be lost by a refactor that
+ // moves parsing off `validate`.
+ let mut cfg = Config::default();
+ cfg.compression.block_size_docs = bad;
+ assert!(cfg.validate().is_err(), "{bad} must fail validate()");
+ }
+
+ for ok in [16u32, 128, 4096] {
+ Config::from_toml_str(&format!("[compression]\nblock_size_docs = {ok}\n"))
+ .unwrap_or_else(|e| panic!("block_size_docs = {ok} is in range, got: {e}"));
+ }
+ }
+
+ /// The zstd level behind each name. Pinned because these numbers are
+ /// quoted in `xerj.default.toml` and on `landing/docs/compression.html`,
+ /// and because `"best"` deliberately means 6 rather than the 19 the enum
+ /// used to document — see the doc comment for the measurements.
+ #[test]
+ fn compression_level_names_map_to_the_documented_zstd_levels() {
+ assert_eq!(CompressionLevel::Fast.zstd_level(), 1);
+ assert_eq!(CompressionLevel::Balanced.zstd_level(), 3);
+ assert_eq!(CompressionLevel::Best.zstd_level(), 6);
+
+ // The default must stay byte-for-byte what flush already writes, so
+ // that adopting this build changes nothing for an operator who never
+ // touched the section.
+ assert_eq!(
+ CompressionConfig::default().level.zstd_level(),
+ 3,
+ "the default level must equal the pinned flush level"
+ );
+ }
+
#[test]
fn onnx_throughput_controls_preserve_single_session_defaults() {
let cfg = Config::from_toml_str("[embedding]\n").unwrap();
diff --git a/engine/crates/xerj-compress/src/codec.rs b/engine/crates/xerj-compress/src/codec.rs
index 11cbe8052..57f032054 100644
--- a/engine/crates/xerj-compress/src/codec.rs
+++ b/engine/crates/xerj-compress/src/codec.rs
@@ -1,5 +1,6 @@
//! Compression codecs: LZ4, Zstd, and passthrough.
+use xerj_common::config::CompressionLevel as ConfigLevel;
use xerj_common::XerjError;
/// Result alias for codec operations.
@@ -27,7 +28,15 @@ pub trait Codec: Send + Sync {
// Compression level
// ─────────────────────────────────────────────────────────────────────────────
-/// Selects the compression algorithm and effort level.
+/// Selects the compression algorithm and effort level for the block codecs in
+/// this crate.
+///
+/// Not the operator-facing type: `compression.level` in the config file
+/// deserialises into [`xerj_common::config::CompressionLevel`], which has no
+/// `None` variant and lives in a crate below this one in the dependency graph.
+/// The two stay in step because the zstd effort each name means is defined
+/// once, in that enum's `zstd_level`, and [`ZstdCodec::balanced`] /
+/// [`ZstdCodec::best`] read it from there.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum CompressionLevel {
/// No compression — raw bytes pass through unchanged.
@@ -35,9 +44,9 @@ pub enum CompressionLevel {
/// LZ4: very fast compression with moderate ratio.
#[default]
Fast,
- /// Zstd level 3: balanced speed vs. ratio.
+ /// Zstd at the `"balanced"` effort level.
Balanced,
- /// Zstd level 19: maximum compression ratio.
+ /// Zstd at the `"best"` effort level.
Best,
}
@@ -118,7 +127,8 @@ impl Codec for Lz4Codec {
/// Zstd-based codec wrapping the [`zstd`] crate.
#[derive(Debug, Clone)]
pub struct ZstdCodec {
- /// Zstd compression level (1–22; defaults to 3 for balanced, 19 for best).
+ /// Zstd compression level, from
+ /// [`xerj_common::config::CompressionLevel::zstd_level`].
level: i32,
}
@@ -127,14 +137,20 @@ impl ZstdCodec {
Self { level }
}
- /// Balanced Zstd (level 3).
+ /// Balanced Zstd — the level `compression.level = "balanced"` selects.
pub fn balanced() -> Self {
- Self::new(3)
+ Self::new(ConfigLevel::Balanced.zstd_level())
}
- /// Maximum Zstd compression (level 19).
+ /// Maximum-ratio Zstd — the level `compression.level = "best"` selects.
+ ///
+ /// This used to be a hardcoded 19 while the config surface's own enum
+ /// documented 19 too and neither reached an encoder (#318). Both now read
+ /// the same table, so "best" cannot mean one thing to an operator and
+ /// another to a codec; see [`ConfigLevel`] for why the measured answer is
+ /// 6 rather than 19.
pub fn best() -> Self {
- Self::new(19)
+ Self::new(ConfigLevel::Best.zstd_level())
}
}
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 2187c51d7..7dee9b176 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -2106,6 +2106,176 @@ mod merge_publication_transaction_tests {
(engine, index, component, segments_dir)
}
+ /// Deterministic, compressible-but-not-trivial documents: a fixed word
+ /// list drawn by an LCG, so every run of this test encodes the same bytes
+ /// and any size difference is attributable to the codec alone.
+ fn compression_fixture_docs(count: usize) -> Vec<serde_json::Value> {
+ const WORDS: [&str; 16] = [
+ "segment",
+ "merge",
+ "posting",
+ "term",
+ "dictionary",
+ "zstandard",
+ "columnar",
+ "envelope",
+ "flush",
+ "durable",
+ "compression",
+ "level",
+ "operator",
+ "ratio",
+ "decode",
+ "window",
+ ];
+ let mut state: u64 = 0x2026_0318;
+ let mut next = || {
+ state = state
+ .wrapping_mul(6364136223846793005)
+ .wrapping_add(1442695040888963407);
+ (state >> 33) as usize
+ };
+ (0..count)
+ .map(|i| {
+ let body: Vec<&str> = (0..40).map(|_| WORDS[next() % WORDS.len()]).collect();
+ serde_json::json!({ "body": format!("document {i} {}", body.join(" ")) })
+ })
+ .collect()
+ }
+
+ /// Ingest the same corpus into a fresh index at `level`, flush two
+ /// segments, merge them, and return `extension → total bytes` for the
+ /// pre-merge (flushed) and post-merge segment families.
+ async fn artifact_bytes_at_level(
+ dir: &TempDir,
+ level: xerj_common::config::CompressionLevel,
+ ) -> (
+ std::collections::BTreeMap<String, u64>,
+ std::collections::BTreeMap<String, u64>,
+ ) {
+ let mut cfg = config(dir);
+ cfg.compression.level = level;
+ // One ingest shard, so each flush publishes exactly one segment and
+ // the merge below is a deterministic 2 → 1. With the default 16
+ // shards a flush yields 16 tiny segments, most below `V2_MIN_DOCS`,
+ // which is the wrong shape for measuring a codec.
+ cfg.engine.ingest_shards = 1;
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("body", FieldType::Text))
+ .unwrap();
+ let engine = Engine::new(cfg).unwrap();
+ engine.create_index("compression", schema).unwrap();
+ let index = engine.get_index("compression").unwrap();
+ index.abort_background_tasks();
+
+ // Two flushes → two segments to merge. Each is comfortably over
+ // `V2_MIN_DOCS` (128), so the stored section takes the columnar v2
+ // path rather than the tiny-segment v1 fallback.
+ let docs = compression_fixture_docs(600);
+ for (half, chunk) in docs.chunks(300).enumerate() {
+ for (i, doc) in chunk.iter().enumerate() {
+ index
+ .index_document(Some(format!("{half}-{i}")), doc.clone())
+ .await
+ .unwrap();
+ }
+ index.flush().await.unwrap();
+ }
+ let segments_dir = index.data_dir.join("segments");
+ let flushed = artifact_bytes_by_extension(&segments_dir);
+
+ assert_eq!(index.store.snapshot().segments.len(), 2);
+ assert_eq!(index.run_merge_once().await.unwrap(), 1);
+ let snapshot = index.store.snapshot();
+ assert_eq!(snapshot.segments.len(), 1);
+ let merged_id = snapshot.segments[0].id.to_string();
+
+ let merged = artifact_bytes_by_extension_for(&segments_dir, &merged_id);
+ (flushed, merged)
+ }
+
+ fn artifact_bytes_by_extension(
+ segments_dir: &std::path::Path,
+ ) -> std::collections::BTreeMap<String, u64> {
+ artifact_bytes_by_extension_for(segments_dir, "")
+ }
+
+ /// Sum on-disk bytes per file extension, restricted to files whose name
+ /// starts with `id_prefix` (pass `""` for every segment in the dir).
+ /// Keyed by extension because merged segment IDs are fresh UUIDs, so the
+ /// two runs being compared never share a filename.
+ fn artifact_bytes_by_extension_for(
+ segments_dir: &std::path::Path,
+ id_prefix: &str,
+ ) -> std::collections::BTreeMap<String, u64> {
+ let mut out = std::collections::BTreeMap::new();
+ for entry in std::fs::read_dir(segments_dir).unwrap() {
+ let entry = entry.unwrap();
+ let name = entry.file_name().to_string_lossy().into_owned();
+ if !name.starts_with(id_prefix) {
+ continue;
+ }
+ let Some(ext) = name.rsplit('.').next() else {
+ continue;
+ };
+ // Only the compressed durable artifacts. `.fst` and `.norms` are
+ // not zstd envelopes, and the marker files are fixed-size.
+ if !matches!(ext, "seg" | "dv" | "post" | "meta") {
+ continue;
+ }
+ *out.entry(ext.to_string()).or_insert(0) += entry.metadata().unwrap().len();
+ }
+ out
+ }
+
+ /// #318 — `[compression]` was accepted and ignored. Two servers on the
+ /// same build with the same corpus, differing only in this section,
+ /// produced byte-identical indices: `.seg` 261,481 / `.dv` 10,679 /
+ /// `.post` 192,287 in both, no error, no warning, while the docs called
+ /// the settings live.
+ ///
+ /// The knob is honoured at MERGE, so this asserts both halves of that
+ /// contract: the merged artifacts must respond to `compression.level`,
+ /// and the flushed ones must not — raising the flush level is the ingest
+ /// collapse recorded in
+ /// `reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`, and a
+ /// well-meaning "make it consistent" change would reintroduce it.
+ #[tokio::test]
+ async fn compression_level_reaches_the_merge_encoder_but_never_the_flush_path() {
+ use xerj_common::config::CompressionLevel;
+
+ let fast_dir = TempDir::new().unwrap();
+ let best_dir = TempDir::new().unwrap();
+ let (fast_flushed, fast_merged) =
+ artifact_bytes_at_level(&fast_dir, CompressionLevel::Fast).await;
+ let (best_flushed, best_merged) =
+ artifact_bytes_at_level(&best_dir, CompressionLevel::Best).await;
+
+ assert_eq!(
+ fast_flushed, best_flushed,
+ "flush must ignore compression.level entirely — it is the \
+ back-pressure-critical path (zstd19 ingest regression)"
+ );
+
+ // Every compressed artifact family the merge re-encodes must respond.
+ // Before the fix each of these was equal, which is the bug.
+ for ext in ["seg", "dv", "post", "meta"] {
+ let fast = fast_merged.get(ext).copied().unwrap_or_default();
+ let best = best_merged.get(ext).copied().unwrap_or_default();
+ assert!(
+ fast > 0 && best > 0,
+ ".{ext} artifacts must exist to compare"
+ );
+ assert!(
+ best < fast,
+ ".{ext}: compression.level = \"best\" must produce a smaller \
+ merged artifact than \"fast\" (got best={best}, fast={fast}); \
+ equal bytes mean the setting reached no encoder"
+ );
+ }
+ }
+
#[tokio::test]
async fn merging_v1_raw_unsafe_fields_advances_marker_before_encoded_output() {
let dir = TempDir::new().unwrap();
@@ -5290,6 +5460,15 @@ pub struct Index {
/// `Config.merge` at index construction; reads are cheap and merge
/// runs hold the snapshot for the duration of one batch.
merge_config: xerj_common::config::MergeConfig,
+ /// Snapshot of `Config.compression`, read by `merge_pass_locked` to pick
+ /// the zstd effort for the merged segment's durable artifacts. Only the
+ /// merge path consults it: flush stays pinned at the level its own
+ /// constants document, because raising it there is the ingest collapse in
+ /// `reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`.
+ ///
+ /// Before #318 this section reached no encoder at all — two nodes
+ /// differing only in `[compression]` wrote byte-identical segments.
+ compression_config: xerj_common::config::CompressionConfig,
/// Retained so a semantic field added after index creation can pin the
/// same embedding identity before its first document is accepted.
embedding_config: xerj_common::config::EmbeddingConfig,
@@ -5851,6 +6030,7 @@ impl Index {
)),
merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),
merge_config: config.merge.clone(),
+ compression_config: config.compression.clone(),
embedding_config: config.embedding.clone(),
max_fields_per_index: config.limits.max_fields_per_index,
embedder: Arc::new(RwLock::new(effective_embedder)),
@@ -6204,6 +6384,7 @@ impl Index {
)),
merge_in_progress: Arc::new(std::sync::atomic::AtomicBool::new(false)),
merge_config: config.merge.clone(),
+ compression_config: config.compression.clone(),
embedding_config: config.embedding.clone(),
max_fields_per_index: config.limits.max_fields_per_index,
embedder: Arc::new(RwLock::new(effective_embedder)),
@@ -8479,6 +8660,12 @@ impl Index {
let excluded_fts_fields_for_task = excluded_fts_fields.clone();
let segments_dir_for_task = segments_dir.clone();
let dv_skip_for_merge = dv_skip.clone();
+ // #318 — the operator's `compression.level`, resolved once per
+ // merge task. Merge is the only writer that honours it: it runs
+ // on the nice-+15 SMALL pool, off the ingest critical path, which
+ // is precisely why raising the effort here is safe and raising it
+ // on flush was not.
+ let merge_zstd_level = self.compression_config.level.zstd_level();
let batch_for_task = batch;
let metas_for_task = metas;
let failed_for_task = Arc::clone(&failed_batches);
@@ -8813,7 +9000,10 @@ impl Index {
return None;
}
};
- let encoded = xerj_storage::stored_codec::encode_stored_v2(&merged_json_buf);
+ let encoded = xerj_storage::stored_codec::encode_stored_v2_at_level(
+ &merged_json_buf,
+ merge_zstd_level,
+ );
drop(merged_json_buf);
if let Err(e) = writer.add_section(SectionType::Stored, &encoded) {
tracing::error!("merge ABORTED: failed to add section: {e}");
@@ -8874,7 +9064,8 @@ impl Index {
&segments_dir_for_task,
merged_meta.id.as_str(),
Arc::clone(&registry_for_task),
- );
+ )
+ .with_zstd_level(merge_zstd_level);
for (field_name, cfg) in &field_configs_for_task {
fts_writer.configure_field(field_name.clone(), cfg.clone());
}
@@ -8952,10 +9143,11 @@ impl Index {
&dv_skip_for_merge,
);
if !columns.is_empty() {
- if let Err(e) = write_doc_values_sidecar(
+ if let Err(e) = write_doc_values_sidecar_at_level(
&segments_dir_for_task,
merged_meta.id.as_str(),
&columns,
+ merge_zstd_level,
) {
tracing::warn!("merge: doc-values write failed: {e}");
}
@@ -18778,9 +18970,26 @@ fn write_doc_values_sidecar(
segments_dir: &std::path::Path,
segment_id: &str,
columns: &std::collections::BTreeMap<String, xerj_storage::doc_values::Column>,
+) -> std::io::Result<()> {
+ write_doc_values_sidecar_at_level(
+ segments_dir,
+ segment_id,
+ columns,
+ xerj_storage::doc_values::DV_ZSTD_LEVEL,
+ )
+}
+
+/// [`write_doc_values_sidecar`] at a caller-chosen zstd level — the merge
+/// path's entry point for the operator's `compression.level` (#318). The
+/// flush path keeps the pinned level; see `DV_ZSTD_LEVEL`.
+fn write_doc_values_sidecar_at_level(
+ segments_dir: &std::path::Path,
+ segment_id: &str,
+ columns: &std::collections::BTreeMap<String, xerj_storage::doc_values::Column>,
+ level: i32,
) -> std::io::Result<()> {
let path = segments_dir.join(format!("{segment_id}.dv"));
- let bytes = xerj_storage::doc_values::encode_columns(columns);
+ let bytes = xerj_storage::doc_values::encode_columns_at_level(columns, level);
xerj_common::fsio::write_file_durable(&path, &bytes)
}
diff --git a/engine/crates/xerj-fts/src/index.rs b/engine/crates/xerj-fts/src/index.rs
index 696bdc374..02a0c083d 100644
--- a/engine/crates/xerj-fts/src/index.rs
+++ b/engine/crates/xerj-fts/src/index.rs
@@ -320,7 +320,12 @@ const POST_MAGIC_ZSTD: &[u8; 4] = b"ZPS1";
/// (merge-dominated long-term storage barely changes; only the
/// freshest tier-0 segments are larger before they merge). See
/// `engine/reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`.
-const ZSTD_DURABLE_LEVEL: i32 = 3;
+///
+/// This is the **flush** level and the writer default. A merge-time caller
+/// raises it via [`FtsIndexWriter::with_zstd_level`] to honour the operator's
+/// `compression.level` (#318) — merge is off the ingest critical path, which
+/// is exactly the distinction the paragraph above is about.
+pub const ZSTD_DURABLE_LEVEL: i32 = 3;
const ZFM3_RECORD_LEN: usize = 4 + 8 + 8 + 4; // 24 bytes: df, ttf, off, len
/// Header byte length for a ZFM3 file (magic + total_docs + total_field_length
@@ -357,6 +362,7 @@ fn encode_field_meta_v4(
has_positions: bool,
sorted_terms: &[String],
term_postings: &HashMap<String, TermPostings>,
+ zstd_level: i32,
) -> Result<Vec<u8>> {
let num_terms = sorted_terms.len();
// Build the records section in the same byte layout that ZFM3
@@ -380,7 +386,7 @@ fn encode_field_meta_v4(
}
let uncompressed_len = records.len() as u32;
let compressed =
- zstd::bulk::compress(&records, ZSTD_DURABLE_LEVEL).with_context(|| "ZFM4 zstd compress")?;
+ zstd::bulk::compress(&records, zstd_level).with_context(|| "ZFM4 zstd compress")?;
let mut out: Vec<u8> = Vec::with_capacity(ZFM3_HEADER_LEN + 4 + 4 + compressed.len());
out.extend_from_slice(META_MAGIC_V4);
out.write_u64::<LittleEndian>(stats.total_docs).unwrap();
@@ -555,6 +561,9 @@ pub struct FtsIndexWriter {
/// Per-field: (config, postings_writer, field_stats, norms)
fields: HashMap<String, FieldData>,
encoded_filename_layout_published: bool,
+ /// Zstd effort for the `.meta` (ZFM4) and `.post` (ZPS1) envelopes.
+ /// [`ZSTD_DURABLE_LEVEL`] unless [`Self::with_zstd_level`] raised it.
+ zstd_level: i32,
}
struct FieldData {
@@ -578,9 +587,23 @@ impl FtsIndexWriter {
registry,
fields: HashMap::new(),
encoded_filename_layout_published: false,
+ zstd_level: ZSTD_DURABLE_LEVEL,
}
}
+ /// Re-encode this segment's `.meta` / `.post` at `level` instead of the
+ /// flush default [`ZSTD_DURABLE_LEVEL`].
+ ///
+ /// For merge callers only — flush must not raise the level, and
+ /// [`ZSTD_DURABLE_LEVEL`] records the measured reason. Both envelopes
+ /// stay byte-format-identical across levels (zstd decode does not consult
+ /// the level a payload was written at), so this changes only how hard the
+ /// encoder works and how large the result is.
+ pub fn with_zstd_level(mut self, level: i32) -> Self {
+ self.zstd_level = level;
+ self
+ }
+
/// Register a field with its indexing configuration.
/// Must be called before `add_document` uses this field.
pub fn configure_field(&mut self, field: impl Into<String>, config: FieldIndexConfig) {
@@ -847,6 +870,7 @@ impl FtsIndexWriter {
let segment_dir = self.segment_dir.clone();
let segment_id = self.segment_id.clone();
+ let zstd_level = self.zstd_level;
// Drain fields into a Vec so we can parallelise the iterator.
// Cloning `stats` before consuming `field_data` — `stats` goes into
@@ -866,8 +890,14 @@ impl FtsIndexWriter {
let results: Vec<Result<(String, FieldStats)>> = fields
.into_par_iter()
.map(|(field_name, stats, field_data)| {
- Self::write_field_static(&segment_dir, &segment_id, &field_name, field_data)
- .with_context(|| format!("writing field '{}'", field_name))?;
+ Self::write_field_static(
+ &segment_dir,
+ &segment_id,
+ &field_name,
+ field_data,
+ zstd_level,
+ )
+ .with_context(|| format!("writing field '{}'", field_name))?;
Ok((field_name, stats))
})
.collect();
@@ -887,6 +917,7 @@ impl FtsIndexWriter {
segment_id: &str,
field_name: &str,
field_data: FieldData,
+ zstd_level: i32,
) -> Result<()> {
// NOTE: `PathBuf::with_extension` replaces the final `.ext` in the path,
// so using `segment_dir.join("segment_id.field_name")` followed by
@@ -959,7 +990,7 @@ impl FtsIndexWriter {
Vec::new()
} else {
let uncompressed_len = post_data.len() as u32;
- let compressed = zstd::bulk::compress(&post_data, ZSTD_DURABLE_LEVEL)
+ let compressed = zstd::bulk::compress(&post_data, zstd_level)
.with_context(|| "ZPS1 zstd compress")?;
let mut out = Vec::with_capacity(4 + 4 + compressed.len());
out.extend_from_slice(POST_MAGIC_ZSTD);
@@ -1030,6 +1061,7 @@ impl FtsIndexWriter {
has_positions,
&sorted_terms,
&term_postings,
+ zstd_level,
)?;
xerj_common::fsio::write_file_durable(&meta_path, &meta_bytes)
.with_context(|| format!("writing meta to {:?}", meta_path))?;
diff --git a/engine/crates/xerj-server/src/main.rs b/engine/crates/xerj-server/src/main.rs
index 0f13c4e02..af88bd1c5 100644
--- a/engine/crates/xerj-server/src/main.rs
+++ b/engine/crates/xerj-server/src/main.rs
@@ -441,6 +441,14 @@ fn load_config(args: &CliArgs) -> Result<Config> {
warn!("{key} is set but has no effect in this build: {effect}");
}
+ // Same treatment for `[compression]` (#318): `level` is wired into the
+ // merge re-encode, `enabled` and `block_size_docs` are not, and an
+ // operator who set all three to trade CPU for $/GB previously got three
+ // no-ops in silence.
+ for (key, effect) in cfg.compression.dormant_overrides() {
+ warn!("{key} is set but has no effect in this build: {effect}");
+ }
+
// Bind address override: `--bind` flag or `XERJ_BIND_ADDRESS` env (flag
// wins). Both exist because the default is loopback (#228): without a way
// to say "expose me" that is not a TOML file, a container image or a
diff --git a/engine/crates/xerj-storage/src/doc_values.rs b/engine/crates/xerj-storage/src/doc_values.rs
index 35cc6d43c..61e817824 100644
--- a/engine/crates/xerj-storage/src/doc_values.rs
+++ b/engine/crates/xerj-storage/src/doc_values.rs
@@ -626,7 +626,24 @@ fn roaring_retained_bytes(bitmap: &RoaringBitmap) -> u64 {
const KIND_FLAG_ZSTD: u8 = 0x80;
const KIND_MASK: u8 = 0x7F;
+/// Zstd level the flush path encodes `.dv` columns at. Pinned for the reason
+/// on `stored_codec::STORED_ZSTD_LEVEL`: flush is the back-pressure-critical
+/// bound on sustained ingest, and this used to be one of three stacked
+/// level-19 encoders in a single flush.
+pub const DV_ZSTD_LEVEL: i32 = 3;
+
+/// Encode at [`DV_ZSTD_LEVEL`]; merge calls [`encode_columns_at_level`] to
+/// honour `compression.level` (#318).
pub fn encode_columns(columns: &BTreeMap<String, Column>) -> Vec<u8> {
+ encode_columns_at_level(columns, DV_ZSTD_LEVEL)
+}
+
+/// [`encode_columns`] at a caller-chosen zstd effort level.
+///
+/// Level is a write-side choice only: the reader dispatches on the
+/// `KIND_FLAG_ZSTD` bit and zstd decode is level-independent, so `.dv` files
+/// written at different levels coexist in one index with no format change.
+pub fn encode_columns_at_level(columns: &BTreeMap<String, Column>, level: i32) -> Vec<u8> {
let mut out = Vec::new();
out.write_u32::<LittleEndian>(MAGIC).unwrap();
out.write_u32::<LittleEndian>(columns.len() as u32).unwrap();
@@ -639,16 +656,19 @@ pub fn encode_columns(columns: &BTreeMap<String, Column>) -> Vec<u8> {
Column::Numeric(n) => n.encode(),
Column::Keyword(k) => k.encode(),
};
- // Zstd level 3 — fast encode (~250 MB/s/core). Reverted from
- // level 19 because the flush path needs to keep up with sustained
- // ingest (1 M+ docs/s) and the level-19 pass collapsed throughput
- // by 70-100× under continuous load. See
+ // `level` is DV_ZSTD_LEVEL (3) on flush — fast encode
+ // (~250 MB/s/core), reverted from level 19 because the flush path
+ // needs to keep up with sustained ingest (1 M+ docs/s) and the
+ // level-19 pass collapsed throughput by 70-100× under continuous
+ // load. See
// `engine/reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`.
+ // Merge passes the operator's `compression.level` instead, which is
+ // off the ingest critical path.
// Legacy LZ4-framed column payloads are still auto-detected by
// the reader via the high bit on the kind byte; level-19-encoded
// segments from older builds remain readable (zstd decompress is
// level-independent).
- let zstd_payload = match zstd::encode_all(&payload[..], 3) {
+ let zstd_payload = match zstd::encode_all(&payload[..], level) {
Ok(c) if c.len() < payload.len() => c,
// If compression doesn't help (rare tiny columns), keep the
// zstd path anyway — decompression still works and avoids a
diff --git a/engine/crates/xerj-storage/src/stored_codec.rs b/engine/crates/xerj-storage/src/stored_codec.rs
index 0ddbe2487..920411060 100644
--- a/engine/crates/xerj-storage/src/stored_codec.rs
+++ b/engine/crates/xerj-storage/src/stored_codec.rs
@@ -165,7 +165,12 @@ const CROSS_DEP_MAX_SRC_CARDINALITY: usize = 256;
/// the on-flush overhead is only paid by the freshest tier-0 segments
/// before they merge anyway). See
/// `engine/reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md`.
-const STORED_ZSTD_LEVEL: i32 = 3;
+///
+/// This is now the level used **on flush**, which the operator cannot raise —
+/// that is the whole content of the paragraph above. Merge re-encodes at
+/// whatever `compression.level` resolves to (#318); see
+/// [`encode_stored_v2_at_level`].
+pub const STORED_ZSTD_LEVEL: i32 = 3;
#[repr(u8)]
#[derive(Copy, Clone, Debug, PartialEq)]
@@ -196,7 +201,22 @@ impl ColCodec {
/// On tiny segments (< `V2_MIN_DOCS` docs) or on any shape that doesn't
/// fit the column assumptions, the function silently falls back to the
/// v1 LZ4 encoder — the caller never has to care which was used.
+///
+/// Encodes at [`STORED_ZSTD_LEVEL`]; merge calls
+/// [`encode_stored_v2_at_level`] instead to honour `compression.level`.
pub fn encode_stored_v2(stored_docs_json: &[u8]) -> Vec<u8> {
+ encode_stored_v2_at_level(stored_docs_json, STORED_ZSTD_LEVEL)
+}
+
+/// [`encode_stored_v2`] at a caller-chosen zstd effort level.
+///
+/// Only the merge path passes anything but [`STORED_ZSTD_LEVEL`] — flush is
+/// back-pressure-critical and stays pinned (see that constant). The output is
+/// the same ZBS2 format at any level: zstd decode does not need to know the
+/// level a payload was written at, so segments written before and after an
+/// operator changes `compression.level` interleave with no migration and no
+/// format flag.
+pub fn encode_stored_v2_at_level(stored_docs_json: &[u8], level: i32) -> Vec<u8> {
// Parse the JSON array of documents. Failure → v1 fallback.
let docs: Vec<serde_json::Value> = match serde_json::from_slice(stored_docs_json) {
Ok(v) => v,
@@ -250,7 +270,13 @@ pub fn encode_stored_v2(stored_docs_json: &[u8]) -> Vec<u8> {
}
}
- encode_v2_columns(num_docs, &col_order, &columns, Some(stored_docs_json))
+ encode_v2_columns(
+ num_docs,
+ &col_order,
+ &columns,
+ Some(stored_docs_json),
+ level,
+ )
}
/// P2.2 — columnar V2 encoder fed directly from already-parsed values.
@@ -394,7 +420,16 @@ fn encode_stored_v2_from_values_inner(
);
}
- encode_v2_columns(num_docs, &col_order, &columns, stored_docs_json)
+ // Flush-only entry point — pinned to the flush level, never the
+ // operator's (#318): raising the level here is the ingest regression
+ // documented on `STORED_ZSTD_LEVEL`.
+ encode_v2_columns(
+ num_docs,
+ &col_order,
+ &columns,
+ stored_docs_json,
+ STORED_ZSTD_LEVEL,
+ )
}
/// Shared tail of the two V2 entry points: per-column codec selection +
@@ -409,6 +444,7 @@ fn encode_v2_columns(
col_order: &[String],
columns: &[Vec<&serde_json::Value>],
stored_docs_json: Option<&[u8]>,
+ level: i32,
) -> Vec<u8> {
// THROWAWAY prof (XERJ_PROF): encode-phase breakdown.
let prof = std::env::var_os("XERJ_PROF").is_some();
@@ -497,7 +533,7 @@ fn encode_v2_columns(
continue;
}
if let Some(src_ix) = cross_dep_src[cix] {
- let (payload, ok) = encode_cross_dep(col, src_ix, &dict_encoded);
+ let (payload, ok) = encode_cross_dep(col, src_ix, &dict_encoded, level);
if ok {
col_payloads.push((cname.clone(), ColCodec::CrossDep as u8, payload));
continue;
@@ -505,15 +541,15 @@ fn encode_v2_columns(
}
if let Some((entries, ids)) = &dict_encoded[cix] {
if entries.len() <= DICT_MAX_CARDINALITY && all_scalar_dict_entries(entries) {
- let payload = encode_dict_bitpack(entries, ids);
+ let payload = encode_dict_bitpack(entries, ids, level);
col_payloads.push((cname.clone(), ColCodec::DictBitpack as u8, payload));
continue;
}
}
// Fallback: zstd over JSON-array of the column's values.
let col_json = serde_json::to_vec(col).unwrap_or_default();
- let zstd_payload = zstd::encode_all(Cursor::new(&col_json), STORED_ZSTD_LEVEL)
- .unwrap_or_else(|_| col_json.clone());
+ let zstd_payload =
+ zstd::encode_all(Cursor::new(&col_json), level).unwrap_or_else(|_| col_json.clone());
// Choose RAW_JSON vs LZ4_JSON by size.
let lz4_payload = lz4_flex::compress_prepend_size(&col_json);
if lz4_payload.len() + 1 < zstd_payload.len() {
@@ -2041,7 +2077,7 @@ fn decode_constant(payload: &[u8], num_docs: usize) -> Result<Vec<serde_json::Va
Ok(values)
}
-fn encode_dict_bitpack(entries: &[serde_json::Value], ids: &[u32]) -> Vec<u8> {
+fn encode_dict_bitpack(entries: &[serde_json::Value], ids: &[u32], level: i32) -> Vec<u8> {
let dict_count = entries.len();
// Reserve one extra id for null → dict_count (which might be past the
// packed range, so we allocate `bit_width` large enough for dict_count
@@ -2059,8 +2095,7 @@ fn encode_dict_bitpack(entries: &[serde_json::Value], ids: &[u32]) -> Vec<u8> {
// Dict entries as zstd(json array).
let dict_json = serde_json::to_vec(entries).unwrap_or_default();
- let dict_zstd =
- zstd::encode_all(Cursor::new(&dict_json), STORED_ZSTD_LEVEL).unwrap_or(dict_json.clone());
+ let dict_zstd = zstd::encode_all(Cursor::new(&dict_json), level).unwrap_or(dict_json.clone());
out.write_u32::<LittleEndian>(dict_zstd.len() as u32)
.unwrap();
out.extend_from_slice(&dict_zstd);
@@ -2069,8 +2104,7 @@ fn encode_dict_bitpack(entries: &[serde_json::Value], ids: &[u32]) -> Vec<u8> {
let packed = bitpack_u32(ids, bit_width);
// zstd over the bit-packed stream — gives another 20-40 % on log
// data because repeated ids cluster.
- let packed_zstd =
- zstd::encode_all(Cursor::new(&packed), STORED_ZSTD_LEVEL).unwrap_or(packed.clone());
+ let packed_zstd = zstd::encode_all(Cursor::new(&packed), level).unwrap_or(packed.clone());
out.write_u32::<LittleEndian>(ids.len() as u32).unwrap();
out.write_u32::<LittleEndian>(packed_zstd.len() as u32)
.unwrap();
@@ -2166,6 +2200,7 @@ fn encode_cross_dep<B: std::borrow::Borrow<serde_json::Value>>(
target_col: &[B],
src_ix: usize,
dict_encoded: &[Option<(Vec<serde_json::Value>, Vec<u32>)>],
+ level: i32,
) -> (Vec<u8>, bool) {
let (_src_entries, src_ids) = match &dict_encoded[src_ix] {
Some(e) => e,
@@ -2234,7 +2269,7 @@ fn encode_cross_dep<B: std::borrow::Borrow<serde_json::Value>>(
prev_ord = *ord;
}
// zstd the whole thing.
- let compressed = zstd::encode_all(Cursor::new(&out), STORED_ZSTD_LEVEL).unwrap_or(out.clone());
+ let compressed = zstd::encode_all(Cursor::new(&out), level).unwrap_or(out.clone());
// Save the smaller of raw vs zstd.
let mut final_out = Vec::with_capacity(1 + compressed.len().max(out.len()));
if compressed.len() + 1 < out.len() {
diff --git a/engine/crates/xerj-storage/src/stored_codec/row_hydration.rs b/engine/crates/xerj-storage/src/stored_codec/row_hydration.rs
index b1d0295eb..2a7df97d2 100644
--- a/engine/crates/xerj-storage/src/stored_codec/row_hydration.rs
+++ b/engine/crates/xerj-storage/src/stored_codec/row_hydration.rs
@@ -871,7 +871,7 @@ mod tests {
(
"__id",
ColCodec::DictBitpack,
- encode_dict_bitpack(&values, &ids),
+ encode_dict_bitpack(&values, &ids, STORED_ZSTD_LEVEL),
),
(
"__seq_no",
@@ -902,7 +902,7 @@ mod tests {
assert_eq!(unpack_id(&packed[..packed.len() - 1], width, 2), None);
}
- let mut corrupt = encode_dict_bitpack(&[json!("only")], &[0, 0, 0]);
+ let mut corrupt = encode_dict_bitpack(&[json!("only")], &[0, 0, 0], STORED_ZSTD_LEVEL);
// Rebuild the packed stream with id=3, which is outside {entry,null}.
let dict_len = u32::from_le_bytes(corrupt[5..9].try_into().unwrap()) as usize;
let packed_offset = 9 + dict_len + 8;
@@ -926,7 +926,7 @@ mod tests {
);
assert!(decode_stored_v2_rows(&bad, &[0]).is_err());
- let mut padding = encode_dict_bitpack(&[json!("only")], &[0, 0, 0]);
+ let mut padding = encode_dict_bitpack(&[json!("only")], &[0, 0, 0], STORED_ZSTD_LEVEL);
let dict_len = u32::from_le_bytes(padding[5..9].try_into().unwrap()) as usize;
let packed_length_offset = 9 + dict_len + 4;
let packed_offset = packed_length_offset + 4;
@@ -1226,7 +1226,7 @@ mod tests {
(
"__id",
ColCodec::DictBitpack,
- encode_dict_bitpack(&[json!("a"), json!("b")], &[0, 1, 0, 1]),
+ encode_dict_bitpack(&[json!("a"), json!("b")], &[0, 1, 0, 1], STORED_ZSTD_LEVEL),
),
("__seq_no", ColCodec::CrossDep, dependent),
],
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index f4f667f7a..6de9bf0b8 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -230,23 +230,36 @@ max_merge_count = 16
[compression]
-# Enable block compression for stored document fields.
-# Disabling compression increases disk usage but reduces CPU overhead on read.
+# ACCEPTED BUT NOT WIRED. Every durable artifact xerj writes is a compressed
+# envelope — stored (ZBS2), .meta (ZFM4), .post (ZPS1), .dv — and none of them
+# has an uncompressed write path; the legacy uncompressed layouts survive only
+# so old segments stay readable. Setting this to false logs a warning at
+# startup and changes nothing on disk.
enabled = true
-# Compression algorithm and level.
+# Zstandard effort level, applied when segments are re-encoded AT MERGE.
#
-# "fast" — LZ4. Maximum throughput (~500 MB/s), moderate compression ratio.
-# Best for hot indexes accessed frequently.
-# "balanced" — Zstandard level 3. Good ratio (~3–4×) with low CPU overhead.
-# Recommended for most workloads.
-# "best" — Zstandard level 19. Maximum ratio (~5–6×) at higher CPU cost.
-# Best for cold storage or archival indexes.
+# "fast" — Zstd level 1. Fastest encode, loosest ratio.
+# "balanced" — Zstd level 3. Recommended for most workloads.
+# "best" — Zstd level 6. Tightest measured ratio per byte of decode
+# window. Best for cold storage or archival indexes.
+#
+# Flush ignores this and always writes level 3. Flush is the back-pressure-
+# critical bound on sustained ingest: raising its level to 19 once collapsed
+# throughput from 1.55M docs/s to 21k docs/s with 75% of documents rejected
+# (reports/2026-04-25T21-50-00_ingest_perf_regression_zstd19.md). Merge runs
+# off that path and produces the long-lived segments where footprint matters.
+#
+# A change therefore takes effect as segments merge, not at restart. To apply
+# it to data already on disk, re-merge: POST /{index}/_forcemerge. Segments
+# written at different levels coexist — zstd decode does not consult the level
+# a payload was written at, so no migration or reindex is needed.
level = "balanced"
-# Number of documents grouped into a single compressed block.
-# Larger blocks compress better (more context) but increase read amplification
-# when fetching individual documents. Range: 16 – 4096.
+# ACCEPTED BUT NOT WIRED. The stored codec is columnar over the whole segment
+# section rather than blocked by document count, so there is no doc-block for
+# this to size. Still range-checked (16 – 4096) and warned about when moved off
+# the default, so a typo fails at startup instead of being ignored.
block_size_docs = 128
# ── Full-Text Search (1 setting) ──────────────────────────────────────────────
diff --git a/landing/docs/compression.html b/landing/docs/compression.html
index 34157e50f..a0f8dcbf0 100644
--- a/landing/docs/compression.html
+++ b/landing/docs/compression.html
@@ -122,13 +122,15 @@
<p>XERJ uses two layers of compression. <strong>Column encodings</strong> are chosen per-field at write time based on the column's shape. <strong>Block compression</strong> runs over the encoded column data. The two together give XERJ a smaller on-disk footprint than Elasticsearch — measured 1.61× smaller vs ES 8.13.4 on the reproducible telemetry-corpus scorecard (the exact ratio varies by data shape).</p>
<h2>Block compression</h2>
- <p>Three levels, controlled by <code class="inline">[compression] level</code>:</p>
+ <p>Three Zstandard effort levels, controlled by <code class="inline">[compression] level</code>:</p>
<ul>
- <li><strong>fast</strong> — LZ4. ~500 MB/s throughput. Use for write-heavy paths.</li>
- <li><strong>balanced</strong> — Zstd L3. Default. ~3-4× ratio.</li>
- <li><strong>best</strong> — Zstd L19. ~5-6× ratio. Use for cold indices you query rarely.</li>
+ <li><strong>fast</strong> — Zstd L1. Fastest encode, loosest ratio.</li>
+ <li><strong>balanced</strong> — Zstd L3. Default.</li>
+ <li><strong>best</strong> — Zstd L6. Tightest measured ratio per byte of decode window. Use for cold indices you query rarely.</li>
</ul>
- <p><code class="inline">block_size_docs</code> (16-4096, default 128) controls the compression block size.</p>
+ <p><strong>The level applies at merge, not at flush.</strong> Freshly flushed segments are always written at L3: flush is the back-pressure-critical bound on ingest, and raising its level once collapsed sustained throughput from 1.55M docs/s to 21k docs/s with 75% of documents rejected. Merges run off that path, and they are what produces the long-lived segments where footprint actually matters. So a level change takes effect as segments merge, not the instant you restart — and re-merging an existing index (<code class="inline">POST /{index}/_forcemerge</code>) applies it to data already on disk.</p>
+ <p>Levels are a write-side choice only. Zstd decoding does not depend on the level a payload was written at, so segments written at different levels coexist in one index with no migration and no reindex.</p>
+ <p><code class="inline">enabled</code> and <code class="inline">block_size_docs</code> are <strong>accepted but not wired</strong>. Every durable artifact XERJ writes is a compressed envelope with no uncompressed write path, so <code class="inline">enabled = false</code> does nothing; and the stored codec is columnar over the whole segment section rather than blocked by document count, so there is no doc-block for <code class="inline">block_size_docs</code> (16-4096, default 128) to size. Setting either off its default logs a warning at startup. Both were silently ignored before <a href="https://github.com/xerj-org/xerj/issues/318">#318</a>.</p>
<h2>Column encodings</h2>
<p>Picked automatically at ingest time from this set:</p>
diff --git a/landing/docs/config.html b/landing/docs/config.html
index 87495f675..ef4a7e18a 100644
--- a/landing/docs/config.html
+++ b/landing/docs/config.html
@@ -124,7 +124,7 @@
<p>Paths for the config are, in precedence order: <code class="inline">--config /path/to/xerj.toml</code> on the command line, then <code class="inline">/etc/xerj/xerj.toml</code>, then <code class="inline">./xerj.toml</code> in the working directory. If none exist, the full default is used.</p>
<h2>Full key table</h2>
- <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Block-level compression. Disabling increases disk usage and lowers read CPU.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (LZ4), "balanced" (Zstd L3, default), "best" (Zstd L19, cold storage).</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Docs per compressed block. Range 16–4096. Larger = better ratio, higher fetch cost.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div>
+ <div class="cfg"><div class="head">KEY</div><div class="head">TYPE</div><div class="head">DEFAULT</div><div class="head">DESCRIPTION</div><div class="group" id="server">[server]</div><div class="cell k">rest_port</div><div class="cell t">u16</div><div class="cell d">8080</div><div class="cell dsc">Native REST API listener port. All native endpoints (/v1/*) live here.</div><div class="cell k">grpc_port</div><div class="cell t">u16</div><div class="cell d">8081</div><div class="cell dsc">Reserved for a future gRPC API. Not wired in v0.1 — leave at default.</div><div class="cell k">es_compat_port</div><div class="cell t">u16</div><div class="cell d">9200</div><div class="cell dsc">Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. Set to 0 to disable.</div><div class="cell k">bind_address</div><div class="cell t">string</div><div class="cell d">"127.0.0.1"</div><div class="cell dsc">Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.</div><div class="cell k">allow_insecure_network_bind</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.</div><div class="cell k">data_dir</div><div class="cell t">path</div><div class="cell d">"./data"</div><div class="cell dsc">Root for indices, WAL files, and segments. Needs fast I/O and enough free space.</div><div class="group" id="auth">[auth]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Require an API key on every request. Set to false only on a trusted network.</div><div class="cell k">admin_api_key</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.</div><div class="group" id="tls">[tls]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.</div><div class="cell k">cert_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM X.509 certificate. Use a CA-signed cert in production.</div><div class="cell k">key_path</div><div class="cell t">path</div><div class="cell d">""</div><div class="cell dsc">PEM private key. Permissions should be 0600.</div><div class="group" id="storage">[storage]</div><div class="cell k">wal_sync</div><div class="cell t">enum</div><div class="cell d">"batched"</div><div class="cell dsc">"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.</div><div class="cell k">wal_batch_ms</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.</div><div class="cell k">wal_max_size_mb</div><div class="cell t">u32</div><div class="cell d">1024</div><div class="cell dsc">WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.</div><div class="cell k">flush_size_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Memtable buffer size that triggers a segment flush (MiB).</div><div class="cell k">flush_interval_secs</div><div class="cell t">u32</div><div class="cell d">30</div><div class="cell dsc">Maximum wall-clock interval between flushes even if the buffer is not full.</div><div class="group" id="merge">[merge]</div><div class="cell k">strategy</div><div class="cell t">enum</div><div class="cell d">"size_tiered"</div><div class="cell dsc">Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.</div><div class="cell k">min_segments</div><div class="cell t">u32</div><div class="cell d">10</div><div class="cell dsc">Accepted and validated (must be ≥ 2) but <strong>not wired</strong>: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_segment_mb</div><div class="cell t">u32</div><div class="cell d">8192</div><div class="cell dsc">Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.</div><div class="cell k">io_rate_mb_per_sec</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.</div><div class="cell k">max_concurrent</div><div class="cell t">u8</div><div class="cell d">1</div><div class="cell dsc">Accepted but <strong>not wired</strong>: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.</div><div class="group" id="compression">[compression]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Accepted but <strong>not wired</strong>: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.</div><div class="cell k">level</div><div class="cell t">enum</div><div class="cell d">"balanced"</div><div class="cell dsc">"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded <strong>at merge</strong>; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.</div><div class="cell k">block_size_docs</div><div class="cell t">u32</div><div class="cell d">128</div><div class="cell dsc">Accepted but <strong>not wired</strong>: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.</div><div class="group" id="fts">[fts]</div><div class="cell k">default_analyzer</div><div class="cell t">enum</div><div class="cell d">"standard"</div><div class="cell dsc">"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping.</div><div class="group" id="vector">[vector]</div><div class="cell k">default_metric</div><div class="cell t">enum</div><div class="cell d">"cosine"</div><div class="cell dsc">"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice.</div><div class="cell k">hnsw_m</div><div class="cell t">u32</div><div class="cell d">16</div><div class="cell dsc">Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_construction</div><div class="cell t">u32</div><div class="cell d">200</div><div class="cell dsc">Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.</div><div class="cell k">hnsw_ef_search</div><div class="cell t">u32</div><div class="cell d">100</div><div class="cell dsc">Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).</div><div class="cell k">default_quantization</div><div class="cell t">enum</div><div class="cell d">"none"</div><div class="cell dsc">"none" (the default) or "scalar8" = 4× RAM saving, 1–2% recall loss. "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping).</div><div class="cell k">max_dimensions</div><div class="cell t">u32</div><div class="cell d">16384</div><div class="cell dsc">Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096.</div><div class="group" id="logs">[logs]</div><div class="cell k">retention_days</div><div class="cell t">u32</div><div class="cell d">90</div><div class="cell dsc">Auto-delete log docs older than N days. 0 disables.</div><div class="cell k">time_partition</div><div class="cell t">enum</div><div class="cell d">"1h"</div><div class="cell dsc">"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning.</div><div class="group" id="embedding">[embedding]</div><div class="cell k">default_endpoint</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".</div><div class="cell k">default_model</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".</div><div class="cell k">batch_size</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.</div><div class="cell k">timeout_ms</div><div class="cell t">u32</div><div class="cell d">5000</div><div class="cell dsc">HTTP timeout for embedding calls (ms). Ingest fails with a timeout error if exceeded.</div><div class="group" id="limits">[limits]</div><div class="cell k">max_query_memory_mb</div><div class="cell t">u32</div><div class="cell d">512</div><div class="cell dsc">Per-query memory cap (MiB). Queries that exceed it are cancelled.</div><div class="cell k">max_concurrent_searches</div><div class="cell t">u32</div><div class="cell d">64</div><div class="cell dsc">Global in-flight search ceiling. Extras are queued.</div><div class="cell k">max_fields_per_index</div><div class="cell t">u32</div><div class="cell d">500</div><div class="cell dsc">Field-explosion protection. ES default is 1000; 500 is intentionally stricter.</div><div class="group" id="indexing">[indexing]</div><div class="cell k">turbo_batch_size</div><div class="cell t">u32</div><div class="cell d">1000</div><div class="cell dsc">Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency.</div><div class="cell k">turbo_parallel</div><div class="cell t">bool</div><div class="cell d">true</div><div class="cell dsc">Parallel tokenisation on Rayon threads. Disable only for debugging.</div><div class="cell k">turbo_fast_analyzer</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Skip stemming and stop-word removal in turbo mode. Trades recall for speed.</div><div class="group" id="cluster">[cluster]</div><div class="cell k">enabled</div><div class="cell t">bool</div><div class="cell d">false</div><div class="cell dsc">Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.</div><div class="cell k">port</div><div class="cell t">u16</div><div class="cell d">9300</div><div class="cell dsc">TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.</div><div class="cell k">peers</div><div class="cell t">array</div><div class="cell d">[]</div><div class="cell dsc">Peer list in "node_id=host:port" format. The local node identifies itself from the entry matching bind_address:port. Example: ["a=10.0.0.1:9300","b=10.0.0.2:9300","c=10.0.0.3:9300"].</div><div class="cell k">tick_ms</div><div class="cell t">u64</div><div class="cell d">50</div><div class="cell dsc">Raft tick interval (ms). Lower = faster leader election at the cost of CPU.</div><div class="cell k">auth_secret</div><div class="cell t">string</div><div class="cell d">""</div><div class="cell dsc">Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.</div></div>
<h2 id="server-section">[server]</h2>
@@ -176,11 +176,11 @@ min_merge_count = 4 # the real per-tier merge trigger
max_merge_count = 16 # caps peak merge RAM</pre>
<h2 id="compression-section">[compression]</h2>
- <p>See <a href="/docs/compression.html">Compression</a> for the encoding catalog. This section picks the outer <em>block</em> codec only — the inner per-column encodings are chosen automatically at write time.</p>
+ <p>See <a href="/docs/compression.html">Compression</a> for the encoding catalog. This section picks the outer <em>block</em> codec effort only — the inner per-column encodings are chosen automatically at write time.</p>
<pre class="code">[compression]
-enabled = true
-level = "balanced" # LZ4 / Zstd L3 / Zstd L19
-block_size_docs = 128</pre>
+enabled = true # accepted, not wired — no uncompressed write path exists
+level = "balanced" # Zstd L1 / L3 / L6, applied at MERGE (flush is always L3)
+block_size_docs = 128 # accepted, not wired — 16-4096, range-checked at startup</pre>
<h2 id="fts-section">[fts]</h2>
<p>Default analyzer applied to untyped <code class="inline">text</code> fields. Override per-field in the mapping when creating an index. See <a href="/docs/analyzers.html">Analyzers</a> for the built-ins.</p>
diff --git a/landing/docs/index.html b/landing/docs/index.html
index 1352d1295..ecbb4e4fc 100644
--- a/landing/docs/index.html
+++ b/landing/docs/index.html
@@ -252,7 +252,7 @@
<span class="label">ENGINE</span>
<a href="/docs/ingest.html">Ingest pipelines<span class="d">Turbo, logs, syslog, OTLP — in-process</span></a>
<a href="/docs/storage.html">Storage & WAL<span class="d">WAL + immutable segments · three files per segment</span></a>
- <a href="/docs/compression.html">Compression<span class="d">LZ4 · Zstd L3 · Zstd L19 · 9 encodings</span></a>
+ <a href="/docs/compression.html">Compression<span class="d">Zstd L1 · L3 · L6 · 9 encodings</span></a>
<a href="/docs/clustering.html">Clustering<span class="d">Embedded Raft · metadata replication · region split/merge</span></a>
<a href="/docs/metrics.html">Metrics<span class="d">Prometheus text · counters · histograms · gauges</span></a>
<a href="/docs/security.html">Auth & TLS<span class="d">API-key auth · PEM certs</span></a>