AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): size the default memory cap and merge batches for small hosts (#1122) (#2340)
xerj-org2819cc9541d4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4e33643c0..48426721c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -130,6 +130,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0recorded only on a catalog dataset document([#2321](https://github.com/xerj-org/xerj/issues/2321)).+- The default process memory cap on a host under 16 GiB is half of that+ RAM, and segment merges follow it (#1122). An 8 GiB machine used to+ take the 8 GiB step, so the budget was the whole machine; a merge+ storm on Windows 10 stayed up only after+ `XERJ_MAX_PROCESS_MEMORY_MB=4096`. That host now gets a 4 GiB cap.+ A merge batch on a budget that cannot hold the historical 512 MiB+ input (8× working set plus the 2 GiB memtable floor) is sized to+ `budget / 64` (64 MiB at 4 GiB) and concurrent merges share a quarter+ of the budget. A batch that would still cross the RSS watermark waits+ for the next check; one batch still runs, because a merge is what+ retires per-segment caches. Hosts at 16 GiB and above keep the 8 / 16 /+ 32 GiB steps, the 512 MiB batch, and the configured merge parallelism.+ A single JSON document over 50 MiB is still refused at the 8 MiB+ ceiling, before it is parsed, and the rest of the bulk is indexed.+- A multi-term question no longer ranks a one-word symbol definition(`type Token`) above a hit that covers more of the question. `xerj search`,`xerj def`, `xerj code`, and the plain-string `xerj_search` tool add adiff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 4cfcb02aa..7c54f70a5 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1740,9 +1740,10 @@ impl Default for EmbeddingConfig {// ─────────────────────────────────────────────────────────────────────────────/// Sentinel value of [`LimitsConfig::max_process_memory_mb`] meaning **AUTO**:-/// pick the process-memory cap from the machine size in binary-GiB steps-/// (8 / 16 / 32 GiB) rather than a flat number. It is the serde default, so an-/// omitted field resolves to AUTO; the governor intercepts it before any budget+/// pick the process-memory cap from the machine size (half of RAM under+/// 16 GiB, then 8 / 16 / 32 GiB) rather than a flat number. It is the serde+/// default, so an omitted field resolves to AUTO; the governor intercepts it+/// before any budget/// math, so this literal never reaches a budget as a byte count. `u64::MAX` is/// used because no operator sets a 16-EiB cap — it is a reserved, unmistakable/// "not a real MiB value" marker, distinct from `0` (uncapped, the whole@@ -1866,14 +1867,19 @@ pub struct LimitsConfig {/// Ceiling on the machine size every other memory budget is derived FROM,/// in MiB. **Omitted / default = AUTO** ([`AUTO_PROCESS_MEMORY_MB`]): XERJ/// picks the cap from the effective (cgroup-aware) memory it can actually- /// see, in binary-GiB steps —+ /// see —///- /// effective < 64 GiB → 8 GiB cap+ /// effective < 16 GiB → half of that RAM+ /// 16 GiB ≤ effective < 64 GiB → 8 GiB cap/// 64 GiB ≤ effective < 128 GiB → 16 GiB cap/// effective ≥ 128 GiB → 32 GiB cap////// so a laptop is never sized for ~20 GiB of memtables and a big server is- /// no longer starved by a flat 8 GiB. Every budget in `ResourceGovernor`+ /// no longer starved by a flat 8 GiB. A host under 16 GiB used to take the+ /// 8 GiB step and then the machine, so an 8 GiB box's budget was all of+ /// its RAM; half of RAM leaves room for the OS during a merge (#1122).+ /// At 16 GiB the half-RAM step and the 8 GiB step are the same number.+ /// Every budget in `ResourceGovernor`/// derives from `effective_memory_limit_bytes()` = min(cgroup limit, total/// system RAM); capping that ONE base is what makes every dependent budget/// shrink coherently instead of each growing its own ceiling.@@ -1918,7 +1924,7 @@ impl Default for LimitsConfig {max_segment_hydration_cache_mb: 0, // 0 = 20% effective memory, no floormemory_watermark_percent: 95,disk_flood_stage_percent: 95,- // AUTO: tier the cap (8/16/32 GiB) off the machine at governor+ // AUTO: half of RAM under 16 GiB, then 8/16/32 GiB, at governor// build time. 0 = uncapped (whole machine); N = a fixed MiB cap.max_process_memory_mb: AUTO_PROCESS_MEMORY_MB,}diff --git a/engine/crates/xerj-common/src/document_limit.rs b/engine/crates/xerj-common/src/document_limit.rsindex 1ff92c757..a536b19c6 100644--- a/engine/crates/xerj-common/src/document_limit.rs+++ b/engine/crates/xerj-common/src/document_limit.rs@@ -158,6 +158,22 @@ mod tests {);}+ #[test]+ fn a_fifty_mebibyte_document_is_refused_before_it_is_indexed() {+ // Field report on #1122: a single JSON file over 50 MB killed the+ // node. The ceiling is 8 MiB, so that file is the same refusal as+ // one byte over the ceiling — checked on the length, before parse.+ let len = 50 * 1024 * 1024;+ assert!(indexed_document_too_large(len));+ let text = indexed_document_refusal(len);+ assert!(text.contains("52428800"), "{text}");+ assert!(+ text.contains(&MAX_INDEXED_DOCUMENT_BYTES.to_string()),+ "{text}"+ );+ assert!(text.contains("not indexed"), "{text}");+ }+#[test]fn refusal_names_the_size_and_the_limit() {let text = indexed_document_refusal(50 * 1024 * 1024);diff --git a/engine/crates/xerj-engine/src/governor.rs b/engine/crates/xerj-engine/src/governor.rsindex 96430277f..a593642f0 100644--- a/engine/crates/xerj-engine/src/governor.rs+++ b/engine/crates/xerj-engine/src/governor.rs@@ -413,6 +413,25 @@ impl ResourceGovernor {))}+ /// Resolved process-memory ceiling in bytes. `0` is not used: an+ /// uncapped config stores the machine size here. Merge planning reads+ /// this so a small host's batches follow the AUTO cap instead of a+ /// fixed 512 MiB.+ pub fn process_memory_limit_bytes(&self) -> u64 {+ self.memory_limit_bytes+ }++ /// RSS admission watermark in bytes. `0` means the check is off.+ pub fn memory_watermark_bytes(&self) -> u64 {+ self.memory_watermark_bytes.load(Ordering::Relaxed)+ }++ /// Larger of the last RSS sample and a live read. `0` when this+ /// platform cannot read process memory.+ pub fn observed_rss_bytes(&self) -> u64 {+ self.observed_process_memory()+ }+/// Whether the parent RSS memory breaker is currently engaged (issue/// #948). Admission (429) is only half of the contract: the resource/// sampler uses this flag, together with [`Self::memtable_breaker_engaged`],@@ -1033,9 +1052,10 @@ fn windows_override_ceiling(}/// The AUTO process-memory cap, in MiB, chosen from the effective memory the-/// process can actually see, in binary-GiB steps:+/// process can actually see:///-/// effective < 64 GiB → 8 GiB+/// effective < 16 GiB → half of that RAM+/// 16 GiB ≤ effective < 64 GiB → 8 GiB/// 64 GiB ≤ effective < 128 GiB → 16 GiB/// effective ≥ 128 GiB → 32 GiB///@@ -1043,11 +1063,21 @@ fn windows_override_ceiling(/// `effective_memory_limit_bytes` — not raw host RAM, so a container smaller/// than its host is sized for the container. This only *chooses* a ceiling;/// `cap_memory_limit` still mins it with the machine, so the cap only ever-/// LOWERS a budget: on a 40 GiB box the 8 GiB tier yields min(40, 8) = 8, and-/// on a 6 GiB box it yields min(6, 8) = 6. It never invents headroom.+/// LOWERS a budget: on a 40 GiB box the 8 GiB tier yields min(40, 8) = 8.+///+/// Hosts under 16 GiB used to take the 8 GiB step and then `min` with the+/// machine, so an 8 GiB box's process budget was all of its RAM. A merge+/// storm on that budget died; the same box stayed up at a 4 GiB cap+/// (#1122). Half of RAM is that cap. At 16 GiB, half is 8 GiB, so the+/// steps above 16 GiB — and the memtable budget they derive — do not move.+/// A sub-mebibyte reading rounds up to 1 MiB and is still min'd with the+/// machine, so the floor does not invent RAM.fn tiered_cap_mb(effective_bytes: u64) -> u64 {const GIB: u64 = 1024 * 1024 * 1024;- if effective_bytes < 64 * GIB {+ const MIB: u64 = 1024 * 1024;+ if effective_bytes < 16 * GIB {+ (effective_bytes / 2 / MIB).max(1)+ } else if effective_bytes < 64 * GIB {8 * 1024} else if effective_bytes < 128 * GIB {16 * 1024@@ -1295,7 +1325,8 @@ fn build(config: &Config) -> ResourceGovernor {//// Capped by `limits.max_process_memory_mb`, resolved through// `resolve_process_memory_cap`: the AUTO default tiers the cap off the- // machine (8/16/32 GiB), `0` means uncapped (whole machine), an explicit+ // machine (half of RAM under 16 GiB, then 8/16/32 GiB), `0` means uncapped+ // (whole machine), an explicit// `N` forces a fixed MiB ceiling, and `XERJ_MAX_PROCESS_MEMORY_MB` can// override any of it. Every budget below derives from this one number, so// capping HERE is what makes the ceiling coherent — capping each dependent@@ -2138,14 +2169,33 @@ mod tests {assert_eq!(cap_memory_limit(40 * GIB, laptop.cap_mb), 8 * GIB);assert_eq!(auto_memtable_budget(8 * GIB), 2 * GIB);- // 6 GiB box → 8 GiB tier, but the machine only has 6, so min = 6.+ // 8 GiB host → half of RAM, 4 GiB. The old tier was min(8, 8) = 8,+ // the whole machine; a merge storm on that budget is #1122.+ let small = resolve_process_memory_cap(8 * GIB, AUTO, None);+ assert_eq!(small.cap_mb, 4 * 1024);+ assert_eq!(cap_memory_limit(8 * GIB, small.cap_mb), 4 * GIB);+ // The memtable floor stays 2 GiB, the same budget the old 8 GiB+ // tier handed this host, so ingest is not given a smaller buffer.+ assert_eq!(auto_memtable_budget(4 * GIB), 2 * GIB);++ // 6 GiB box → half of RAM, 3 GiB, not min(6, 8) = 6.let tiny = resolve_process_memory_cap(6 * GIB, AUTO, None);- assert_eq!(tiny.cap_mb, 8 * 1024, "the tier is chosen before the min");- assert_eq!(- cap_memory_limit(6 * GIB, tiny.cap_mb),- 6 * GIB,- "cap only lowers"- );+ assert_eq!(tiny.cap_mb, 3 * 1024);+ assert_eq!(cap_memory_limit(6 * GIB, tiny.cap_mb), 3 * GIB);+ }++ /// Under 16 GiB the cap is half of RAM. 16 GiB is the old 8 GiB step.+ #[test]+ fn a_host_under_sixteen_gib_is_capped_at_half_its_ram() {+ const GIB: u64 = 1024 * 1024 * 1024;+ assert_eq!(tiered_cap_mb(4 * GIB), 2 * 1024);+ assert_eq!(tiered_cap_mb(8 * GIB), 4 * 1024);+ assert_eq!(tiered_cap_mb(12 * GIB), 6 * 1024);+ // One byte under 16 GiB is still the half-RAM step, in whole MiB.+ assert_eq!(tiered_cap_mb(16 * GIB - 1), 8191);+ assert_eq!(tiered_cap_mb(16 * GIB), 8 * 1024);+ // And the 64 GiB step is unchanged.+ assert_eq!(tiered_cap_mb(64 * GIB - 1), 8 * 1024);}/// The tier steps exactly on the 64 GiB and 128 GiB binary boundaries.diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 1a3e12035..5a76524ab 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -10052,6 +10052,11 @@ pub struct Index {/// paying the old O(text) cost shows it here and nowhere else./// `Arc` because the merge encode runs on the merge pool, off the caller.merge_fts_reanalysed_docs: Arc<AtomicU64>,+ /// Set for the pass that just finished when at least one selected+ /// batch was left unmerged because its estimated working set would+ /// cross the memory watermark (#1122). The debounced check retries;+ /// a pass that merges everything leaves this false.+ merge_batches_deferred: std::sync::atomic::AtomicBool,}/// Counts one `Index::refresh` that got past the test hold, on every exit.@@ -10484,6 +10489,7 @@ impl Index {merge_scheduling_stopped: std::sync::atomic::AtomicBool::new(false),merge_evaluations: AtomicU64::new(0),merge_fts_reanalysed_docs: Arc::new(AtomicU64::new(0)),+ merge_batches_deferred: std::sync::atomic::AtomicBool::new(false),schema_persist_failures: AtomicU64::new(0),});// #871 — event-driven merge scheduling: the store fires this hook on@@ -10944,6 +10950,7 @@ impl Index {merge_scheduling_stopped: std::sync::atomic::AtomicBool::new(false),merge_evaluations: AtomicU64::new(0),merge_fts_reanalysed_docs: Arc::new(AtomicU64::new(0)),+ merge_batches_deferred: std::sync::atomic::AtomicBool::new(false),schema_persist_failures: AtomicU64::new(0),});// #871 — event-driven merge scheduling (see `Index::create` and@@ -13252,6 +13259,18 @@ impl Index {let n = self.merge_pass_locked(Some(target)).await?;total += n;let seg_count = self.store.snapshot().segments.len();+ // A small budget deferred every remaining batch. Waiting is+ // not "already at `target`": the inputs are still there.+ // The request timeout bounds the wait.+ if n == 0+ && seg_count > target+ && self+ .merge_batches_deferred+ .load(std::sync::atomic::Ordering::Relaxed)+ {+ tokio::time::sleep(std::time::Duration::from_secs(1)).await;+ continue;+ }if n == 0 || seg_count <= target {break;}@@ -13268,6 +13287,26 @@ impl Index {Ok(total)}+ /// Process-memory budget the merge scheduler plans against, in bytes.+ ///+ /// A positive numeric `XERJ_MAX_PROCESS_MEMORY_MB` wins. Otherwise the+ /// installed governor's resolved cap. `0` when neither is present+ /// (engine unit tests that never call `Engine::new`): planning then+ /// keeps the historical 512 MiB batch.+ fn merge_process_budget_bytes() -> u64 {+ const MB: u64 = 1024 * 1024;+ if let Some(limit_mb) = std::env::var("XERJ_MAX_PROCESS_MEMORY_MB")+ .ok()+ .and_then(|raw| raw.trim().parse::<u64>().ok())+ .filter(|mb| *mb > 0)+ {+ return limit_mb.saturating_mul(MB);+ }+ crate::governor::global()+ .map(|governor| governor.process_memory_limit_bytes())+ .unwrap_or(0)+ }+/// Upper bound on the total INPUT bytes of one merge batch. Issue #948.////// The merge executor materialises decoded and re-encoded copies of@@ -13282,17 +13321,19 @@ impl Index {/// `LogMergePolicy` filters candidates by size before batching/// (`max_docs_before_merge`, `src/indexer/log_merge_policy.rs:22,94`).///- /// Derived: `clamp(process cap / 64, 32 MiB, 512 MiB)` — at #948's- /// 16 GiB auto-cap that is a 256 MiB batch, i.e. a worst-case merge- /// working set of ~1.5-2.4 GiB at the measured multipliers.+ /// An explicit numeric `XERJ_MAX_PROCESS_MEMORY_MB` keeps+ /// `clamp(cap / 64, 32 MiB, 512 MiB)` — at a 16 GiB cap that is a+ /// 256 MiB batch. `auto`, `off`, and an unset variable read the+ /// governor's resolved budget ([`crate::merge_budget`]). A budget+ /// that still covers the historical 512 MiB batch keeps it, so a+ /// 16 GiB host does not merge in smaller pieces. A 4 GiB budget+ /// (the AUTO cap on an 8 GiB host) is 64 MiB./// `XERJ_MERGE_BATCH_MAX_INPUT_MB` overrides (0 disables the bound).- /// A non-numeric cap (`auto`, `off`) falls back to the 512 MiB- /// default.fn merge_batch_cap_bytes() -> u64 {const MB: u64 = 1024 * 1024;if let Ok(raw) = std::env::var("XERJ_MERGE_BATCH_MAX_INPUT_MB") {if let Ok(mb) = raw.trim().parse::<u64>() {- return mb * MB;+ return mb.saturating_mul(MB);}}match std::env::var("XERJ_MAX_PROCESS_MEMORY_MB")@@ -13300,7 +13341,9 @@ impl Index {.and_then(|raw| raw.trim().parse::<u64>().ok()){Some(limit_mb) if limit_mb > 0 => (limit_mb / 64).clamp(32, 512) * MB,- _ => 512 * MB,+ _ => {+ crate::merge_budget::merge_batch_input_cap_bytes(Self::merge_process_budget_bytes())+ }}}@@ -13324,6 +13367,8 @@ impl Index {// so it is exactly what the counter measures. Background debounced// checks, hand-driven `run_merge_once`, and `_forcemerge` all count.self.merge_evaluations.fetch_add(1, AtomicOrdering::Relaxed);+ self.merge_batches_deferred+ .store(false, AtomicOrdering::Relaxed);// V4 M4.5 — larger merge batches so a bursty-ingest index with// thousands of small segments converges in far fewer passes.@@ -13466,10 +13511,21 @@ impl Index {// each give ~2.4 GB peak and let the index converge in// predictable time without swapping. Override via env var// for machines with more RAM.- let merge_parallelism: usize = std::env::var("XERJ_MERGE_PARALLELISM")+ let configured_parallelism: usize = std::env::var("XERJ_MERGE_PARALLELISM").ok().and_then(|s| s.parse().ok()).unwrap_or(1);+ let merge_parallelism = crate::merge_budget::merge_parallelism_under_budget(+ configured_parallelism,+ Self::merge_process_budget_bytes(),+ );+ if merge_parallelism < configured_parallelism.max(1) {+ tracing::debug!(+ configured = configured_parallelism.max(1),+ running = merge_parallelism,+ "merge parallelism reduced to stay inside the process memory budget"+ );+ }use tokio::task::JoinHandle;type MergeOutput = (Vec<xerj_storage::segment::SegmentId>,@@ -13504,6 +13560,50 @@ impl Index {}}+ // #1122 — on a small process budget, leave batches whose estimated+ // working set would cross the watermark. The first batch still+ // runs: merge is what retires per-segment caches. A budget that+ // still uses the 512 MiB batch starts every batch, which is the+ // path a 16 GiB host and larger already took.+ let budget_bytes = Self::merge_process_budget_bytes();+ let (rss_bytes, watermark_bytes) = match crate::governor::global() {+ Some(governor) => (+ governor.observed_rss_bytes(),+ governor.memory_watermark_bytes(),+ ),+ None => (0, 0),+ };+ let input_bytes: Vec<u64> = launch_queue+ .iter()+ .map(|(_, metas)| metas.iter().map(|meta| meta.size_bytes).sum())+ .collect();+ let plan = crate::merge_budget::throttle_merge_batches(+ &input_bytes,+ budget_bytes,+ rss_bytes,+ watermark_bytes,+ );+ let mut deferred = 0usize;+ let mut kept = Vec::with_capacity(launch_queue.len());+ for (item, start) in launch_queue.into_iter().zip(plan) {+ if start {+ kept.push(item);+ } else {+ deferred += 1;+ }+ }+ let launch_queue = kept;+ if deferred > 0 {+ self.merge_batches_deferred+ .store(true, AtomicOrdering::Relaxed);+ tracing::info!(+ deferred,+ rss_mb = rss_bytes / (1024 * 1024),+ watermark_mb = watermark_bytes / (1024 * 1024),+ "merge batches deferred: estimated working set would cross the memory watermark; inputs kept"+ );+ }+// Derived once from the schema, then moved into each merge task: a// force-merge must not resurrect a doc-values column that flush// correctly skipped.@@ -14963,7 +15063,12 @@ impl Index {if *n > 0 {tracing::debug!(batches = n, "merge pass ran");}- false+ // Batches left behind under a small budget need+ // another check after the debounce sleep. A pass+ // that merged something already bumps+ // `merge_events`; this covers the pass that+ // started nothing because every extra batch waited.+ idx.merge_batches_deferred.load(AtomicOrdering::Relaxed)}};if retry || idx.merge_events.load(AtomicOrdering::SeqCst) != seen {diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rsindex bde8b93c3..9159caf93 100644--- a/engine/crates/xerj-engine/src/lib.rs+++ b/engine/crates/xerj-engine/src/lib.rs@@ -30,6 +30,7 @@ pub mod index_guard;pub mod ingest_memory;pub mod lifecycle;pub mod memtable;+mod merge_budget;mod merge_reserve;mod node_lock;pub mod painless;diff --git a/engine/crates/xerj-engine/src/merge_budget.rs b/engine/crates/xerj-engine/src/merge_budget.rsnew file mode 100644index 000000000..7a2d73c2f--- /dev/null+++ b/engine/crates/xerj-engine/src/merge_budget.rs@@ -0,0 +1,205 @@+//! Merge admission against the process memory budget (#1122).+//!+//! An 8 GiB host used to take the 8 GiB AUTO step, so the process budget+//! was the whole machine. A segment-merge storm then allocated a working+//! set measured at 5–9× the input bytes and the process died. These+//! functions size and defer batches from the resolved budget. They do+//! not allocate.+//!+//! A budget whose 95% watermark still covers a historical 512 MiB batch+//! (8× working set plus the 2 GiB memtable floor) keeps that batch and+//! the configured parallelism. That is the 8 GiB tier and above — a+//! 16 GiB host and larger — so those machines merge as they did before.++/// Historical input ceiling for one merge batch. The AUTO path used this+/// whenever `XERJ_MAX_PROCESS_MEMORY_MB` was unset.+pub const HISTORICAL_BATCH_CAP_BYTES: u64 = 512 * 1024 * 1024;++/// Smallest input cap. Below this a tier can stop making progress.+pub const MIN_BATCH_CAP_BYTES: u64 = 32 * 1024 * 1024;++/// Planning multiplier for one batch's heap. Mailbox-shaped merges were+/// measured at 5–9× the input bytes (`merge_parsed` held 191 MB on a+/// 20k-doc index). 8× is the figure a batch must fit under.+pub const MERGE_WORKING_SET_NUMERATOR: u64 = 8;++/// Upper bound on the input bytes of one merge batch.+///+/// `budget_bytes == 0` means the budget is unknown (no governor). That+/// keeps the historical 512 MiB ceiling. A positive budget that can hold+/// the historical batch also keeps 512 MiB. A smaller budget uses+/// `budget / 64`, clamped to [`MIN_BATCH_CAP_BYTES`]..=[`HISTORICAL_BATCH_CAP_BYTES`].+/// A 4 GiB budget — the AUTO cap on an 8 GiB host — is 64 MiB.+pub fn merge_batch_input_cap_bytes(budget_bytes: u64) -> u64 {+ if budget_bytes == 0 || historical_batch_fits(budget_bytes) {+ return HISTORICAL_BATCH_CAP_BYTES;+ }+ (budget_bytes / 64).clamp(MIN_BATCH_CAP_BYTES, HISTORICAL_BATCH_CAP_BYTES)+}++/// `true` when a 512 MiB batch plus the 2 GiB memtable floor still sits+/// under the 95% watermark of `budget_bytes`.+fn historical_batch_fits(budget_bytes: u64) -> bool {+ let historical = HISTORICAL_BATCH_CAP_BYTES+ .saturating_mul(MERGE_WORKING_SET_NUMERATOR)+ .saturating_add(2 * 1024 * 1024 * 1024);+ budget_bytes.saturating_mul(95) / 100 >= historical+}++/// How many merge batches may run at once.+///+/// A budget that still uses the historical 512 MiB batch returns+/// `configured` unchanged (at least 1). A smaller budget limits+/// concurrency so the batches' estimated working sets share a quarter+/// of the budget, and never drops below one.+pub fn merge_parallelism_under_budget(configured: usize, budget_bytes: u64) -> usize {+ let configured = configured.max(1);+ let cap = merge_batch_input_cap_bytes(budget_bytes);+ if cap >= HISTORICAL_BATCH_CAP_BYTES {+ return configured;+ }+ let per = cap.saturating_mul(MERGE_WORKING_SET_NUMERATOR).max(1);+ let slots = usize::try_from((budget_bytes / 4) / per).unwrap_or(usize::MAX);+ configured.min(slots.max(1))+}++/// Whether one batch may start, ignoring the one-batch drain exception.+///+/// `rss_bytes == 0` means RSS is unknown. `watermark_bytes == 0` means+/// the RSS check is off. Either one starts the batch: a guess must not+/// stall merges.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum MergeBatchDecision {+ Start,+ Defer,+}++pub fn decide_merge_batch(+ input_bytes: u64,+ rss_bytes: u64,+ watermark_bytes: u64,+) -> MergeBatchDecision {+ if watermark_bytes == 0 || rss_bytes == 0 {+ return MergeBatchDecision::Start;+ }+ let estimate = input_bytes.saturating_mul(MERGE_WORKING_SET_NUMERATOR);+ if rss_bytes.saturating_add(estimate) >= watermark_bytes {+ MergeBatchDecision::Defer+ } else {+ MergeBatchDecision::Start+ }+}++/// Which of `input_bytes` (one entry per selected batch, policy order)+/// starts in this pass.+///+/// On a budget that still uses the 512 MiB batch, every entry starts.+/// On a smaller budget, a batch whose estimated working set would cross+/// the watermark waits for the next check. The first batch still starts:+/// its input is already capped, and a merge is what retires the+/// per-segment caches that hold RSS up. Zero batches would leave that+/// RSS in place.+pub fn throttle_merge_batches(+ input_bytes: &[u64],+ budget_bytes: u64,+ rss_bytes: u64,+ watermark_bytes: u64,+) -> Vec<bool> {+ let small = merge_batch_input_cap_bytes(budget_bytes) < HISTORICAL_BATCH_CAP_BYTES;+ let mut start = Vec::with_capacity(input_bytes.len());+ let mut started = 0usize;+ for bytes in input_bytes {+ if !small {+ start.push(true);+ continue;+ }+ let fits =+ decide_merge_batch(*bytes, rss_bytes, watermark_bytes) == MergeBatchDecision::Start;+ if fits || started == 0 {+ start.push(true);+ started += 1;+ } else {+ start.push(false);+ }+ }+ start+}++#[cfg(test)]+mod tests {+ use super::*;++ const GIB: u64 = 1024 * 1024 * 1024;+ const MIB: u64 = 1024 * 1024;++ #[test]+ fn merges_are_throttled_under_a_small_budget() {+ // AUTO cap on an 8 GiB host: half of RAM, the setting that kept+ // the Windows 10 field report up (`XERJ_MAX_PROCESS_MEMORY_MB=4096`).+ let budget = 4 * GIB;+ let cap = merge_batch_input_cap_bytes(budget);+ assert_eq!(+ cap,+ 64 * MIB,+ "4 GiB / 64 = 64 MiB, not the 512 MiB default"+ );+ assert_eq!(+ merge_parallelism_under_budget(4, budget),+ 2,+ "four workers would hold 4 × 512 MiB; the budget allows two"+ );++ let watermark = budget * 95 / 100;+ // RSS already within 32 MiB of the watermark. One capped batch+ // still runs (it retires segments). The other two wait.+ let rss = watermark - 32 * MIB;+ let plan = throttle_merge_batches(&[cap, cap, cap], budget, rss, watermark);+ assert_eq!(plan, vec![true, false, false]);++ // The same batches with headroom all start. Parallelism is a+ // separate bound, applied by the caller.+ let open = throttle_merge_batches(&[cap, cap, cap], budget, 256 * MIB, watermark);+ assert_eq!(open, vec![true, true, true]);++ // Unknown RSS does not stall the drain.+ assert_eq!(+ decide_merge_batch(cap, 0, watermark),+ MergeBatchDecision::Start+ );+ assert_eq!(decide_merge_batch(cap, rss, 0), MergeBatchDecision::Start);+ }++ #[test]+ fn a_normal_host_keeps_the_historical_batch_and_parallelism() {+ // The 8 GiB tier (16 GiB of RAM and up, under 64 GiB) and the+ // 16 / 32 GiB tiers all cover a 512 MiB batch under the watermark.+ for budget in [8 * GIB, 16 * GIB, 32 * GIB] {+ assert_eq!(+ merge_batch_input_cap_bytes(budget),+ HISTORICAL_BATCH_CAP_BYTES+ );+ assert_eq!(merge_parallelism_under_budget(4, budget), 4);+ let watermark = budget * 95 / 100;+ let plan = throttle_merge_batches(+ &[HISTORICAL_BATCH_CAP_BYTES; 3],+ budget,+ watermark,+ watermark,+ );+ assert_eq!(+ plan,+ vec![true, true, true],+ "budget {budget} must not defer a historical batch"+ );+ }+ // No governor: the pre-change fallback.+ assert_eq!(merge_batch_input_cap_bytes(0), HISTORICAL_BATCH_CAP_BYTES);+ assert_eq!(merge_parallelism_under_budget(1, 0), 1);+ }++ #[test]+ fn the_batch_cap_floors_at_thirty_two_mebibytes() {+ assert_eq!(merge_batch_input_cap_bytes(GIB), MIN_BATCH_CAP_BYTES);+ assert_eq!(merge_batch_input_cap_bytes(2 * GIB), MIN_BATCH_CAP_BYTES);+ }+}diff --git a/engine/crates/xerj-engine/tests/es_compat_tests.rs b/engine/crates/xerj-engine/tests/es_compat_tests.rsindex 17d1f4444..6322c6cc2 100644--- a/engine/crates/xerj-engine/tests/es_compat_tests.rs+++ b/engine/crates/xerj-engine/tests/es_compat_tests.rs@@ -814,6 +814,9 @@ async fn test_es_bulk_index_multiple() {/// One JSON document over [`MAX_INDEXED_DOCUMENT_BYTES`] must not be parsed/// or tokenized. Its neighbour in the same bulk still indexes, and a later/// single-document write of the same shape is refused the same way (#1122).+/// A 50 MiB file is the same gate: anything past 8 MiB, including the+/// single JSON files that killed a Windows node, is refused on its length+/// before `serde_json`.#[tokio::test]async fn oversized_json_document_is_refused_and_its_neighbour_is_indexed() {use xerj_common::document_limit::{indexed_document_too_large, MAX_INDEXED_DOCUMENT_BYTES};@@ -823,6 +826,7 @@ async fn oversized_json_document_is_refused_and_its_neighbour_is_indexed() {.create_index("bulk_oversize", Schema::empty()).unwrap();+ assert!(indexed_document_too_large(50 * 1024 * 1024));let big = "x".repeat(MAX_INDEXED_DOCUMENT_BYTES + 1);let big_line = format!(r#"{{"blob":"{big}"}}"#);assert!(indexed_document_too_large(big_line.len()));diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex b457cb12b..ed274f047 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -476,8 +476,9 @@ timeout_ms = 5000# Process-wide memory ceiling (MiB) that bounds every internal budget — segment# hydration, query working sets, and the ingest circuit breaker. OMIT it (the-# default) to AUTO-size the cap from the machine in binary-GiB tiers-# (8 / 16 / 32 GiB): a laptop stays bounded, a large box climbs. Set an explicit+# default) to AUTO-size the cap from the machine: half of RAM under 16 GiB,+# then 8 / 16 / 32 GiB. An 8 GiB host gets 4 GiB, so a merge has room; a+# 16 GiB host stays at 8 GiB and a large box climbs. Set an explicit# MiB ceiling to raise headroom on a big host — e.g. when a large `xerj autoindex`# trips `429 circuit_breaking_exception`. Set 0 to uncap to the whole machine# (not recommended). Env override: XERJ_MAX_PROCESS_MEMORY_MB=auto|off|<MiB>.