Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): size the default memory cap and merge batches for small hosts (#1122) (#2340)

xerj-org2819cc9541d4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4e33643c0..48426721c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -130,6 +130,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
recorded only on a catalog dataset document
([#2321](https://github.com/xerj-org/xerj/issues/2321)).
+- The default process memory cap on a host under 16 GiB is half of that
+ RAM, and segment merges follow it (#1122). An 8 GiB machine used to
+ take the 8 GiB step, so the budget was the whole machine; a merge
+ storm on Windows 10 stayed up only after
+ `XERJ_MAX_PROCESS_MEMORY_MB=4096`. That host now gets a 4 GiB cap.
+ A merge batch on a budget that cannot hold the historical 512 MiB
+ input (8× working set plus the 2 GiB memtable floor) is sized to
+ `budget / 64` (64 MiB at 4 GiB) and concurrent merges share a quarter
+ of the budget. A batch that would still cross the RSS watermark waits
+ for the next check; one batch still runs, because a merge is what
+ retires per-segment caches. Hosts at 16 GiB and above keep the 8 / 16 /
+ 32 GiB steps, the 512 MiB batch, and the configured merge parallelism.
+ A single JSON document over 50 MiB is still refused at the 8 MiB
+ ceiling, before it is parsed, and the rest of the bulk is indexed.
+
- A multi-term question no longer ranks a one-word symbol definition
(`type Token`) above a hit that covers more of the question. `xerj search`,
`xerj def`, `xerj code`, and the plain-string `xerj_search` tool add a
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 4cfcb02aa..7c54f70a5 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -1740,9 +1740,10 @@ impl Default for EmbeddingConfig {
// ─────────────────────────────────────────────────────────────────────────────
/// Sentinel value of [`LimitsConfig::max_process_memory_mb`] meaning **AUTO**:
-/// pick the process-memory cap from the machine size in binary-GiB steps
-/// (8 / 16 / 32 GiB) rather than a flat number. It is the serde default, so an
-/// omitted field resolves to AUTO; the governor intercepts it before any budget
+/// pick the process-memory cap from the machine size (half of RAM under
+/// 16 GiB, then 8 / 16 / 32 GiB) rather than a flat number. It is the serde
+/// default, so an omitted field resolves to AUTO; the governor intercepts it
+/// before any budget
/// math, so this literal never reaches a budget as a byte count. `u64::MAX` is
/// used because no operator sets a 16-EiB cap — it is a reserved, unmistakable
/// "not a real MiB value" marker, distinct from `0` (uncapped, the whole
@@ -1866,14 +1867,19 @@ pub struct LimitsConfig {
/// Ceiling on the machine size every other memory budget is derived FROM,
/// in MiB. **Omitted / default = AUTO** ([`AUTO_PROCESS_MEMORY_MB`]): XERJ
/// picks the cap from the effective (cgroup-aware) memory it can actually
- /// see, in binary-GiB steps —
+ /// see —
///
- /// effective < 64 GiB → 8 GiB cap
+ /// effective < 16 GiB → half of that RAM
+ /// 16 GiB ≤ effective < 64 GiB → 8 GiB cap
/// 64 GiB ≤ effective < 128 GiB → 16 GiB cap
/// effective ≥ 128 GiB → 32 GiB cap
///
/// so a laptop is never sized for ~20 GiB of memtables and a big server is
- /// no longer starved by a flat 8 GiB. Every budget in `ResourceGovernor`
+ /// no longer starved by a flat 8 GiB. A host under 16 GiB used to take the
+ /// 8 GiB step and then the machine, so an 8 GiB box's budget was all of
+ /// its RAM; half of RAM leaves room for the OS during a merge (#1122).
+ /// At 16 GiB the half-RAM step and the 8 GiB step are the same number.
+ /// Every budget in `ResourceGovernor`
/// derives from `effective_memory_limit_bytes()` = min(cgroup limit, total
/// system RAM); capping that ONE base is what makes every dependent budget
/// shrink coherently instead of each growing its own ceiling.
@@ -1918,7 +1924,7 @@ impl Default for LimitsConfig {
max_segment_hydration_cache_mb: 0, // 0 = 20% effective memory, no floor
memory_watermark_percent: 95,
disk_flood_stage_percent: 95,
- // AUTO: tier the cap (8/16/32 GiB) off the machine at governor
+ // AUTO: half of RAM under 16 GiB, then 8/16/32 GiB, at governor
// build time. 0 = uncapped (whole machine); N = a fixed MiB cap.
max_process_memory_mb: AUTO_PROCESS_MEMORY_MB,
}
diff --git a/engine/crates/xerj-common/src/document_limit.rs b/engine/crates/xerj-common/src/document_limit.rs
index 1ff92c757..a536b19c6 100644
--- a/engine/crates/xerj-common/src/document_limit.rs
+++ b/engine/crates/xerj-common/src/document_limit.rs
@@ -158,6 +158,22 @@ mod tests {
);
}
+ #[test]
+ fn a_fifty_mebibyte_document_is_refused_before_it_is_indexed() {
+ // Field report on #1122: a single JSON file over 50 MB killed the
+ // node. The ceiling is 8 MiB, so that file is the same refusal as
+ // one byte over the ceiling — checked on the length, before parse.
+ let len = 50 * 1024 * 1024;
+ assert!(indexed_document_too_large(len));
+ let text = indexed_document_refusal(len);
+ assert!(text.contains("52428800"), "{text}");
+ assert!(
+ text.contains(&MAX_INDEXED_DOCUMENT_BYTES.to_string()),
+ "{text}"
+ );
+ assert!(text.contains("not indexed"), "{text}");
+ }
+
#[test]
fn refusal_names_the_size_and_the_limit() {
let text = indexed_document_refusal(50 * 1024 * 1024);
diff --git a/engine/crates/xerj-engine/src/governor.rs b/engine/crates/xerj-engine/src/governor.rs
index 96430277f..a593642f0 100644
--- a/engine/crates/xerj-engine/src/governor.rs
+++ b/engine/crates/xerj-engine/src/governor.rs
@@ -413,6 +413,25 @@ impl ResourceGovernor {
))
}
+ /// Resolved process-memory ceiling in bytes. `0` is not used: an
+ /// uncapped config stores the machine size here. Merge planning reads
+ /// this so a small host's batches follow the AUTO cap instead of a
+ /// fixed 512 MiB.
+ pub fn process_memory_limit_bytes(&self) -> u64 {
+ self.memory_limit_bytes
+ }
+
+ /// RSS admission watermark in bytes. `0` means the check is off.
+ pub fn memory_watermark_bytes(&self) -> u64 {
+ self.memory_watermark_bytes.load(Ordering::Relaxed)
+ }
+
+ /// Larger of the last RSS sample and a live read. `0` when this
+ /// platform cannot read process memory.
+ pub fn observed_rss_bytes(&self) -> u64 {
+ self.observed_process_memory()
+ }
+
/// Whether the parent RSS memory breaker is currently engaged (issue
/// #948). Admission (429) is only half of the contract: the resource
/// sampler uses this flag, together with [`Self::memtable_breaker_engaged`],
@@ -1033,9 +1052,10 @@ fn windows_override_ceiling(
}
/// The AUTO process-memory cap, in MiB, chosen from the effective memory the
-/// process can actually see, in binary-GiB steps:
+/// process can actually see:
///
-/// effective < 64 GiB → 8 GiB
+/// effective < 16 GiB → half of that RAM
+/// 16 GiB ≤ effective < 64 GiB → 8 GiB
/// 64 GiB ≤ effective < 128 GiB → 16 GiB
/// effective ≥ 128 GiB → 32 GiB
///
@@ -1043,11 +1063,21 @@ fn windows_override_ceiling(
/// `effective_memory_limit_bytes` — not raw host RAM, so a container smaller
/// than its host is sized for the container. This only *chooses* a ceiling;
/// `cap_memory_limit` still mins it with the machine, so the cap only ever
-/// LOWERS a budget: on a 40 GiB box the 8 GiB tier yields min(40, 8) = 8, and
-/// on a 6 GiB box it yields min(6, 8) = 6. It never invents headroom.
+/// LOWERS a budget: on a 40 GiB box the 8 GiB tier yields min(40, 8) = 8.
+///
+/// Hosts under 16 GiB used to take the 8 GiB step and then `min` with the
+/// machine, so an 8 GiB box's process budget was all of its RAM. A merge
+/// storm on that budget died; the same box stayed up at a 4 GiB cap
+/// (#1122). Half of RAM is that cap. At 16 GiB, half is 8 GiB, so the
+/// steps above 16 GiB — and the memtable budget they derive — do not move.
+/// A sub-mebibyte reading rounds up to 1 MiB and is still min'd with the
+/// machine, so the floor does not invent RAM.
fn tiered_cap_mb(effective_bytes: u64) -> u64 {
const GIB: u64 = 1024 * 1024 * 1024;
- if effective_bytes < 64 * GIB {
+ const MIB: u64 = 1024 * 1024;
+ if effective_bytes < 16 * GIB {
+ (effective_bytes / 2 / MIB).max(1)
+ } else if effective_bytes < 64 * GIB {
8 * 1024
} else if effective_bytes < 128 * GIB {
16 * 1024
@@ -1295,7 +1325,8 @@ fn build(config: &Config) -> ResourceGovernor {
//
// Capped by `limits.max_process_memory_mb`, resolved through
// `resolve_process_memory_cap`: the AUTO default tiers the cap off the
- // machine (8/16/32 GiB), `0` means uncapped (whole machine), an explicit
+ // machine (half of RAM under 16 GiB, then 8/16/32 GiB), `0` means uncapped
+ // (whole machine), an explicit
// `N` forces a fixed MiB ceiling, and `XERJ_MAX_PROCESS_MEMORY_MB` can
// override any of it. Every budget below derives from this one number, so
// capping HERE is what makes the ceiling coherent — capping each dependent
@@ -2138,14 +2169,33 @@ mod tests {
assert_eq!(cap_memory_limit(40 * GIB, laptop.cap_mb), 8 * GIB);
assert_eq!(auto_memtable_budget(8 * GIB), 2 * GIB);
- // 6 GiB box → 8 GiB tier, but the machine only has 6, so min = 6.
+ // 8 GiB host → half of RAM, 4 GiB. The old tier was min(8, 8) = 8,
+ // the whole machine; a merge storm on that budget is #1122.
+ let small = resolve_process_memory_cap(8 * GIB, AUTO, None);
+ assert_eq!(small.cap_mb, 4 * 1024);
+ assert_eq!(cap_memory_limit(8 * GIB, small.cap_mb), 4 * GIB);
+ // The memtable floor stays 2 GiB, the same budget the old 8 GiB
+ // tier handed this host, so ingest is not given a smaller buffer.
+ assert_eq!(auto_memtable_budget(4 * GIB), 2 * GIB);
+
+ // 6 GiB box → half of RAM, 3 GiB, not min(6, 8) = 6.
let tiny = resolve_process_memory_cap(6 * GIB, AUTO, None);
- assert_eq!(tiny.cap_mb, 8 * 1024, "the tier is chosen before the min");
- assert_eq!(
- cap_memory_limit(6 * GIB, tiny.cap_mb),
- 6 * GIB,
- "cap only lowers"
- );
+ assert_eq!(tiny.cap_mb, 3 * 1024);
+ assert_eq!(cap_memory_limit(6 * GIB, tiny.cap_mb), 3 * GIB);
+ }
+
+ /// Under 16 GiB the cap is half of RAM. 16 GiB is the old 8 GiB step.
+ #[test]
+ fn a_host_under_sixteen_gib_is_capped_at_half_its_ram() {
+ const GIB: u64 = 1024 * 1024 * 1024;
+ assert_eq!(tiered_cap_mb(4 * GIB), 2 * 1024);
+ assert_eq!(tiered_cap_mb(8 * GIB), 4 * 1024);
+ assert_eq!(tiered_cap_mb(12 * GIB), 6 * 1024);
+ // One byte under 16 GiB is still the half-RAM step, in whole MiB.
+ assert_eq!(tiered_cap_mb(16 * GIB - 1), 8191);
+ assert_eq!(tiered_cap_mb(16 * GIB), 8 * 1024);
+ // And the 64 GiB step is unchanged.
+ assert_eq!(tiered_cap_mb(64 * GIB - 1), 8 * 1024);
}
/// The tier steps exactly on the 64 GiB and 128 GiB binary boundaries.
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 1a3e12035..5a76524ab 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -10052,6 +10052,11 @@ pub struct Index {
/// paying the old O(text) cost shows it here and nowhere else.
/// `Arc` because the merge encode runs on the merge pool, off the caller.
merge_fts_reanalysed_docs: Arc<AtomicU64>,
+ /// Set for the pass that just finished when at least one selected
+ /// batch was left unmerged because its estimated working set would
+ /// cross the memory watermark (#1122). The debounced check retries;
+ /// a pass that merges everything leaves this false.
+ merge_batches_deferred: std::sync::atomic::AtomicBool,
}
/// Counts one `Index::refresh` that got past the test hold, on every exit.
@@ -10484,6 +10489,7 @@ impl Index {
merge_scheduling_stopped: std::sync::atomic::AtomicBool::new(false),
merge_evaluations: AtomicU64::new(0),
merge_fts_reanalysed_docs: Arc::new(AtomicU64::new(0)),
+ merge_batches_deferred: std::sync::atomic::AtomicBool::new(false),
schema_persist_failures: AtomicU64::new(0),
});
// #871 — event-driven merge scheduling: the store fires this hook on
@@ -10944,6 +10950,7 @@ impl Index {
merge_scheduling_stopped: std::sync::atomic::AtomicBool::new(false),
merge_evaluations: AtomicU64::new(0),
merge_fts_reanalysed_docs: Arc::new(AtomicU64::new(0)),
+ merge_batches_deferred: std::sync::atomic::AtomicBool::new(false),
schema_persist_failures: AtomicU64::new(0),
});
// #871 — event-driven merge scheduling (see `Index::create` and
@@ -13252,6 +13259,18 @@ impl Index {
let n = self.merge_pass_locked(Some(target)).await?;
total += n;
let seg_count = self.store.snapshot().segments.len();
+ // A small budget deferred every remaining batch. Waiting is
+ // not "already at `target`": the inputs are still there.
+ // The request timeout bounds the wait.
+ if n == 0
+ && seg_count > target
+ && self
+ .merge_batches_deferred
+ .load(std::sync::atomic::Ordering::Relaxed)
+ {
+ tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+ continue;
+ }
if n == 0 || seg_count <= target {
break;
}
@@ -13268,6 +13287,26 @@ impl Index {
Ok(total)
}
+ /// Process-memory budget the merge scheduler plans against, in bytes.
+ ///
+ /// A positive numeric `XERJ_MAX_PROCESS_MEMORY_MB` wins. Otherwise the
+ /// installed governor's resolved cap. `0` when neither is present
+ /// (engine unit tests that never call `Engine::new`): planning then
+ /// keeps the historical 512 MiB batch.
+ fn merge_process_budget_bytes() -> u64 {
+ const MB: u64 = 1024 * 1024;
+ if let Some(limit_mb) = std::env::var("XERJ_MAX_PROCESS_MEMORY_MB")
+ .ok()
+ .and_then(|raw| raw.trim().parse::<u64>().ok())
+ .filter(|mb| *mb > 0)
+ {
+ return limit_mb.saturating_mul(MB);
+ }
+ crate::governor::global()
+ .map(|governor| governor.process_memory_limit_bytes())
+ .unwrap_or(0)
+ }
+
/// Upper bound on the total INPUT bytes of one merge batch. Issue #948.
///
/// The merge executor materialises decoded and re-encoded copies of
@@ -13282,17 +13321,19 @@ impl Index {
/// `LogMergePolicy` filters candidates by size before batching
/// (`max_docs_before_merge`, `src/indexer/log_merge_policy.rs:22,94`).
///
- /// Derived: `clamp(process cap / 64, 32 MiB, 512 MiB)` — at #948's
- /// 16 GiB auto-cap that is a 256 MiB batch, i.e. a worst-case merge
- /// working set of ~1.5-2.4 GiB at the measured multipliers.
+ /// An explicit numeric `XERJ_MAX_PROCESS_MEMORY_MB` keeps
+ /// `clamp(cap / 64, 32 MiB, 512 MiB)` — at a 16 GiB cap that is a
+ /// 256 MiB batch. `auto`, `off`, and an unset variable read the
+ /// governor's resolved budget ([`crate::merge_budget`]). A budget
+ /// that still covers the historical 512 MiB batch keeps it, so a
+ /// 16 GiB host does not merge in smaller pieces. A 4 GiB budget
+ /// (the AUTO cap on an 8 GiB host) is 64 MiB.
/// `XERJ_MERGE_BATCH_MAX_INPUT_MB` overrides (0 disables the bound).
- /// A non-numeric cap (`auto`, `off`) falls back to the 512 MiB
- /// default.
fn merge_batch_cap_bytes() -> u64 {
const MB: u64 = 1024 * 1024;
if let Ok(raw) = std::env::var("XERJ_MERGE_BATCH_MAX_INPUT_MB") {
if let Ok(mb) = raw.trim().parse::<u64>() {
- return mb * MB;
+ return mb.saturating_mul(MB);
}
}
match std::env::var("XERJ_MAX_PROCESS_MEMORY_MB")
@@ -13300,7 +13341,9 @@ impl Index {
.and_then(|raw| raw.trim().parse::<u64>().ok())
{
Some(limit_mb) if limit_mb > 0 => (limit_mb / 64).clamp(32, 512) * MB,
- _ => 512 * MB,
+ _ => {
+ crate::merge_budget::merge_batch_input_cap_bytes(Self::merge_process_budget_bytes())
+ }
}
}
@@ -13324,6 +13367,8 @@ impl Index {
// so it is exactly what the counter measures. Background debounced
// checks, hand-driven `run_merge_once`, and `_forcemerge` all count.
self.merge_evaluations.fetch_add(1, AtomicOrdering::Relaxed);
+ self.merge_batches_deferred
+ .store(false, AtomicOrdering::Relaxed);
// V4 M4.5 — larger merge batches so a bursty-ingest index with
// thousands of small segments converges in far fewer passes.
@@ -13466,10 +13511,21 @@ impl Index {
// each give ~2.4 GB peak and let the index converge in
// predictable time without swapping. Override via env var
// for machines with more RAM.
- let merge_parallelism: usize = std::env::var("XERJ_MERGE_PARALLELISM")
+ let configured_parallelism: usize = std::env::var("XERJ_MERGE_PARALLELISM")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(1);
+ let merge_parallelism = crate::merge_budget::merge_parallelism_under_budget(
+ configured_parallelism,
+ Self::merge_process_budget_bytes(),
+ );
+ if merge_parallelism < configured_parallelism.max(1) {
+ tracing::debug!(
+ configured = configured_parallelism.max(1),
+ running = merge_parallelism,
+ "merge parallelism reduced to stay inside the process memory budget"
+ );
+ }
use tokio::task::JoinHandle;
type MergeOutput = (
Vec<xerj_storage::segment::SegmentId>,
@@ -13504,6 +13560,50 @@ impl Index {
}
}
+ // #1122 — on a small process budget, leave batches whose estimated
+ // working set would cross the watermark. The first batch still
+ // runs: merge is what retires per-segment caches. A budget that
+ // still uses the 512 MiB batch starts every batch, which is the
+ // path a 16 GiB host and larger already took.
+ let budget_bytes = Self::merge_process_budget_bytes();
+ let (rss_bytes, watermark_bytes) = match crate::governor::global() {
+ Some(governor) => (
+ governor.observed_rss_bytes(),
+ governor.memory_watermark_bytes(),
+ ),
+ None => (0, 0),
+ };
+ let input_bytes: Vec<u64> = launch_queue
+ .iter()
+ .map(|(_, metas)| metas.iter().map(|meta| meta.size_bytes).sum())
+ .collect();
+ let plan = crate::merge_budget::throttle_merge_batches(
+ &input_bytes,
+ budget_bytes,
+ rss_bytes,
+ watermark_bytes,
+ );
+ let mut deferred = 0usize;
+ let mut kept = Vec::with_capacity(launch_queue.len());
+ for (item, start) in launch_queue.into_iter().zip(plan) {
+ if start {
+ kept.push(item);
+ } else {
+ deferred += 1;
+ }
+ }
+ let launch_queue = kept;
+ if deferred > 0 {
+ self.merge_batches_deferred
+ .store(true, AtomicOrdering::Relaxed);
+ tracing::info!(
+ deferred,
+ rss_mb = rss_bytes / (1024 * 1024),
+ watermark_mb = watermark_bytes / (1024 * 1024),
+ "merge batches deferred: estimated working set would cross the memory watermark; inputs kept"
+ );
+ }
+
// Derived once from the schema, then moved into each merge task: a
// force-merge must not resurrect a doc-values column that flush
// correctly skipped.
@@ -14963,7 +15063,12 @@ impl Index {
if *n > 0 {
tracing::debug!(batches = n, "merge pass ran");
}
- false
+ // Batches left behind under a small budget need
+ // another check after the debounce sleep. A pass
+ // that merged something already bumps
+ // `merge_events`; this covers the pass that
+ // started nothing because every extra batch waited.
+ idx.merge_batches_deferred.load(AtomicOrdering::Relaxed)
}
};
if retry || idx.merge_events.load(AtomicOrdering::SeqCst) != seen {
diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rs
index bde8b93c3..9159caf93 100644
--- a/engine/crates/xerj-engine/src/lib.rs
+++ b/engine/crates/xerj-engine/src/lib.rs
@@ -30,6 +30,7 @@ pub mod index_guard;
pub mod ingest_memory;
pub mod lifecycle;
pub mod memtable;
+mod merge_budget;
mod merge_reserve;
mod node_lock;
pub mod painless;
diff --git a/engine/crates/xerj-engine/src/merge_budget.rs b/engine/crates/xerj-engine/src/merge_budget.rs
new file mode 100644
index 000000000..7a2d73c2f
--- /dev/null
+++ b/engine/crates/xerj-engine/src/merge_budget.rs
@@ -0,0 +1,205 @@
+//! Merge admission against the process memory budget (#1122).
+//!
+//! An 8 GiB host used to take the 8 GiB AUTO step, so the process budget
+//! was the whole machine. A segment-merge storm then allocated a working
+//! set measured at 5–9× the input bytes and the process died. These
+//! functions size and defer batches from the resolved budget. They do
+//! not allocate.
+//!
+//! A budget whose 95% watermark still covers a historical 512 MiB batch
+//! (8× working set plus the 2 GiB memtable floor) keeps that batch and
+//! the configured parallelism. That is the 8 GiB tier and above — a
+//! 16 GiB host and larger — so those machines merge as they did before.
+
+/// Historical input ceiling for one merge batch. The AUTO path used this
+/// whenever `XERJ_MAX_PROCESS_MEMORY_MB` was unset.
+pub const HISTORICAL_BATCH_CAP_BYTES: u64 = 512 * 1024 * 1024;
+
+/// Smallest input cap. Below this a tier can stop making progress.
+pub const MIN_BATCH_CAP_BYTES: u64 = 32 * 1024 * 1024;
+
+/// Planning multiplier for one batch's heap. Mailbox-shaped merges were
+/// measured at 5–9× the input bytes (`merge_parsed` held 191 MB on a
+/// 20k-doc index). 8× is the figure a batch must fit under.
+pub const MERGE_WORKING_SET_NUMERATOR: u64 = 8;
+
+/// Upper bound on the input bytes of one merge batch.
+///
+/// `budget_bytes == 0` means the budget is unknown (no governor). That
+/// keeps the historical 512 MiB ceiling. A positive budget that can hold
+/// the historical batch also keeps 512 MiB. A smaller budget uses
+/// `budget / 64`, clamped to [`MIN_BATCH_CAP_BYTES`]..=[`HISTORICAL_BATCH_CAP_BYTES`].
+/// A 4 GiB budget — the AUTO cap on an 8 GiB host — is 64 MiB.
+pub fn merge_batch_input_cap_bytes(budget_bytes: u64) -> u64 {
+ if budget_bytes == 0 || historical_batch_fits(budget_bytes) {
+ return HISTORICAL_BATCH_CAP_BYTES;
+ }
+ (budget_bytes / 64).clamp(MIN_BATCH_CAP_BYTES, HISTORICAL_BATCH_CAP_BYTES)
+}
+
+/// `true` when a 512 MiB batch plus the 2 GiB memtable floor still sits
+/// under the 95% watermark of `budget_bytes`.
+fn historical_batch_fits(budget_bytes: u64) -> bool {
+ let historical = HISTORICAL_BATCH_CAP_BYTES
+ .saturating_mul(MERGE_WORKING_SET_NUMERATOR)
+ .saturating_add(2 * 1024 * 1024 * 1024);
+ budget_bytes.saturating_mul(95) / 100 >= historical
+}
+
+/// How many merge batches may run at once.
+///
+/// A budget that still uses the historical 512 MiB batch returns
+/// `configured` unchanged (at least 1). A smaller budget limits
+/// concurrency so the batches' estimated working sets share a quarter
+/// of the budget, and never drops below one.
+pub fn merge_parallelism_under_budget(configured: usize, budget_bytes: u64) -> usize {
+ let configured = configured.max(1);
+ let cap = merge_batch_input_cap_bytes(budget_bytes);
+ if cap >= HISTORICAL_BATCH_CAP_BYTES {
+ return configured;
+ }
+ let per = cap.saturating_mul(MERGE_WORKING_SET_NUMERATOR).max(1);
+ let slots = usize::try_from((budget_bytes / 4) / per).unwrap_or(usize::MAX);
+ configured.min(slots.max(1))
+}
+
+/// Whether one batch may start, ignoring the one-batch drain exception.
+///
+/// `rss_bytes == 0` means RSS is unknown. `watermark_bytes == 0` means
+/// the RSS check is off. Either one starts the batch: a guess must not
+/// stall merges.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum MergeBatchDecision {
+ Start,
+ Defer,
+}
+
+pub fn decide_merge_batch(
+ input_bytes: u64,
+ rss_bytes: u64,
+ watermark_bytes: u64,
+) -> MergeBatchDecision {
+ if watermark_bytes == 0 || rss_bytes == 0 {
+ return MergeBatchDecision::Start;
+ }
+ let estimate = input_bytes.saturating_mul(MERGE_WORKING_SET_NUMERATOR);
+ if rss_bytes.saturating_add(estimate) >= watermark_bytes {
+ MergeBatchDecision::Defer
+ } else {
+ MergeBatchDecision::Start
+ }
+}
+
+/// Which of `input_bytes` (one entry per selected batch, policy order)
+/// starts in this pass.
+///
+/// On a budget that still uses the 512 MiB batch, every entry starts.
+/// On a smaller budget, a batch whose estimated working set would cross
+/// the watermark waits for the next check. The first batch still starts:
+/// its input is already capped, and a merge is what retires the
+/// per-segment caches that hold RSS up. Zero batches would leave that
+/// RSS in place.
+pub fn throttle_merge_batches(
+ input_bytes: &[u64],
+ budget_bytes: u64,
+ rss_bytes: u64,
+ watermark_bytes: u64,
+) -> Vec<bool> {
+ let small = merge_batch_input_cap_bytes(budget_bytes) < HISTORICAL_BATCH_CAP_BYTES;
+ let mut start = Vec::with_capacity(input_bytes.len());
+ let mut started = 0usize;
+ for bytes in input_bytes {
+ if !small {
+ start.push(true);
+ continue;
+ }
+ let fits =
+ decide_merge_batch(*bytes, rss_bytes, watermark_bytes) == MergeBatchDecision::Start;
+ if fits || started == 0 {
+ start.push(true);
+ started += 1;
+ } else {
+ start.push(false);
+ }
+ }
+ start
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const GIB: u64 = 1024 * 1024 * 1024;
+ const MIB: u64 = 1024 * 1024;
+
+ #[test]
+ fn merges_are_throttled_under_a_small_budget() {
+ // AUTO cap on an 8 GiB host: half of RAM, the setting that kept
+ // the Windows 10 field report up (`XERJ_MAX_PROCESS_MEMORY_MB=4096`).
+ let budget = 4 * GIB;
+ let cap = merge_batch_input_cap_bytes(budget);
+ assert_eq!(
+ cap,
+ 64 * MIB,
+ "4 GiB / 64 = 64 MiB, not the 512 MiB default"
+ );
+ assert_eq!(
+ merge_parallelism_under_budget(4, budget),
+ 2,
+ "four workers would hold 4 × 512 MiB; the budget allows two"
+ );
+
+ let watermark = budget * 95 / 100;
+ // RSS already within 32 MiB of the watermark. One capped batch
+ // still runs (it retires segments). The other two wait.
+ let rss = watermark - 32 * MIB;
+ let plan = throttle_merge_batches(&[cap, cap, cap], budget, rss, watermark);
+ assert_eq!(plan, vec![true, false, false]);
+
+ // The same batches with headroom all start. Parallelism is a
+ // separate bound, applied by the caller.
+ let open = throttle_merge_batches(&[cap, cap, cap], budget, 256 * MIB, watermark);
+ assert_eq!(open, vec![true, true, true]);
+
+ // Unknown RSS does not stall the drain.
+ assert_eq!(
+ decide_merge_batch(cap, 0, watermark),
+ MergeBatchDecision::Start
+ );
+ assert_eq!(decide_merge_batch(cap, rss, 0), MergeBatchDecision::Start);
+ }
+
+ #[test]
+ fn a_normal_host_keeps_the_historical_batch_and_parallelism() {
+ // The 8 GiB tier (16 GiB of RAM and up, under 64 GiB) and the
+ // 16 / 32 GiB tiers all cover a 512 MiB batch under the watermark.
+ for budget in [8 * GIB, 16 * GIB, 32 * GIB] {
+ assert_eq!(
+ merge_batch_input_cap_bytes(budget),
+ HISTORICAL_BATCH_CAP_BYTES
+ );
+ assert_eq!(merge_parallelism_under_budget(4, budget), 4);
+ let watermark = budget * 95 / 100;
+ let plan = throttle_merge_batches(
+ &[HISTORICAL_BATCH_CAP_BYTES; 3],
+ budget,
+ watermark,
+ watermark,
+ );
+ assert_eq!(
+ plan,
+ vec![true, true, true],
+ "budget {budget} must not defer a historical batch"
+ );
+ }
+ // No governor: the pre-change fallback.
+ assert_eq!(merge_batch_input_cap_bytes(0), HISTORICAL_BATCH_CAP_BYTES);
+ assert_eq!(merge_parallelism_under_budget(1, 0), 1);
+ }
+
+ #[test]
+ fn the_batch_cap_floors_at_thirty_two_mebibytes() {
+ assert_eq!(merge_batch_input_cap_bytes(GIB), MIN_BATCH_CAP_BYTES);
+ assert_eq!(merge_batch_input_cap_bytes(2 * GIB), MIN_BATCH_CAP_BYTES);
+ }
+}
diff --git a/engine/crates/xerj-engine/tests/es_compat_tests.rs b/engine/crates/xerj-engine/tests/es_compat_tests.rs
index 17d1f4444..6322c6cc2 100644
--- a/engine/crates/xerj-engine/tests/es_compat_tests.rs
+++ b/engine/crates/xerj-engine/tests/es_compat_tests.rs
@@ -814,6 +814,9 @@ async fn test_es_bulk_index_multiple() {
/// One JSON document over [`MAX_INDEXED_DOCUMENT_BYTES`] must not be parsed
/// or tokenized. Its neighbour in the same bulk still indexes, and a later
/// single-document write of the same shape is refused the same way (#1122).
+/// A 50 MiB file is the same gate: anything past 8 MiB, including the
+/// single JSON files that killed a Windows node, is refused on its length
+/// before `serde_json`.
#[tokio::test]
async fn oversized_json_document_is_refused_and_its_neighbour_is_indexed() {
use xerj_common::document_limit::{indexed_document_too_large, MAX_INDEXED_DOCUMENT_BYTES};
@@ -823,6 +826,7 @@ async fn oversized_json_document_is_refused_and_its_neighbour_is_indexed() {
.create_index("bulk_oversize", Schema::empty())
.unwrap();
+ assert!(indexed_document_too_large(50 * 1024 * 1024));
let big = "x".repeat(MAX_INDEXED_DOCUMENT_BYTES + 1);
let big_line = format!(r#"{{"blob":"{big}"}}"#);
assert!(indexed_document_too_large(big_line.len()));
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index b457cb12b..ed274f047 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -476,8 +476,9 @@ timeout_ms = 5000
# Process-wide memory ceiling (MiB) that bounds every internal budget — segment
# hydration, query working sets, and the ingest circuit breaker. OMIT it (the
-# default) to AUTO-size the cap from the machine in binary-GiB tiers
-# (8 / 16 / 32 GiB): a laptop stays bounded, a large box climbs. Set an explicit
+# default) to AUTO-size the cap from the machine: half of RAM under 16 GiB,
+# then 8 / 16 / 32 GiB. An 8 GiB host gets 4 GiB, so a merge has room; a
+# 16 GiB host stays at 8 GiB and a large box climbs. Set an explicit
# MiB ceiling to raise headroom on a big host — e.g. when a large `xerj autoindex`
# trips `429 circuit_breaking_exception`. Set 0 to uncap to the whole machine
# (not recommended). Env override: XERJ_MAX_PROCESS_MEMORY_MB=auto|off|<MiB>.