Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(memory): bound merge/FST buffers under the governor and add breaker release hysteresis (#2427) (#2452)

xerj-org045aada989e2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1bc26037f..d04e93123 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -79,6 +79,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- The parent memory breaker no longer flaps, and background merges and flush
+ finalizes are bounded while it is engaged (#2427). A 3,000 MB autoindex run
+ engaged the breaker 20 times in 16 minutes, often releasing 100-500 ms
+ later, and RSS still reached 3,808 MB because merges, FST builds and flush
+ buffers kept allocating after writes were getting 429s. The breaker now
+ releases below 85% of the watermark (`XERJ_MEMORY_BREAKER_RELEASE_PCT`,
+ 50-99) and only after it has been engaged for 2 s
+ (`XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS`). RSS that stays flat inside that
+ band for 30 s still releases, so a live set no drain can shrink does not
+ block writes forever. The breaker never releases while RSS is at or above
+ the watermark: the 1 GB release gate (watermark 2,918 MB) released at
+ 5,121 MB through the old flat-footprint release (#1981), re-engaged 100 ms
+ later and was OOM-killed at 8.4 GB. A footprint that climbs while the
+ breaker is engaged is drained again within 2 s instead of waiting out the
+ drain backoff. While the breaker is engaged, searches run one at a time,
+ and at or above the process memory limit new searches get a 429
+ `circuit_breaking_exception`. Every segment merge reserves its estimated working
+ set (8x its inputs) against the watermark in one process-wide ledger.
+ Background merges wait while the breaker is engaged or when the
+ reservation does not fit, and run on the next merge check. One merge
+ always runs when nothing else is merging and the breaker is not engaged.
+ `_forcemerge` may run alone while engaged. While the breaker is engaged,
+ flush finalizes (segment, FST and doc-values builds) run one at a time.
+ A background merge batch that is already running stops once the breaker is
+ engaged and RSS reaches the process limit, keeps its inputs, and halves the
+ input cap of later batches until batches complete again. On the 1 GB gate
+ corpus one admitted batch (JSON segments that compress 10-20x) grew a
+ 3,072 MB `xerj autoindex` node to 8,122 MB in 18 s while writes were
+ getting 429s; plain `_bulk` into one index peaked at 4,967 MB.
+
- `xerj autoindex` no longer aborts a long run with `Resource temporarily
unavailable (os error 11)` when the machine refuses to start a thread. The
sync replay used to start a new thread for every operation (hundreds of
diff --git a/engine/crates/xerj-engine/src/governor.rs b/engine/crates/xerj-engine/src/governor.rs
index ee07fa2e8..6d9c7c6cd 100644
--- a/engine/crates/xerj-engine/src/governor.rs
+++ b/engine/crates/xerj-engine/src/governor.rs
@@ -46,7 +46,8 @@ static GOVERNOR: OnceLock<Arc<ResourceGovernor>> = OnceLock::new();
/// The engage edge fires immediately. While the breaker stays engaged, the
/// next request waits `interval`, which starts at the 2 s floor and doubles
/// up to [`MEMORY_DRAIN_MAX_INTERVAL`] whenever a request did not lower RSS.
-/// A drop resets the spacing. Disengaging resets everything, so the next
+/// A drop resets the spacing, and a climb past the noise band requests a
+/// drain as soon as the minimum spacing allows (#2427). Disengaging resets everything, so the next
/// real engage is immediate again.
#[derive(Debug)]
pub(crate) struct MemoryDrainSchedule {
@@ -74,13 +75,30 @@ impl MemoryDrainSchedule {
self.rss_at_request = None;
return false;
}
+ // #2427: the latch no longer releases above the watermark, so a
+ // footprint that CLIMBS while engaged must not sit out a backed-off
+ // gap (up to 15 min) — the release gate climbed from 5.1 GB to an
+ // 8.4 GB OOM in 84 s. A climb past the noise band since the last
+ // request is new pressure: drain again at the minimum spacing.
+ let climbed = self
+ .rss_at_request
+ .is_some_and(|prev| rss.saturating_sub(prev) > MEMORY_RSS_NOISE_BYTES);
let due = match self.last_request {
None => true,
- Some(at) => now.saturating_duration_since(at) >= self.interval,
+ Some(at) => {
+ let since = now.saturating_duration_since(at);
+ since >= self.interval || (climbed && since >= self.min)
+ }
};
if !due {
return false;
}
+ if climbed {
+ self.interval = self.min;
+ self.rss_at_request = Some(rss);
+ self.last_request = Some(now);
+ return true;
+ }
if let Some(prev) = self.rss_at_request {
// A one-page dip is compressor noise, not a drain that worked.
// Resetting on it would pin the spacing at 2 s for a footprint
@@ -108,21 +126,6 @@ impl MemoryDrainSchedule {
/// tokio pool is saturated.
pub const SAMPLE_INTERVAL_MS: u64 = 100;
-/// How long RSS may sit flat at or above the watermark before the breaker
-/// releases (#1981).
-///
-/// One memory-pressure drain is 2 s apart from the next. That is enough time
-/// for the drain to reclaim caches and purge arenas. If RSS still has not
-/// fallen, another drain will not either — on macOS jemalloc has no
-/// background thread, so a footprint the purge cannot shrink (compressed
-/// pages in `ri_phys_footprint`, or a live set the caches are not) used to
-/// keep the sampler requesting a drain every 2 s for the rest of the
-/// process. Each request flushes every index and walks the caches. That is
-/// the idle-CPU climb: the node is quiet until RSS crosses the watermark,
-/// then one core stays busy with drains that change nothing, and a restart
-/// (RSS back under the watermark) looks like a cure.
-pub const MEMORY_FLAT_RELEASE: Duration = Duration::from_secs(2);
-
/// Ceiling on the automatic drain spacing once a drain stops lowering RSS.
/// The floor is the sampler's 2 s minimum; doubling from there reaches this
/// in a few minutes instead of staying at 2 s for an 18-hour uptime.
@@ -133,11 +136,94 @@ pub const MEMORY_DRAIN_MAX_INTERVAL: Duration = Duration::from_secs(15 * 60);
///
/// macOS reports `ri_phys_footprint`, which moves by pages while the
/// compressor runs even when the process is idle. An exact-byte clock never
-/// stays flat for [`MEMORY_FLAT_RELEASE`] on that signal, and a one-byte dip
-/// would also reset the drain backoff, so the 2 s flush loop would survive
-/// the latch fix. A climb past this band is a real working-set change.
+/// stays flat on that signal, and a one-byte dip would also reset the drain
+/// backoff, so the 2 s flush loop would survive the latch fix. A move past
+/// this band is a real working-set change.
pub const MEMORY_RSS_NOISE_BYTES: u64 = 4 * 1024 * 1024;
+/// Default release point of the RSS breaker, as a percentage of the
+/// watermark that tripped it (#2427).
+///
+/// The old band was 95% and released on the first sample that did not fall.
+/// A 3,000 MB autoindex run engaged at 2,918 MB and flapped 20 times in 16
+/// minutes, often releasing 100-500 ms after it engaged: each release
+/// re-admitted the write burst that tripped it. Releasing at 85% leaves a
+/// band the drain has to clear before writes come back.
+/// `XERJ_MEMORY_BREAKER_RELEASE_PCT` overrides it (50..=99).
+pub const DEFAULT_MEMORY_RELEASE_PERCENT: u8 = 85;
+
+/// Lowest accepted `XERJ_MEMORY_BREAKER_RELEASE_PCT`. Below this, a footprint
+/// whose live set is over half the watermark could never release.
+pub const MIN_MEMORY_RELEASE_PERCENT: u8 = 50;
+
+/// Highest accepted `XERJ_MEMORY_BREAKER_RELEASE_PCT`.
+pub const MAX_MEMORY_RELEASE_PERCENT: u8 = 99;
+
+/// Default minimum time the RSS breaker stays engaged before a release below
+/// the watermark (#2427). `XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS` overrides it.
+///
+/// This stops a 100 ms engage/release cycle: a release needs the drain to
+/// have run at least once. A live retune (#1323) or Windows commit recovery
+/// are not subject to it. No release ever happens at or above the watermark.
+pub const DEFAULT_MEMORY_MIN_ENGAGED: Duration = Duration::from_secs(2);
+
+/// Ceiling on `XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS`.
+pub const MAX_MEMORY_MIN_ENGAGED: Duration = Duration::from_secs(60);
+
+/// How long RSS may sit without falling past [`MEMORY_RSS_NOISE_BYTES`]
+/// inside the release band `[release, watermark)` before the breaker stops
+/// waiting for the drain and releases (#2427).
+///
+/// Without this the band could hold writes forever on a live set between
+/// 85% and 100% of the watermark that no drain can shrink (the #1250
+/// concern). Thirty seconds is fifteen drain intervals, and bounds a
+/// band-stall re-engage cycle to at most one every 30 s.
+pub const MEMORY_BAND_STALL_RELEASE: Duration = Duration::from_secs(30);
+
+/// Bytes at which an engaged RSS breaker releases: `release_pct` percent of
+/// `watermark`. `0` for a disabled watermark.
+pub fn memory_release_bytes(watermark: u64, release_pct: u8) -> u64 {
+ ((watermark as u128 * u128::from(release_pct.min(100))) / 100) as u64
+}
+
+/// Parse `XERJ_MEMORY_BREAKER_RELEASE_PCT`. Unset, unparsable, or out of
+/// `MIN..=MAX` falls back to [`DEFAULT_MEMORY_RELEASE_PERCENT`].
+pub fn resolve_memory_release_percent(raw: Option<&str>) -> u8 {
+ let Some(raw) = raw.map(str::trim).filter(|s| !s.is_empty()) else {
+ return DEFAULT_MEMORY_RELEASE_PERCENT;
+ };
+ match raw.trim_end_matches('%').parse::<u8>() {
+ Ok(pct) if (MIN_MEMORY_RELEASE_PERCENT..=MAX_MEMORY_RELEASE_PERCENT).contains(&pct) => pct,
+ _ => {
+ tracing::warn!(
+ value = raw,
+ default = DEFAULT_MEMORY_RELEASE_PERCENT,
+ "XERJ_MEMORY_BREAKER_RELEASE_PCT must be {MIN_MEMORY_RELEASE_PERCENT}..={MAX_MEMORY_RELEASE_PERCENT}; using the default"
+ );
+ DEFAULT_MEMORY_RELEASE_PERCENT
+ }
+ }
+}
+
+/// Parse `XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS`. Unset or unparsable falls back
+/// to [`DEFAULT_MEMORY_MIN_ENGAGED`]; larger values are clamped to
+/// [`MAX_MEMORY_MIN_ENGAGED`]. `0` turns the minimum off.
+pub fn resolve_memory_min_engaged(raw: Option<&str>) -> Duration {
+ let Some(raw) = raw.map(str::trim).filter(|s| !s.is_empty()) else {
+ return DEFAULT_MEMORY_MIN_ENGAGED;
+ };
+ match raw.parse::<u64>() {
+ Ok(ms) => Duration::from_millis(ms).min(MAX_MEMORY_MIN_ENGAGED),
+ Err(_) => {
+ tracing::warn!(
+ value = raw,
+ "XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS is not a whole number of milliseconds; using the default"
+ );
+ DEFAULT_MEMORY_MIN_ENGAGED
+ }
+ }
+}
+
/// Bytes of analyze-time reservation charged per raw JSON input byte (#1122).
///
/// `document_limit` measured about 56 bytes of RSS per token for a
@@ -223,17 +309,28 @@ pub struct ResourceGovernor {
/// tripped; when an operator re-tunes the watermark live
/// (`indices.breaker.total.limit`, #1323) the latch is re-evaluated
/// against the new value without the band, so a live raise releases at
- /// once instead of waiting for RSS to fall 5% below the NEW watermark.
+ /// once instead of waiting for RSS to fall below the release point of
+ /// the NEW watermark.
memory_trip_watermark: AtomicU64,
- /// When RSS first stopped falling while the breaker was engaged at or
- /// above the watermark. `None` while RSS is falling or the breaker is
+ /// When RSS last stopped falling inside the release band while the
+ /// breaker was engaged (#2427). `None` outside the band or while
/// disengaged. Aged by tests; the sampler's wall clock fills it in
- /// production (#1981).
+ /// production.
memory_flat_since: Mutex<Option<Instant>>,
- /// RSS at which a flat footprint was released. Later samples at or below
- /// this stay disengaged, so the latch cannot flap back on at the same
- /// reading one sample later. `0` means no floor.
- memory_stall_floor: AtomicU64,
+ /// Release point of the RSS breaker, percent of the tripped watermark
+ /// (#2427). See [`DEFAULT_MEMORY_RELEASE_PERCENT`].
+ memory_release_percent: u8,
+ /// Minimum time engaged before an in-band or below-band release (#2427).
+ memory_min_engaged: Duration,
+ /// When the RSS breaker last engaged. `None` while released.
+ memory_engaged_since: Mutex<Option<Instant>>,
+ /// RSS at which the in-band stall clock started (#2427). A sample more
+ /// than [`MEMORY_RSS_NOISE_BYTES`] under it is a falling drain and
+ /// restarts the clock. `0` = no band clock.
+ memory_band_anchor: AtomicU64,
+ /// Process-wide ledger of in-flight segment merges and their estimated
+ /// working set (#2427). See [`ResourceGovernor::try_reserve_merge`].
+ merge_ledger: Arc<Mutex<MergeLedger>>,
// ── item 1: per-query memory guard (`max_query_memory_mb`) ──────────
/// Maximum bytes a single query may be estimated to allocate. `0` = off.
@@ -249,6 +346,10 @@ pub struct ResourceGovernor {
search_inflight: Arc<AtomicU64>,
/// High-water mark of concurrent searches observed (proof of the cap).
search_inflight_peak: Arc<AtomicU64>,
+ /// #2427: one permit. While the RSS breaker is engaged every search
+ /// also holds this, so searches run one at a time instead of
+ /// `max_concurrent_searches` wide.
+ search_pressure: Arc<Semaphore>,
// ── item 2b: global bulk pool (concurrent bulk parse cap) ─────────
/// Global bulk-concurrency permits. Caps the number of concurrent
@@ -487,8 +588,8 @@ impl ResourceGovernor {
.unwrap_or_else(|e| e.into_inner()) = None;
}
- /// #1981 test hook: age the flat-footprint clock without sleeping, so a
- /// test can cross [`MEMORY_FLAT_RELEASE`] on a paused schedule.
+ /// Test hook: age the flat-footprint clock without sleeping, so a test
+ /// can cross [`MEMORY_BAND_STALL_RELEASE`] on a paused schedule.
#[cfg(test)]
fn age_memory_flat_for_test(&self, by: Duration) {
let mut slot = self
@@ -500,6 +601,139 @@ impl ResourceGovernor {
}
}
+ /// Release point of the RSS breaker as a percentage of the watermark.
+ pub fn memory_release_percent(&self) -> u8 {
+ self.memory_release_percent
+ }
+
+ /// Minimum time the RSS breaker stays engaged before an in-band release.
+ pub fn memory_min_engaged(&self) -> Duration {
+ self.memory_min_engaged
+ }
+
+ /// How long the RSS breaker has been engaged. A latch with no recorded
+ /// edge (the startup commit trip) counts as having met any minimum.
+ fn memory_engaged_for(&self) -> Duration {
+ match *self
+ .memory_engaged_since
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ {
+ Some(at) => at.elapsed(),
+ None => Duration::MAX,
+ }
+ }
+
+ fn set_memory_engaged_since(&self, at: Option<Instant>) {
+ *self
+ .memory_engaged_since
+ .lock()
+ .unwrap_or_else(|e| e.into_inner()) = at;
+ }
+
+ /// #2427 test hook: age the engaged clock without sleeping.
+ #[cfg(test)]
+ fn age_memory_engaged_for_test(&self, by: Duration) {
+ let mut slot = self
+ .memory_engaged_since
+ .lock()
+ .unwrap_or_else(|e| e.into_inner());
+ if let Some(at) = *slot {
+ *slot = Some(at.checked_sub(by).unwrap_or(at));
+ }
+ }
+
+ /// Bytes reserved by in-flight segment merges (#2427).
+ pub fn merge_reserved_bytes(&self) -> u64 {
+ self.merge_ledger
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .reserved
+ }
+
+ /// Segment merges currently holding a [`MergeReservation`] (#2427).
+ pub fn merges_in_flight(&self) -> u64 {
+ self.merge_ledger
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .in_flight
+ }
+
+ /// Reserve the estimated working set of one merge batch whose inputs
+ /// total `input_bytes` against the RSS watermark (#2427).
+ ///
+ /// Before this, every index started its first merge batch whatever the
+ /// process footprint, and each batch was checked against an RSS snapshot
+ /// that did not see other indices' merges that had started but not yet
+ /// allocated. Under autoindex (hundreds of indices) that put many merge
+ /// and FST working sets on top of a node already at the watermark: RSS
+ /// climbed about 1 GB past a 3,000 MB cap while writes were getting 429s.
+ ///
+ /// The reservation is `input_bytes * MERGE_WORKING_SET_NUMERATOR` and is
+ /// held until the guard drops. See `merge_budget::decide_merge_admission`
+ /// for the rule.
+ /// `None` means defer: the caller leaves the batch for a later pass.
+ pub fn try_reserve_merge(&self, input_bytes: u64, forced: bool) -> Option<MergeReservation> {
+ let bytes = crate::merge_budget::merge_working_set_bytes(input_bytes);
+ let watermark = self.memory_watermark_bytes();
+ let observed = self.observed_process_memory();
+ let engaged = self.memory_breaker_engaged();
+ let mut ledger = self.merge_ledger.lock().unwrap_or_else(|e| e.into_inner());
+ if !crate::merge_budget::decide_merge_admission(
+ watermark,
+ observed,
+ ledger.reserved,
+ ledger.in_flight,
+ engaged,
+ bytes,
+ forced,
+ ) {
+ return None;
+ }
+ ledger.reserved = ledger.reserved.saturating_add(bytes);
+ ledger.in_flight += 1;
+ Some(MergeReservation {
+ ledger: Arc::clone(&self.merge_ledger),
+ bytes,
+ })
+ }
+
+ /// Whether an in-flight background merge batch must stop now (#2427).
+ /// See `merge_budget::merge_must_yield`.
+ pub fn merge_must_yield(&self, forced: bool) -> bool {
+ crate::merge_budget::merge_must_yield(
+ self.observed_process_memory(),
+ self.memory_limit_bytes,
+ self.memory_breaker_engaged(),
+ forced,
+ )
+ }
+
+ /// Record a background batch of `input_bytes` that stopped under memory
+ /// pressure: the next batches are planned under half of it (#2427).
+ pub fn note_merge_yield(&self, input_bytes: u64) -> u64 {
+ let mut ledger = self.merge_ledger.lock().unwrap_or_else(|e| e.into_inner());
+ ledger.pressure_cap =
+ crate::merge_budget::pressure_cap_after_yield(ledger.pressure_cap, input_bytes);
+ ledger.pressure_cap
+ }
+
+ /// Record a background batch of `input_bytes` that completed: a batch
+ /// that used the adaptive cap widens it again (#2427).
+ pub fn note_merge_completed(&self, input_bytes: u64) {
+ let mut ledger = self.merge_ledger.lock().unwrap_or_else(|e| e.into_inner());
+ ledger.pressure_cap =
+ crate::merge_budget::pressure_cap_after_success(ledger.pressure_cap, input_bytes);
+ }
+
+ /// The adaptive merge batch input cap, `0` when none is set (#2427).
+ pub fn merge_pressure_cap_bytes(&self) -> u64 {
+ self.merge_ledger
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .pressure_cap
+ }
+
/// Whether the summed-memtable budget breaker is currently engaged.
pub fn memtable_breaker_engaged(&self) -> bool {
self.memtable_tripped.load(Ordering::Relaxed)
@@ -671,19 +905,65 @@ impl ResourceGovernor {
/// the permit (and decrements the in-flight gauge) on drop. Excess
/// searches queue on the semaphore, exactly like ES's search thread
/// pool bounds active workers.
+ ///
+ /// Under memory pressure (#2427) searches are non-write memory the RSS
+ /// breaker never saw: the 1 GB release gate kept climbing from 5.1 GB to
+ /// an 8.4 GB OOM with writes blocked, and its search phase ran 30 s
+ /// multi-index searches against a node already over its watermark.
+ /// While the breaker is engaged each search also takes the single
+ /// `search_pressure` permit, so they run one at a time. At or above the
+ /// hard process limit a new search is rejected with a 429
+ /// `circuit_breaking_exception` instead of queueing.
pub async fn acquire_search(&self) -> Result<SearchPermit, XerjError> {
+ self.check_search_hard_limit()?;
let permit = Arc::clone(&self.search_pool)
.acquire_owned()
.await
.map_err(|_| XerjError::internal("global search pool closed — shutting down"))?;
+ let pressure = if self.memory_breaker_engaged() {
+ let p = Arc::clone(&self.search_pressure)
+ .acquire_owned()
+ .await
+ .map_err(|_| XerjError::internal("search pressure gate closed — shutting down"))?;
+ // RSS may have crossed the hard limit while this search queued.
+ self.check_search_hard_limit()?;
+ Some(p)
+ } else {
+ None
+ };
let now = self.search_inflight.fetch_add(1, Ordering::Relaxed) + 1;
self.search_inflight_peak.fetch_max(now, Ordering::Relaxed);
Ok(SearchPermit {
_permit: permit,
+ _pressure: pressure,
inflight: Arc::clone(&self.search_inflight),
})
}
+ /// RSS line at which new searches are rejected (#2427): the process
+ /// memory limit, or the live watermark if an operator raised it past
+ /// that (#1323). `0` when no limit is known.
+ fn search_hard_limit_bytes(&self) -> u64 {
+ let watermark = self.memory_watermark_bytes.load(Ordering::Relaxed);
+ if watermark == 0 {
+ return 0;
+ }
+ self.memory_limit_bytes.max(watermark)
+ }
+
+ fn check_search_hard_limit(&self) -> Result<(), XerjError> {
+ let limit = self.search_hard_limit_bytes();
+ let rss = self.rss_bytes.load(Ordering::Relaxed);
+ if limit != 0 && rss >= limit {
+ return Err(XerjError::circuit_breaking(format!(
+ "[parent] Data too large: process memory [{rss}/{}] is at or above the memory limit [{limit}/{}]; search rejected until the memory breaker drains",
+ human_bytes(rss),
+ human_bytes(limit)
+ )));
+ }
+ Ok(())
+ }
+
/// Current in-flight search count (for stats / proof).
pub fn search_inflight(&self) -> u64 {
self.search_inflight.load(Ordering::Relaxed)
@@ -748,10 +1028,9 @@ impl ResourceGovernor {
/// [`Self::refresh_memory_disk`] plus an optional Windows available-commit
/// sample. `None` leaves the commit latch as it was (Linux, and tests
/// that only move RSS). A `Some` below the commit reserve engages the
- /// breaker even when `rss` is under the watermark (#1122). That hold is
- /// independent of the flat-footprint release (#1981): a footprint that
- /// does not fall still releases once commit is back above the reserve,
- /// and commit starvation does not clear the flat clock.
+ /// breaker even when `rss` is under the watermark (#1122). Once commit is
+ /// back above the reserve the RSS latch alone decides, and that latch
+ /// never releases at or above the watermark (#2427).
pub fn refresh_memory_observing(
&self,
rss: u64,
@@ -759,43 +1038,32 @@ impl ResourceGovernor {
disk_avail_bytes: u64,
available_commit: Option<u64>,
) {
- // ── RSS watermark, with a bounded release hysteresis (#1250) ──
+ // ── RSS watermark, with release hysteresis (#1250, #2427) ──
//
- // Engage at `rss >= watermark`. Once engaged, hold the latch while RSS
- // sits in the band `[95% of watermark, watermark)` only for as long as
- // RSS is still FALLING — the memory-pressure drain is making progress,
- // and releasing now would re-admit the write burst that re-trips the
- // breaker a sample later (the rss=15,596MB vs watermark=15,564MB
- // chatter in #1250). If RSS stops falling inside the band the drain
- // has nothing more to free: holding would only reject writes the node
- // can take and keep the sampler re-draining every
- // `MEMORY_DRAIN_MIN_INTERVAL` forever, so the latch releases exactly
- // as it did before the band existed.
+ // Engage at `rss >= watermark`. Once engaged, the latch releases below
+ // `memory_release_percent` of the watermark (85% by default) after the
+ // breaker has been engaged for `memory_min_engaged`. Inside the band
+ // `[release, watermark)` it holds while the drain is still lowering
+ // RSS. A 95% band that released on the first sample that did not fall
+ // flapped 20 times in 16 minutes under autoindex (#2427), each release
+ // re-admitting the write burst that tripped it. If RSS does not fall
+ // past [`MEMORY_RSS_NOISE_BYTES`] inside the band for
+ // [`MEMORY_BAND_STALL_RELEASE`], the drain has nothing more to free:
+ // holding would only reject writes the node can take and keep the
+ // sampler re-draining forever, so the latch releases.
//
- // The same "stopped falling" rule applies AT or ABOVE the watermark
- // (#1981), after [`MEMORY_FLAT_RELEASE`], with a
- // [`MEMORY_RSS_NOISE_BYTES`] band so page-sized footprint noise is
- // not a climb or a recovery. A footprint that does not fall past
- // that band is not a drain the sampler should repeat for the life
- // of the process. The release records `memory_stall_floor` so a
- // later sample inside the band does not engage again — without that
- // floor `rss >= watermark` would re-trip one sample later and the
- // drain edge would fire forever. A climb past the band is new
- // pressure and engages. A live watermark change (#1323) drops the
- // band and the floor for that sample: the operator's new value
- // decides.
- let prev_rss = self.rss_bytes.swap(rss, Ordering::Relaxed);
+ // At or above the watermark the latch never releases (#2427). The
+ // #1981 flat-footprint release that used to apply there is gone: it
+ // released a node at 5,121MB against a 2,918MB watermark. A live
+ // watermark change (#1323) is judged against the operator's new
+ // value on that sample.
+ self.rss_bytes.store(rss, Ordering::Relaxed);
let watermark = self.memory_watermark_bytes.load(Ordering::Relaxed);
let cur_mem = self.memory_tripped.load(Ordering::Relaxed);
let retuned = self.memory_trip_watermark.load(Ordering::Relaxed) != watermark;
- let release_bytes = watermark - (watermark / 20);
- let stall_floor = self.memory_stall_floor.load(Ordering::Relaxed);
- // In-band hysteresis (#1250) still uses an exact comparison: one
- // sample that does not fall releases. Above the watermark the same
- // comparison would treat compressor noise as a climb or a recovery.
- let falling = rss < prev_rss;
- let dropped = prev_rss.saturating_sub(rss) > MEMORY_RSS_NOISE_BYTES;
- let climbed = rss.saturating_sub(prev_rss) > MEMORY_RSS_NOISE_BYTES;
+ // #2427: the release point is `memory_release_percent` of the
+ // watermark (85% by default, was a fixed 95%).
+ let release_bytes = memory_release_bytes(watermark, self.memory_release_percent);
// `None` keeps the previous latch so an RSS-only refresh (Linux, and
// the test helper) does not clear a commit trip it did not sample.
// Reserve 0 means this platform has no commit probe: starved stays
@@ -810,61 +1078,74 @@ impl ResourceGovernor {
self.commit_starved.load(Ordering::Relaxed)
};
self.commit_starved.store(starved, Ordering::Relaxed);
- // RSS latch (#1250, #1981). Commit starvation is applied after this
- // so a flat footprint can release, and a pagefileless box still
- // stays engaged while available commit is under the reserve.
+ // RSS latch (#1250, #2427). Commit starvation is applied after this
+ // so a pagefileless box stays engaged while available commit is
+ // under the reserve.
let rss_next = watermark != 0
&& if rss >= watermark {
- let within_floor =
- stall_floor != 0 && rss <= stall_floor.saturating_add(MEMORY_RSS_NOISE_BYTES);
- if !cur_mem && within_floor && !retuned {
- false
- } else if cur_mem && !retuned && !dropped {
- if climbed {
- // Still climbing past noise: the drain has not
- // caught the working set. Forget the flat clock.
+ // #2427: at or above the watermark the RSS latch ALWAYS holds
+ // (or engages). There is no release up here, whatever the
+ // footprint is doing. The #1981 flat-footprint release that
+ // used to live in this arm let a 1 GB release-gate node
+ // release at rss=5,121MB against a 2,918MB watermark, re-engage
+ // 100 ms later at 5,125MB, and get OOM-killed at 8.4 GB. The
+ // drain backoff (`MemoryDrainSchedule`) is what keeps a
+ // footprint that will not fall from re-draining every 2 s.
+ self.memory_band_anchor.store(0, Ordering::Relaxed);
+ self.clear_memory_flat();
+ true
+ } else if cur_mem && !retuned {
+ // #2427 release hysteresis, strictly below the watermark.
+ // Below the release point, release once the minimum engaged
+ // time has passed. Inside the band, hold while the drain is
+ // still lowering RSS; if RSS has not fallen past the noise
+ // band for MEMORY_BAND_STALL_RELEASE, the drain has nothing
+ // left to free, so release rather than reject writes forever
+ // (#1250).
+ let min_met = self.memory_engaged_for() >= self.memory_min_engaged;
+ if rss < release_bytes {
+ self.clear_memory_flat();
+ self.memory_band_anchor.store(0, Ordering::Relaxed);
+ !min_met
+ } else {
+ let anchor = self.memory_band_anchor.load(Ordering::Relaxed);
+ if anchor == 0 || rss.saturating_add(MEMORY_RSS_NOISE_BYTES) < anchor {
+ // Entered the band, or fell past the noise band:
+ // the drain is working. Restart the stall clock.
+ self.memory_band_anchor.store(rss, Ordering::Relaxed);
self.clear_memory_flat();
- self.memory_stall_floor.store(0, Ordering::Relaxed);
+ self.note_memory_flat();
true
} else {
- let flat_for = self.note_memory_flat();
- // Starvation must not record a stall floor: that
- // floor means "released at this footprint", and the
- // breaker has not released. The clock keeps running
- // so the release happens on the first sample after
- // commit recovers, once the window has elapsed.
- if !starved && flat_for >= MEMORY_FLAT_RELEASE {
- self.memory_stall_floor.store(rss, Ordering::Relaxed);
+ let stalled = self.note_memory_flat() >= MEMORY_BAND_STALL_RELEASE;
+ if stalled && min_met {
self.clear_memory_flat();
+ self.memory_band_anchor.store(0, Ordering::Relaxed);
false
} else {
true
}
}
- } else {
- // Dropped past the noise band, a live retune, or the
- // engage edge. Reaching this arm means one of those is
- // true, so the flat clock and the floor are stale.
- self.clear_memory_flat();
- self.memory_stall_floor.store(0, Ordering::Relaxed);
- true
}
- } else if cur_mem && !retuned {
- self.clear_memory_flat();
- rss >= release_bytes && falling
} else {
+ // Disengaged under the watermark, or a live retune (#1323)
+ // that put RSS under the operator's new value.
self.clear_memory_flat();
- if !cur_mem || retuned {
- self.memory_stall_floor.store(0, Ordering::Relaxed);
- }
+ self.memory_band_anchor.store(0, Ordering::Relaxed);
false
};
+ // #2427 invariant: the RSS latch never reports released while RSS is
+ // at or above the live watermark.
+ debug_assert!(rss_next || watermark == 0 || rss < watermark);
let mem_next = starved || rss_next;
if mem_next {
self.memory_trip_watermark
.store(watermark, Ordering::Relaxed);
}
if mem_next != self.memory_tripped.swap(mem_next, Ordering::Relaxed) {
+ // Only an RSS engage starts the minimum-engaged clock: a Windows
+ // commit trip releases as soon as commit recovers.
+ self.set_memory_engaged_since((mem_next && rss_next).then(Instant::now));
if mem_next && starved {
tracing::warn!(
available_commit_mb = self.available_commit_bytes.load(Ordering::Relaxed)
@@ -882,6 +1163,7 @@ impl ResourceGovernor {
} else {
tracing::info!(
rss_mb = rss / (1024 * 1024),
+ release_mb = release_bytes / (1024 * 1024),
"RSS back below the memory watermark — releasing the memory circuit breaker"
);
}
@@ -984,6 +1266,9 @@ impl ResourceGovernor {
/// gauge on drop; the underlying semaphore permit is released with it.
pub struct SearchPermit {
_permit: OwnedSemaphorePermit,
+ /// The memory-pressure gate's single permit, held when the search was
+ /// admitted while the RSS breaker was engaged (#2427).
+ _pressure: Option<OwnedSemaphorePermit>,
inflight: Arc<AtomicU64>,
}
@@ -1483,8 +1768,20 @@ fn build(config: &Config) -> ResourceGovernor {
);
}
+ let memory_release_percent = resolve_memory_release_percent(
+ std::env::var("XERJ_MEMORY_BREAKER_RELEASE_PCT")
+ .ok()
+ .as_deref(),
+ );
+ let memory_min_engaged = resolve_memory_min_engaged(
+ std::env::var("XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS")
+ .ok()
+ .as_deref(),
+ );
tracing::info!(
memtable_budget_mb = memtable_budget_bytes / (1024 * 1024),
+ memory_release_pct = memory_release_percent,
+ memory_min_engaged_ms = memory_min_engaged.as_millis() as u64,
memory_limit_mb = memory_limit_bytes / (1024 * 1024),
memory_limit_source = ?resolved_process_cap.source,
memory_watermark_mb = memory_watermark_bytes / (1024 * 1024),
@@ -1521,12 +1818,17 @@ fn build(config: &Config) -> ResourceGovernor {
0
}),
memory_flat_since: Mutex::new(None),
- memory_stall_floor: AtomicU64::new(0),
+ memory_release_percent,
+ memory_min_engaged,
+ memory_engaged_since: Mutex::new(None),
+ memory_band_anchor: AtomicU64::new(0),
+ merge_ledger: Arc::new(Mutex::new(MergeLedger::default())),
max_query_memory_bytes,
search_pool: Arc::new(Semaphore::new(max_concurrent_searches)),
max_concurrent_searches,
search_inflight: Arc::new(AtomicU64::new(0)),
search_inflight_peak: Arc::new(AtomicU64::new(0)),
+ search_pressure: Arc::new(Semaphore::new(1)),
bulk_pool: Arc::new(Semaphore::new(max_concurrent_bulks)),
max_concurrent_bulks,
disk_flood_pct_default: limits.disk_flood_stage_percent.min(100),
@@ -1938,6 +2240,49 @@ fn parse_byte_size(s: &str) -> Option<u64> {
// Small formatting helpers
// ─────────────────────────────────────────────────────────────────────────
+/// Process-wide merge accounting behind [`MergeReservation`] (#2427).
+#[derive(Debug, Default)]
+struct MergeLedger {
+ reserved: u64,
+ in_flight: u64,
+ /// Adaptive batch input cap learned from batches that had to yield
+ /// under memory pressure. `0` = none. See
+ /// `merge_budget::pressure_cap_after_yield`.
+ pressure_cap: u64,
+}
+
+/// Holds one merge batch's estimated working set against the RSS watermark
+/// (#2427). Returned by [`ResourceGovernor::try_reserve_merge`]; dropping it
+/// returns the bytes and the in-flight slot.
+#[must_use = "dropping the reservation releases the merge budget at once"]
+pub struct MergeReservation {
+ ledger: Arc<Mutex<MergeLedger>>,
+ bytes: u64,
+}
+
+impl MergeReservation {
+ /// Bytes this reservation holds.
+ pub fn bytes(&self) -> u64 {
+ self.bytes
+ }
+}
+
+impl std::fmt::Debug for MergeReservation {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("MergeReservation")
+ .field("bytes", &self.bytes)
+ .finish()
+ }
+}
+
+impl Drop for MergeReservation {
+ fn drop(&mut self) {
+ let mut ledger = self.ledger.lock().unwrap_or_else(|e| e.into_inner());
+ ledger.reserved = ledger.reserved.saturating_sub(self.bytes);
+ ledger.in_flight = ledger.in_flight.saturating_sub(1);
+ }
+}
+
/// Releases an analyze reservation when the batch's tokens exist (#1122).
#[derive(Debug)]
pub(crate) struct AnalyzeReservation {
@@ -2800,6 +3145,8 @@ mod tests {
g.memory_breaker_engaged(),
"RSS at the watermark engages the breaker"
);
+ // #2427: past the minimum engaged time.
+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));
g.refresh(0, watermark / 2, 0);
assert!(
!g.memory_breaker_engaged(),
@@ -2831,6 +3178,8 @@ mod tests {
!g.thread_spawn_safe(),
"breaker engaged: spawning would risk #1122's abort"
);
+ // #2427: past the minimum engaged time.
+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));
g.refresh(0, watermark / 2, 0);
assert!(
g.thread_spawn_safe(),
@@ -2948,14 +3297,13 @@ mod tests {
assert!(g.thread_spawn_safe());
}
- /// #1122 x #1981: commit starvation and a flat footprint are different
- /// holds. A footprint that sits on the watermark must not release the
- /// breaker while available commit is still under the reserve — that
- /// release is what lets the next allocation abort. The flat clock keeps
- /// running, so the first sample after commit recovers releases instead
- /// of starting the 2 s window over, and that footprint does not re-arm.
+ /// #1122 x #2427: commit starvation and RSS above the watermark are
+ /// independent holds, and neither releases while RSS is at or above the
+ /// watermark. Commit coming back does not release a footprint that is
+ /// still over the line, however long it has sat flat; RSS falling under
+ /// the release point (after the minimum engaged time) does.
#[test]
- fn commit_starvation_holds_a_flat_footprint_until_commit_returns() {
+ fn commit_recovery_does_not_release_rss_above_the_watermark() {
const MIB: u64 = 1024 * 1024;
const GIB: u64 = 1024 * MIB;
let mut cfg = Config::default();
@@ -2975,29 +3323,23 @@ mod tests {
g.refresh_memory_observing(stuck, 0, 0, Some(128 * MIB));
assert!(g.memory_breaker_engaged(), "crossing the watermark engages");
g.refresh_memory_observing(stuck, 0, 0, Some(128 * MIB));
- g.age_memory_flat_for_test(MEMORY_FLAT_RELEASE + Duration::from_millis(1));
+ g.age_memory_engaged_for_test(Duration::from_secs(3600));
g.refresh_memory_observing(stuck, 0, 0, Some(128 * MIB));
- assert!(
- g.memory_breaker_engaged(),
- "a flat footprint must not release a breaker commit starvation is holding"
- );
+ assert!(g.memory_breaker_engaged());
let msg = format!("{}", g.check_ingest_admission().unwrap_err());
assert!(msg.contains("pagefile"), "{msg}");
- g.refresh_memory_observing(stuck, 0, 0, Some(2 * GIB));
- assert!(
- !g.memory_breaker_engaged(),
- "once commit is back, a footprint that already sat flat releases"
- );
- g.refresh_memory_observing(stuck, 0, 0, Some(2 * GIB));
+ for _ in 0..50 {
+ g.refresh_memory_observing(stuck, 0, 0, Some(2 * GIB));
+ assert!(
+ g.memory_breaker_engaged(),
+ "commit recovery must not release RSS that is still above the watermark"
+ );
+ }
+ g.refresh_memory_observing(wm / 2, 0, 0, Some(2 * GIB));
assert!(
!g.memory_breaker_engaged(),
- "the same footprint must not re-arm"
- );
- g.refresh_memory_observing(stuck + MEMORY_RSS_NOISE_BYTES + 1, 0, 0, Some(2 * GIB));
- assert!(
- g.memory_breaker_engaged(),
- "a climb past the noise band is new pressure"
+ "RSS under the release point with commit back releases"
);
}
@@ -3287,6 +3629,80 @@ mod tests {
drop(p2);
}
+ /// #2427 (b): while the RSS breaker is engaged, searches run one at a
+ /// time; released, the full pool is available again.
+ #[tokio::test]
+ async fn engaged_memory_breaker_serializes_searches() {
+ let mut cfg = Config::default();
+ cfg.limits.max_concurrent_searches = 8;
+ cfg.limits.max_total_memtable_mb = 0;
+ cfg.limits.memory_watermark_percent = 95;
+ let g = Arc::new(build(&cfg));
+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);
+ assert!(wm > 0, "this test needs a live RSS watermark");
+
+ // Engaged, but under the hard limit: admitted, one at a time.
+ g.refresh(0, wm, 0);
+ assert!(g.memory_breaker_engaged());
+ assert!(
+ wm < g.search_hard_limit_bytes(),
+ "95% watermark sits under the limit"
+ );
+ let p1 = g.acquire_search().await.unwrap();
+ let g2 = Arc::clone(&g);
+ let second = tokio::spawn(async move { g2.acquire_search().await.map(|_p| ()) });
+ assert!(
+ tokio::time::timeout(Duration::from_millis(150), second)
+ .await
+ .is_err(),
+ "a second search must wait while the breaker is engaged"
+ );
+ drop(p1);
+ let p2 = tokio::time::timeout(Duration::from_millis(500), g.acquire_search())
+ .await
+ .expect("the gate frees with the first search")
+ .unwrap();
+ drop(p2);
+
+ // Released: concurrency is back to the pool size.
+ g.age_memory_engaged_for_test(Duration::from_secs(60));
+ g.refresh(0, 0, 0);
+ assert!(!g.memory_breaker_engaged());
+ let a = g.acquire_search().await.unwrap();
+ let b = tokio::time::timeout(Duration::from_millis(500), g.acquire_search())
+ .await
+ .expect("a released breaker does not serialize")
+ .unwrap();
+ assert_eq!(g.search_inflight(), 2);
+ drop((a, b));
+ }
+
+ /// #2427 (b): at or above the hard process limit a new search is a 429,
+ /// not another allocation on a node the kernel is about to kill.
+ #[tokio::test]
+ async fn search_is_rejected_at_the_hard_memory_limit() {
+ let mut cfg = Config::default();
+ cfg.limits.max_total_memtable_mb = 0;
+ cfg.limits.memory_watermark_percent = 95;
+ let g = build(&cfg);
+ let limit = g.search_hard_limit_bytes();
+ assert!(limit > 0, "this test needs a live memory limit");
+
+ g.refresh(0, limit, 0);
+ let err = match g.acquire_search().await {
+ Ok(_) => panic!("a search at the hard limit must be rejected"),
+ Err(e) => e,
+ };
+ assert_eq!(err.http_status(), 429, "{err}");
+ assert_eq!(g.search_inflight(), 0, "a rejected search holds no permit");
+
+ g.refresh(0, limit - 1, 0);
+ assert!(
+ g.acquire_search().await.is_ok(),
+ "just under the hard limit is admitted (serialized)"
+ );
+ }
+
#[test]
fn memory_watermark_releases_with_hysteresis() {
let mut cfg = Config::default();
@@ -3299,7 +3715,7 @@ mod tests {
g.refresh(0, wm, 0);
assert!(g.check_ingest_admission().is_err(), "trips at watermark");
- // 98% of watermark is above the 95% release threshold and RSS is
+ // 98% of watermark is above the 85% release threshold and RSS is
// still falling (the drain is working): stays tripped.
g.refresh(0, wm * 98 / 100, 0);
let err = g.check_ingest_admission().unwrap_err().to_string();
@@ -3324,19 +3740,40 @@ mod tests {
"must remain tripped while RSS drains through the band"
);
- // 94% of watermark is below the 95% release threshold: releases.
+ // #2427: 94% and 86% are inside the 85% release band now, and RSS
+ // is still falling: stays tripped.
g.refresh(0, wm * 94 / 100, 0);
+ assert!(
+ g.check_ingest_admission().is_err(),
+ "94% of watermark is inside the 85% release band"
+ );
+ g.refresh(0, wm * 86 / 100, 0);
+ assert!(
+ g.check_ingest_admission().is_err(),
+ "86% is still in the band"
+ );
+
+ // Below 85%, but the breaker engaged moments ago: the minimum
+ // engaged time holds it so a drain can run.
+ g.refresh(0, wm * 84 / 100, 0);
+ assert!(
+ g.memory_breaker_engaged(),
+ "a release inside the minimum engaged time is the 100 ms flap"
+ );
+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));
+ g.refresh(0, wm * 84 / 100, 0);
assert!(
g.check_ingest_admission().is_ok(),
- "must release once RSS drops below 95% of watermark"
+ "must release once RSS is below 85% of watermark and the minimum has passed"
);
}
- /// #1250 review: a node whose irreducible resident set settles INSIDE the
- /// band after a drain must not stay write-blocked (and re-draining every
- /// `MEMORY_DRAIN_MIN_INTERVAL`) forever. Once RSS stops falling the band
- /// has nothing left to wait for and the latch releases, which is exactly
- /// the pre-band behaviour.
+ /// #1250 review, re-tuned for #2427: a node whose irreducible resident
+ /// set settles INSIDE the band after a drain must not stay write-blocked
+ /// (and re-draining every `MEMORY_DRAIN_MIN_INTERVAL`) forever. One flat
+ /// sample is no longer enough (that released 100-500 ms after engaging
+ /// under autoindex); RSS has to stay flat for
+ /// [`MEMORY_BAND_STALL_RELEASE`].
#[test]
fn memory_hysteresis_releases_when_rss_stalls_inside_the_band() {
let mut cfg = Config::default();
@@ -3352,6 +3789,16 @@ mod tests {
assert!(g.memory_breaker_engaged(), "still draining");
// The drain freed nothing more: RSS is flat inside the band.
g.refresh(0, wm * 97 / 100, 0);
+ assert!(
+ g.memory_breaker_engaged(),
+ "one flat in-band sample must not release (#2427 flap)"
+ );
+ // Page-sized wander is still flat.
+ g.refresh(0, wm * 97 / 100 + 4096, 0);
+ assert!(g.memory_breaker_engaged());
+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));
+ g.age_memory_flat_for_test(MEMORY_BAND_STALL_RELEASE + Duration::from_millis(1));
+ g.refresh(0, wm * 97 / 100, 0);
assert!(
!g.memory_breaker_engaged(),
"a stalled in-band RSS must release, not latch forever"
@@ -3359,87 +3806,284 @@ mod tests {
assert!(g.check_ingest_admission().is_ok());
}
- /// #1981: a footprint that sits flat at or above the watermark is not a
- /// drain the sampler should repeat until restart. After
- /// [`MEMORY_FLAT_RELEASE`] the latch releases, and the same RSS does not
- /// re-engage (that flap would request a drain on every engage edge). A
- /// higher RSS is new pressure and engages again. One flat sample is not
- /// enough — the in-flight drain has not had its window.
+ /// #2427: a drain that keeps lowering RSS through the band restarts the
+ /// stall clock, so a slow but real drain is not cut short.
#[test]
- fn flat_rss_above_the_watermark_releases_and_does_not_rearm() {
+ fn falling_rss_inside_the_band_restarts_the_stall_clock() {
let mut cfg = Config::default();
cfg.limits.max_total_memtable_mb = 0;
cfg.limits.memory_watermark_percent = 95;
let g = build(&cfg);
let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);
- assert!(wm > 0, "this test needs a live RSS watermark");
- let stuck = wm + 100;
+ assert!(
+ wm > 64 * MEMORY_RSS_NOISE_BYTES,
+ "this test needs a live RSS watermark"
+ );
- g.refresh(0, stuck, 0);
- assert!(g.memory_breaker_engaged(), "crossing the watermark engages");
- g.refresh(0, stuck, 0);
+ g.refresh(0, wm, 0);
+ assert!(g.memory_breaker_engaged());
+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));
+ let mut rss = wm * 97 / 100;
+ g.refresh(0, rss, 0);
+ // Stall for most of the window, then fall past the noise band.
+ g.age_memory_flat_for_test(MEMORY_BAND_STALL_RELEASE - Duration::from_millis(100));
+ rss -= 2 * MEMORY_RSS_NOISE_BYTES;
+ g.refresh(0, rss, 0);
assert!(
g.memory_breaker_engaged(),
- "one flat sample must stay engaged so a drain can run"
+ "a falling drain holds the latch"
);
- g.age_memory_flat_for_test(MEMORY_FLAT_RELEASE + Duration::from_millis(1));
- g.refresh(0, stuck, 0);
+ // The clock restarted: the old window has not elapsed for the new
+ // anchor, so another flat sample still holds.
+ g.age_memory_flat_for_test(Duration::from_millis(200));
+ g.refresh(0, rss, 0);
assert!(
- !g.memory_breaker_engaged(),
- "RSS that does not fall must release instead of draining forever"
+ g.memory_breaker_engaged(),
+ "the stall clock must restart when RSS falls past noise"
);
- assert!(g.check_ingest_admission().is_ok());
+ }
- // The old latch re-tripped here: rss >= watermark on a disengaged
- // governor is an engage edge, and the sampler would drain again.
- g.refresh(0, stuck, 0);
- assert!(
- !g.memory_breaker_engaged(),
- "the same footprint must not re-arm the drain"
+ /// #2427: the measured flap. Engage at the watermark, dip just under it,
+ /// climb back: the latch must stay engaged through the whole cycle
+ /// instead of releasing and re-engaging every 100 ms sample.
+ #[test]
+ fn breaker_does_not_flap_around_the_watermark() {
+ let mut cfg = Config::default();
+ cfg.limits.max_total_memtable_mb = 0;
+ cfg.limits.memory_watermark_percent = 95;
+ let g = build(&cfg);
+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);
+ assert!(wm > 0, "this test needs a live RSS watermark");
+
+ let mut transitions = 0;
+ let mut last = g.memory_breaker_engaged();
+ for i in 0..200u64 {
+ // 2,918 MB / 2,850 MB style oscillation: above, then 1-3% under.
+ let rss = if i % 2 == 0 {
+ wm + wm / 50
+ } else {
+ wm - wm / 100 * (1 + i % 3)
+ };
+ g.refresh(0, rss, 0);
+ let now = g.memory_breaker_engaged();
+ if now != last {
+ transitions += 1;
+ last = now;
+ }
+ }
+ assert_eq!(
+ transitions, 1,
+ "only the first engage; no release/re-engage flap"
);
- // Page-sized noise around the floor is the macOS signal, not a new
- // working set. Re-arming on it would engage, drain, and release on
- // every sample.
- g.refresh(0, stuck + 1, 0);
- assert!(
- !g.memory_breaker_engaged(),
- "a one-byte bump around the floor must not re-arm the drain"
+ }
+
+ #[test]
+ fn release_ratio_and_min_engaged_parse_with_safe_defaults() {
+ assert_eq!(
+ resolve_memory_release_percent(None),
+ DEFAULT_MEMORY_RELEASE_PERCENT
+ );
+ assert_eq!(
+ resolve_memory_release_percent(Some("")),
+ DEFAULT_MEMORY_RELEASE_PERCENT
+ );
+ assert_eq!(resolve_memory_release_percent(Some("90")), 90);
+ assert_eq!(resolve_memory_release_percent(Some(" 80% ")), 80);
+ assert_eq!(
+ resolve_memory_release_percent(Some("100")),
+ DEFAULT_MEMORY_RELEASE_PERCENT
+ );
+ assert_eq!(
+ resolve_memory_release_percent(Some("10")),
+ DEFAULT_MEMORY_RELEASE_PERCENT
+ );
+ assert_eq!(
+ resolve_memory_release_percent(Some("x")),
+ DEFAULT_MEMORY_RELEASE_PERCENT
);
- g.refresh(0, stuck + MEMORY_RSS_NOISE_BYTES + 1, 0);
- assert!(
- g.memory_breaker_engaged(),
- "a climb past the noise band is new pressure"
+ assert_eq!(resolve_memory_min_engaged(None), DEFAULT_MEMORY_MIN_ENGAGED);
+ assert_eq!(resolve_memory_min_engaged(Some("0")), Duration::ZERO);
+ assert_eq!(
+ resolve_memory_min_engaged(Some("5000")),
+ Duration::from_secs(5)
+ );
+ assert_eq!(
+ resolve_memory_min_engaged(Some("999999999")),
+ MAX_MEMORY_MIN_ENGAGED
+ );
+ assert_eq!(
+ resolve_memory_min_engaged(Some("2s")),
+ DEFAULT_MEMORY_MIN_ENGAGED
+ );
+
+ assert_eq!(memory_release_bytes(2_850, 85), 2_422);
+ assert_eq!(memory_release_bytes(0, 85), 0);
+ assert_eq!(
+ memory_release_bytes(u64::MAX, 85),
+ (u64::MAX as u128 * 85 / 100) as u64
);
}
- /// #1981: `ri_phys_footprint` does not sit on one byte. A quiet node
- /// whose samples wander inside [`MEMORY_RSS_NOISE_BYTES`] must still
- /// release once that wander has lasted [`MEMORY_FLAT_RELEASE`].
+ /// #2427: merges reserve their working set process-wide, and background
+ /// merges wait while the breaker is engaged.
#[test]
- fn rss_noise_around_a_flat_footprint_still_releases() {
+ fn merge_reservations_are_process_wide_and_respect_the_breaker() {
let mut cfg = Config::default();
cfg.limits.max_total_memtable_mb = 0;
cfg.limits.memory_watermark_percent = 95;
let g = build(&cfg);
let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);
- assert!(wm > MEMORY_RSS_NOISE_BYTES);
- let stuck = wm + MEMORY_RSS_NOISE_BYTES;
+ assert!(wm > 0, "this test needs a live RSS watermark");
- g.refresh(0, stuck, 0);
+ // RSS at half the watermark. The first merge always starts.
+ g.refresh(0, wm / 2 + 1, 0);
+ // Each estimate is a little over a quarter of the watermark.
+ let per = wm / 4 / crate::merge_budget::MERGE_WORKING_SET_NUMERATOR + 1;
+ let first = g.try_reserve_merge(per, false).expect("first merge starts");
+ assert_eq!(g.merges_in_flight(), 1);
+ assert_eq!(g.merge_reserved_bytes(), first.bytes());
+ // 50% + 25% + 25% reaches the watermark: the second must wait.
+ assert!(g.try_reserve_merge(per, false).is_none());
+ // A smaller one still fits.
+ let second = g.try_reserve_merge(per / 4, false).expect("fits");
+ assert_eq!(g.merges_in_flight(), 2);
+ drop(first);
+ drop(second);
+ assert_eq!(g.merges_in_flight(), 0);
+ assert_eq!(g.merge_reserved_bytes(), 0);
+
+ // Breaker engaged: no background merge, a forced one may run alone.
+ g.refresh(0, wm, 0);
assert!(g.memory_breaker_engaged());
- g.refresh(0, stuck + 4096, 0);
- assert!(g.memory_breaker_engaged(), "noise must not release early");
- g.refresh(0, stuck - 4096, 0);
+ assert!(g.try_reserve_merge(1, false).is_none());
+ let forced = g
+ .try_reserve_merge(1, true)
+ .expect("forcemerge progresses alone");
+ assert!(g.try_reserve_merge(1, true).is_none());
+ drop(forced);
+ }
+
+ /// #2427: on the 1 GB gate an admitted merge batch grew a 3,072 MB node
+ /// to 8,122 MB while the breaker was engaged. A background batch yields
+ /// once RSS reaches the process limit with the breaker engaged, never in
+ /// the watermark band, never when forced; the yield narrows later batches
+ /// and completed batches widen them back.
+ #[test]
+ fn a_background_merge_yields_at_the_limit_and_narrows_later_batches() {
+ const MIB: u64 = 1024 * 1024;
+ let mut cfg = Config::default();
+ cfg.limits.max_total_memtable_mb = 0;
+ cfg.limits.memory_watermark_percent = 95;
+ let g = build(&cfg);
+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);
+ let limit = g.process_memory_limit_bytes();
+ assert!(wm > 0 && limit > wm, "this test needs a live RSS watermark");
+
+ // Engaged inside the watermark band: keep merging.
+ g.refresh(0, wm, 0);
assert!(g.memory_breaker_engaged());
- g.age_memory_flat_for_test(MEMORY_FLAT_RELEASE + Duration::from_millis(1));
- g.refresh(0, stuck + 8192, 0);
+ assert!(!g.merge_must_yield(false));
+ // Engaged at the limit: a background batch yields, a forced one not.
+ g.refresh(0, limit, 0);
+ assert!(g.memory_breaker_engaged());
+ assert!(g.merge_must_yield(false));
+ assert!(!g.merge_must_yield(true));
+
+ assert_eq!(g.merge_pressure_cap_bytes(), 0);
+ assert_eq!(g.note_merge_yield(66 * MIB), 33 * MIB);
+ assert_eq!(g.merge_pressure_cap_bytes(), 33 * MIB);
+ g.note_merge_completed(MIB);
+ assert_eq!(g.merge_pressure_cap_bytes(), 33 * MIB);
+ g.note_merge_completed(33 * MIB);
+ assert_eq!(g.merge_pressure_cap_bytes(), 66 * MIB);
+ }
+
+ /// #2427: the 1 GB release gate released at rss=5,121MB against a
+ /// 2,918MB watermark (the old #1981 flat-footprint release), re-engaged
+ /// 100 ms later at 5,125MB, and was OOM-killed at 8.4 GB. A footprint at
+ /// or above the watermark must hold the latch however long it sits flat,
+ /// whatever page-sized noise it shows, and however long the breaker has
+ /// been engaged.
+ #[test]
+ fn flat_rss_above_the_watermark_never_releases() {
+ let mut cfg = Config::default();
+ cfg.limits.max_total_memtable_mb = 0;
+ cfg.limits.memory_watermark_percent = 95;
+ let g = build(&cfg);
+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);
+ assert!(
+ wm > MEMORY_RSS_NOISE_BYTES,
+ "this test needs a live RSS watermark"
+ );
+ let stuck = wm + wm / 4;
+
+ g.refresh(0, stuck, 0);
+ assert!(g.memory_breaker_engaged(), "crossing the watermark engages");
+ for i in 0..200u64 {
+ g.age_memory_flat_for_test(Duration::from_secs(3600));
+ g.age_memory_engaged_for_test(Duration::from_secs(3600));
+ let rss = match i % 4 {
+ 0 => stuck,
+ 1 => stuck + 4096,
+ 2 => stuck - 8192,
+ _ => wm,
+ };
+ g.refresh(0, rss, 0);
+ assert!(
+ g.memory_breaker_engaged(),
+ "rss={rss} >= watermark={wm} must never release (sample {i})"
+ );
+ assert!(g.check_ingest_admission().is_err());
+ }
+ g.refresh(0, wm - 1, 0);
+ assert!(
+ g.memory_breaker_engaged(),
+ "one byte under the watermark is inside the release band"
+ );
+ g.refresh(0, wm / 2, 0);
assert!(
!g.memory_breaker_engaged(),
- "page-sized wander is a flat footprint"
+ "under the release point releases"
);
- g.refresh(0, stuck - 8192, 0);
- assert!(!g.memory_breaker_engaged(), "the same band must not re-arm");
+ }
+
+ /// #2427: the latch's release decision is a pure function of the
+ /// sample, so walk a long random-ish RSS trace (the gate's 2,750-5,125MB
+ /// shape scaled to the watermark) and check every release lands strictly
+ /// under the watermark and every engage edge at or above it.
+ #[test]
+ fn no_release_edge_at_or_above_the_watermark_on_a_noisy_trace() {
+ let mut cfg = Config::default();
+ cfg.limits.max_total_memtable_mb = 0;
+ cfg.limits.memory_watermark_percent = 95;
+ let g = build(&cfg);
+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);
+ assert!(wm > 0, "this test needs a live RSS watermark");
+ let mut x: u64 = 0x2427;
+ let mut was = false;
+ let mut releases = 0;
+ for _ in 0..5_000 {
+ // xorshift: deterministic, no rand dependency.
+ x ^= x << 13;
+ x ^= x >> 7;
+ x ^= x << 17;
+ // 80%..180% of the watermark.
+ let rss = wm / 100 * (80 + x % 101);
+ g.age_memory_flat_for_test(Duration::from_secs(60));
+ g.age_memory_engaged_for_test(Duration::from_secs(60));
+ g.refresh(0, rss, 0);
+ let now = g.memory_breaker_engaged();
+ if was && !now {
+ releases += 1;
+ assert!(rss < wm, "released at rss={rss} >= watermark={wm}");
+ }
+ if !was && now {
+ assert!(rss >= wm, "engaged at rss={rss} < watermark={wm}");
+ }
+ was = now;
+ }
+ assert!(releases > 0, "the trace should exercise releases");
}
/// #1981: while the breaker stays engaged and RSS does not fall, drain
@@ -3498,9 +4142,42 @@ mod tests {
assert!(schedule.observe(now + min + min, dropped, true));
}
+ /// #2427: the latch holds above the watermark now, so a climbing
+ /// footprint must not wait out a backed-off gap. Back the schedule off
+ /// on a flat footprint, then climb: the next sample past the minimum
+ /// spacing drains, and a further flat sample is back on the floor.
+ #[test]
+ fn a_climb_while_engaged_drains_inside_a_backed_off_gap() {
+ let min = Duration::from_secs(2);
+ let mut schedule = MemoryDrainSchedule::new(min);
+ let t0 = Instant::now();
+ let stuck = 64 * MEMORY_RSS_NOISE_BYTES;
+ let mut t = Duration::ZERO;
+ while t < Duration::from_secs(600) {
+ schedule.observe(t0 + t, stuck, true);
+ t += min;
+ }
+ assert!(schedule.interval > min, "a flat footprint backs off");
+ let last = schedule.last_request.expect("requested at least once");
+ let climbed = stuck + MEMORY_RSS_NOISE_BYTES + 1;
+ assert!(
+ !schedule.observe(last + min / 2, climbed, true),
+ "the minimum spacing still applies to a climb"
+ );
+ assert!(
+ schedule.observe(last + min, climbed, true),
+ "a climb drains at the minimum spacing, not after the backed-off gap"
+ );
+ assert_eq!(schedule.interval, min);
+ assert!(
+ !schedule.observe(last + min + Duration::from_millis(100), climbed, true),
+ "one climb is one request"
+ );
+ }
+
/// #1250 x #1323: the watermark is live-tunable. Raising it while the
/// breaker is engaged must release at once whenever RSS is under the NEW
- /// watermark — not wait for RSS to fall 5% below it — or the live fix
+ /// watermark — not wait for RSS to fall below the release point — or the live fix
/// #1323 shipped would be undone by the band.
#[test]
fn live_watermark_raise_releases_without_the_hysteresis_band() {
@@ -3516,7 +4193,7 @@ mod tests {
assert!(g.check_ingest_admission().is_err());
// Raise just above the current RSS: RSS is now inside the new
- // watermark's 5% band, and still (barely) falling.
+ // watermark's release band, and still (barely) falling.
let raised = (rss + rss / 50).min(g.machine_memory_bytes);
assert!(raised > rss, "test box must have headroom over 1 GiB");
g.set_memory_watermark_override(Some(raised));
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 40390de23..cd6642ffc 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -13798,7 +13798,21 @@ impl Index {
/// 16 GiB host does not merge in smaller pieces. A 4 GiB budget
/// (the AUTO cap on an 8 GiB host) is 64 MiB.
/// `XERJ_MERGE_BATCH_MAX_INPUT_MB` overrides (0 disables the bound).
+ ///
+ /// #2427: a batch that had to yield under memory pressure narrows this
+ /// for every index through the governor's adaptive cap
+ /// (`merge_budget::effective_batch_cap_bytes`).
fn merge_batch_cap_bytes() -> u64 {
+ let pressure_cap = crate::governor::global()
+ .map(|governor| governor.merge_pressure_cap_bytes())
+ .unwrap_or(0);
+ crate::merge_budget::effective_batch_cap_bytes(
+ Self::merge_static_batch_cap_bytes(),
+ pressure_cap,
+ )
+ }
+
+ fn merge_static_batch_cap_bytes() -> u64 {
const MB: u64 = 1024 * 1024;
if let Ok(raw) = std::env::var("XERJ_MERGE_BATCH_MAX_INPUT_MB") {
if let Ok(mb) = raw.trim().parse::<u64>() {
@@ -14087,9 +14101,31 @@ impl Index {
// pass returns an ERROR so callers (forcemerge HTTP → 500, background
// task → error log) hear about it loudly instead of losing data.
let failed_batches = Arc::new(std::sync::atomic::AtomicUsize::new(0));
+ // #2427 — background batches that stopped because the breaker was
+ // engaged with RSS at the process limit. Not a failure: inputs stay,
+ // the pass is flagged deferred, and the governor narrows the next
+ // batches (`merge_budget::merge_must_yield`).
+ let yielded_batches = Arc::new(std::sync::atomic::AtomicUsize::new(0));
+
+ // #2427 — process-wide admission. Every launch reserves the batch's
+ // estimated working set against the RSS watermark through the
+ // governor's merge ledger, so concurrent merges across indices see
+ // each other before they allocate, and background merges wait while
+ // the breaker is engaged. `None` = defer; `Some(None)` = no governor.
+ let forced_merge = force_max_segments.is_some();
+ let admit_merge = |metas: &[xerj_storage::segment::SegmentMeta]|
+ -> Option<Option<crate::governor::MergeReservation>> {
+ let Some(governor) = crate::governor::global() else {
+ return Some(None);
+ };
+ let input: u64 = metas.iter().map(|meta| meta.size_bytes).sum();
+ governor.try_reserve_merge(input, forced_merge).map(Some)
+ };
+ let mut governor_deferred = 0usize;
let spawn_one = |batch: Vec<xerj_storage::segment::SegmentId>,
- metas: Vec<xerj_storage::segment::SegmentMeta>|
+ metas: Vec<xerj_storage::segment::SegmentMeta>,
+ reservation: Option<crate::governor::MergeReservation>|
-> JoinHandle<Option<MergeOutput>> {
let store_for_task = Arc::clone(&self.store);
let registry_for_task = Arc::clone(&self.registry);
@@ -14121,6 +14157,30 @@ impl Index {
let batch_for_task = batch;
let metas_for_task = metas;
let failed_for_task = Arc::clone(&failed_batches);
+ let yielded_for_task = Arc::clone(&yielded_batches);
+ let forced_for_task = force_max_segments.is_some();
+ let input_bytes_for_task: u64 = metas_for_task.iter().map(|meta| meta.size_bytes).sum();
+ // Checked at each input segment, every 8192 surviving docs, and
+ // before the output segment is created — never after, so a
+ // yielded batch leaves no partial output behind.
+ let must_yield = move || -> bool {
+ let Some(governor) = crate::governor::global() else {
+ return false;
+ };
+ if !governor.merge_must_yield(forced_for_task) {
+ return false;
+ }
+ let cap = governor.note_merge_yield(input_bytes_for_task);
+ tracing::warn!(
+ input_mb = input_bytes_for_task / (1024 * 1024),
+ rss_mb = governor.observed_rss_bytes() / (1024 * 1024),
+ limit_mb = governor.process_memory_limit_bytes() / (1024 * 1024),
+ next_batch_cap_mb = cap / (1024 * 1024),
+ "merge batch yielded: breaker engaged with RSS at the process limit; \
+ inputs kept, later batches narrowed"
+ );
+ true
+ };
#[cfg(test)]
let hold_merge_encode = Arc::clone(&self.test_hold_merge_encode);
#[cfg(test)]
@@ -14338,6 +14398,10 @@ impl Index {
let mut tomb_union: HashMap<String, u64> = HashMap::new();
for (source_index, meta) in metas_for_task.iter().enumerate() {
+ if must_yield() {
+ yielded_for_task.fetch_add(1, Ordering::Relaxed);
+ return None;
+ }
min_seq = min_seq.min(meta.min_seq_no);
max_seq = max_seq.max(meta.max_seq_no);
let seg_path = segments_dir_for_task.join(&meta.seg_path);
@@ -14664,6 +14728,10 @@ impl Index {
// stored-side peak.
drain_checkpoint += 1;
if drain_checkpoint.is_multiple_of(8192) {
+ if must_yield() {
+ yielded_for_task.fetch_add(1, Ordering::Relaxed);
+ return None;
+ }
crate::ingest_memory::observe_checkpoint(
crate::ingest_memory::Category::MergeJsonBuffer,
merged_json_buf.len(),
@@ -14699,6 +14767,10 @@ impl Index {
fts_input_bytes,
);
+ if must_yield() {
+ yielded_for_task.fetch_add(1, Ordering::Relaxed);
+ return None;
+ }
// Write merged stored section using the columnar v2 codec.
let mut writer = match SegmentWriter::new(&segments_dir_for_task, 1, 0, 0) {
Ok(w) => w,
@@ -14998,8 +15070,18 @@ impl Index {
let _ = encode_tx.send(encoded);
});
tokio::spawn(async move {
+ // Held until the encode (the allocation-heavy part: stored
+ // decode, FST and doc-values build) has finished.
+ let _reservation = reservation;
match encode_rx.await {
- Ok(Ok(value)) => value,
+ Ok(Ok(value)) => {
+ if value.is_some() && !forced_for_task {
+ if let Some(governor) = crate::governor::global() {
+ governor.note_merge_completed(input_bytes_for_task);
+ }
+ }
+ value
+ }
Ok(Err(payload)) => std::panic::resume_unwind(payload),
// The encode task vanished without a result. Panic so the
// driver counts a failed batch (the `JoinError` arm)
@@ -15010,15 +15092,33 @@ impl Index {
})
};
+ // Launch the pending batch if the governor admits it (#2427). A
+ // refusal leaves it and every later batch of this pass unmerged and
+ // flags the pass deferred, so the scheduler retries after the drain.
+ macro_rules! launch_next {
+ () => {
+ match pending.take() {
+ Some((batch, metas)) => match admit_merge(&metas) {
+ Some(reservation) => {
+ in_flight.push(spawn_one(batch, metas, reservation));
+ pending = queue_iter.next();
+ true
+ }
+ None => {
+ governor_deferred += 1 + queue_iter.by_ref().count();
+ false
+ }
+ },
+ None => false,
+ }
+ };
+ }
+
// Fill in_flight up to the cap, then drain one-by-one as they
// complete, launching replacements from `queue_iter`.
while in_flight.len() < merge_parallelism {
- match pending.take() {
- Some((batch, metas)) => {
- in_flight.push(spawn_one(batch, metas));
- pending = queue_iter.next();
- }
- None => break,
+ if !launch_next!() {
+ break;
}
}
while !in_flight.is_empty() {
@@ -15225,10 +15325,7 @@ impl Index {
// `continue` bypasses the common top-up below. Launch
// the next disjoint batch first so one failed apply
// does not silently suppress the rest of this pass.
- if let Some((batch, metas)) = pending.take() {
- in_flight.push(spawn_one(batch, metas));
- pending = queue_iter.next();
- }
+ launch_next!();
continue;
}
// `apply_merge` has made the output authoritative. No
@@ -15290,10 +15387,7 @@ impl Index {
// Same pipeline rule as the failed-apply path: the
// `continue` below bypasses the loop's bottom
// top-up, so launch the next disjoint batch here.
- if let Some((batch, metas)) = pending.take() {
- in_flight.push(spawn_one(batch, metas));
- pending = queue_iter.next();
- }
+ launch_next!();
continue;
}
// Disk-space fix (2026-07): the input segments are
@@ -15359,9 +15453,32 @@ impl Index {
}
}
// Top up in_flight.
- if let Some((batch, metas)) = pending.take() {
- in_flight.push(spawn_one(batch, metas));
- pending = queue_iter.next();
+ launch_next!();
+ }
+ let yielded = yielded_batches.load(AtomicOrdering::Relaxed);
+ if yielded > 0 {
+ self.merge_batches_deferred
+ .store(true, AtomicOrdering::Relaxed);
+ tracing::info!(
+ index = %self.name,
+ yielded,
+ "merge batches yielded under memory pressure; inputs kept"
+ );
+ }
+ if governor_deferred > 0 {
+ self.merge_batches_deferred
+ .store(true, AtomicOrdering::Relaxed);
+ if let Some(governor) = crate::governor::global() {
+ tracing::info!(
+ index = %self.name,
+ deferred = governor_deferred,
+ rss_mb = governor.observed_rss_bytes() / (1024 * 1024),
+ watermark_mb = governor.memory_watermark_bytes() / (1024 * 1024),
+ merge_reserved_mb = governor.merge_reserved_bytes() / (1024 * 1024),
+ merges_in_flight = governor.merges_in_flight(),
+ breaker_engaged = governor.memory_breaker_engaged(),
+ "merge batches deferred by the memory governor; inputs kept"
+ );
}
}
@@ -35918,6 +36035,30 @@ async fn do_flush_shard(
}
}
};
+ // #2427 — while the parent RSS breaker is engaged, finalizes run one at
+ // a time (see `flush_pressure_gate`). Same deadline and same
+ // before-the-freeze failure as the finalize gate above.
+ let pressure_permit = if !bypass_finalize_gate
+ && crate::governor::global().is_some_and(|g| g.memory_breaker_engaged())
+ {
+ match wait_for_flush_permit(
+ &index_name,
+ "the memory-pressure flush gate",
+ admit_timeout,
+ crate::flush_pressure_gate().acquire(),
+ )
+ .await
+ {
+ Ok(permit) => permit,
+ Err(error) => {
+ drop(fin_permit);
+ on_drained();
+ return Err(error);
+ }
+ }
+ } else {
+ None
+ };
// #1015 — NO collection-publication guard here. The drain below is a
// FREEZE, not a removal: the claimed docs stay reader-visible in the
// shard's frozen generations for the whole segment build, so there is
@@ -36312,6 +36453,7 @@ async fn do_flush_shard(
}
let joined = tokio::task::spawn_blocking(move || {
let _fin_permit = fin_permit;
+ let _pressure_permit = pressure_permit;
// This lease lives in the blocking worker, not either async waiter.
// It spans build→publish exactly as before: it tracks the caches
// `build_fts` warms so a failed publication can evict them.
diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rs
index 9159caf93..3b93c1a44 100644
--- a/engine/crates/xerj-engine/src/lib.rs
+++ b/engine/crates/xerj-engine/src/lib.rs
@@ -267,6 +267,22 @@ pub(crate) fn flush_finalize_gate() -> &'static tokio::sync::Semaphore {
GATE.get_or_init(|| tokio::sync::Semaphore::new(crate::pools::sizing().flush_finalize))
}
+/// One permit: while the parent RSS breaker is engaged, flush finalizes
+/// (segment build, FST build, doc-values encode) also take this, so only one
+/// runs at a time (#2427).
+///
+/// [`flush_finalize_gate`] allows `max(2, ncores/8)` concurrent finalizes.
+/// Each holds a frozen memtable plus its FST and column build buffers. Under
+/// the breaker, those buffers stacked on a node that was already at the
+/// watermark: RSS climbed about 1 GB past a 3,000 MB cap while writes were
+/// already getting 429s. Serializing them while the breaker is engaged bounds
+/// the overshoot to one build. The drain still makes progress; it just does
+/// not multiply the peak.
+pub(crate) fn flush_pressure_gate() -> &'static tokio::sync::Semaphore {
+ static GATE: std::sync::OnceLock<tokio::sync::Semaphore> = std::sync::OnceLock::new();
+ GATE.get_or_init(|| tokio::sync::Semaphore::new(1))
+}
+
// ── Analyzer helpers ──────────────────────────────────────────────────────────
use std::sync::Arc;
diff --git a/engine/crates/xerj-engine/src/merge_budget.rs b/engine/crates/xerj-engine/src/merge_budget.rs
index 7a2d73c2f..81b34b0b3 100644
--- a/engine/crates/xerj-engine/src/merge_budget.rs
+++ b/engine/crates/xerj-engine/src/merge_budget.rs
@@ -125,10 +125,171 @@ pub fn throttle_merge_batches(
start
}
+/// Estimated peak working set of a merge batch whose inputs total
+/// `input_bytes`: the same `input * MERGE_WORKING_SET_NUMERATOR` that
+/// [`decide_merge_batch`] uses.
+pub fn merge_working_set_bytes(input_bytes: u64) -> u64 {
+ input_bytes.saturating_mul(MERGE_WORKING_SET_NUMERATOR)
+}
+
+/// Process-wide merge admission against the RSS watermark (#2427).
+///
+/// [`throttle_merge_batches`] plans one index's pass from an RSS snapshot
+/// and always lets that index's first batch start. With hundreds of indices
+/// (autoindex) every index started one, and none of them saw the others'
+/// working sets until they had allocated. This is checked under the
+/// governor's merge ledger at launch time instead:
+///
+/// * `watermark == 0` (breaker off) or `observed_rss == 0` (no probe):
+/// start.
+/// * No merge in flight anywhere in the process: start one, so merging can
+/// always make progress, unless the breaker is engaged and the merge is a
+/// background one. A `forced` merge (`_forcemerge`) is still allowed as
+/// the single in-flight merge while engaged; it is a deliberate operator
+/// request and it waits on its own.
+/// * Breaker engaged: defer. Background merges are the one memory consumer
+/// the node can postpone, so they wait for the drain.
+/// * Otherwise start only if `observed_rss + reserved + request` stays under
+/// the watermark.
+pub fn decide_merge_admission(
+ watermark: u64,
+ observed_rss: u64,
+ reserved: u64,
+ in_flight: u64,
+ breaker_engaged: bool,
+ request: u64,
+ forced: bool,
+) -> bool {
+ if watermark == 0 || observed_rss == 0 {
+ return true;
+ }
+ if in_flight == 0 {
+ return !breaker_engaged || forced;
+ }
+ if breaker_engaged {
+ return false;
+ }
+ observed_rss
+ .saturating_add(reserved)
+ .saturating_add(request)
+ < watermark
+}
+
+/// Whether an in-flight background merge batch must stop (#2427).
+///
+/// Admission checks a batch against `input * MERGE_WORKING_SET_NUMERATOR`
+/// before it starts. On the 1 GB gate corpus that estimate was several
+/// times too small: JSON-heavy segments compress 10–20×, and a merge holds
+/// the decoded survivors plus a parsed `Value` per document. One admitted
+/// batch took a 3,072 MB node to 8,122 MB in 18 s while the breaker was
+/// engaged and writes were already getting 429s. Nothing else could free
+/// that memory, because the merge was the thing allocating it.
+///
+/// A batch stops when the breaker is engaged and RSS has reached the full
+/// process limit (past the 95% watermark). A stopped batch commits nothing
+/// and keeps its inputs, like any aborted merge. `forced` (`_forcemerge`)
+/// batches are not stopped. `limit == 0` or `rss == 0` (unknown) never
+/// stops a batch.
+pub fn merge_must_yield(rss: u64, limit: u64, breaker_engaged: bool, forced: bool) -> bool {
+ !forced && breaker_engaged && limit > 0 && rss > 0 && rss >= limit
+}
+
+/// The adaptive batch input cap after a batch of `input_bytes` stopped
+/// under memory pressure (#2427). `current == 0` means no adaptive cap yet.
+/// The next batch may use at most half of what did not fit. Never below 1.
+pub fn pressure_cap_after_yield(current: u64, input_bytes: u64) -> u64 {
+ let halved = (input_bytes / 2).max(1);
+ if current == 0 {
+ halved
+ } else {
+ current.min(halved)
+ }
+}
+
+/// The adaptive batch input cap after a background batch of `input_bytes`
+/// completed (#2427). A batch that used at least half of the cap doubles
+/// it. Once the cap reaches [`HISTORICAL_BATCH_CAP_BYTES`] it is removed
+/// (`0`), and the static cap applies again.
+pub fn pressure_cap_after_success(current: u64, input_bytes: u64) -> u64 {
+ if current == 0 {
+ return 0;
+ }
+ if input_bytes.saturating_mul(2) < current {
+ return current;
+ }
+ let doubled = current.saturating_mul(2);
+ if doubled >= HISTORICAL_BATCH_CAP_BYTES {
+ 0
+ } else {
+ doubled
+ }
+}
+
+/// The batch input cap to plan with: the static cap, narrowed by the
+/// adaptive pressure cap when one is set. A static cap of `0` (the
+/// `XERJ_MERGE_BATCH_MAX_INPUT_MB=0` opt-out) still takes a pressure cap.
+pub fn effective_batch_cap_bytes(static_cap: u64, pressure_cap: u64) -> u64 {
+ match (static_cap, pressure_cap) {
+ (s, 0) => s,
+ (0, p) => p,
+ (s, p) => s.min(p),
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
+ #[test]
+ fn a_background_merge_yields_only_when_engaged_at_the_limit() {
+ const MIB: u64 = 1024 * 1024;
+ let limit = 1024 * MIB;
+ assert!(merge_must_yield(1024 * MIB, limit, true, false));
+ assert!(merge_must_yield(1600 * MIB, limit, true, false));
+ // Watermark band (95%..100%) and below: keep going.
+ assert!(!merge_must_yield(1000 * MIB, limit, true, false));
+ // Not engaged: keep going (the sampler will engage first).
+ assert!(!merge_must_yield(1600 * MIB, limit, false, false));
+ // `_forcemerge` is never stopped.
+ assert!(!merge_must_yield(1600 * MIB, limit, true, true));
+ // Unknown limit or RSS.
+ assert!(!merge_must_yield(1600 * MIB, 0, true, false));
+ assert!(!merge_must_yield(0, limit, true, false));
+ }
+
+ #[test]
+ fn the_pressure_cap_halves_on_yield_and_recovers_on_success() {
+ const MIB: u64 = 1024 * 1024;
+ let cap = pressure_cap_after_yield(0, 66 * MIB);
+ assert_eq!(cap, 33 * MIB);
+ // A second, larger yield never widens it.
+ assert_eq!(pressure_cap_after_yield(cap, 200 * MIB), 33 * MIB);
+ let cap = pressure_cap_after_yield(cap, 30 * MIB);
+ assert_eq!(cap, 15 * MIB);
+ // A tiny batch's success does not widen it.
+ assert_eq!(pressure_cap_after_success(cap, MIB), cap);
+ // Batches that use the cap double it until it is removed.
+ let mut cap = cap;
+ let mut rounds = 0;
+ while cap != 0 {
+ cap = pressure_cap_after_success(cap, cap);
+ rounds += 1;
+ assert!(rounds < 16);
+ }
+ assert_eq!(pressure_cap_after_success(0, 512 * MIB), 0);
+ assert_eq!(pressure_cap_after_yield(0, 0), 1);
+ }
+
+ #[test]
+ fn the_pressure_cap_narrows_the_static_cap() {
+ const MIB: u64 = 1024 * 1024;
+ assert_eq!(effective_batch_cap_bytes(32 * MIB, 0), 32 * MIB);
+ assert_eq!(effective_batch_cap_bytes(32 * MIB, 8 * MIB), 8 * MIB);
+ assert_eq!(effective_batch_cap_bytes(32 * MIB, 64 * MIB), 32 * MIB);
+ assert_eq!(effective_batch_cap_bytes(0, 8 * MIB), 8 * MIB);
+ assert_eq!(effective_batch_cap_bytes(0, 0), 0);
+ }
+
const GIB: u64 = 1024 * 1024 * 1024;
const MIB: u64 = 1024 * 1024;
@@ -202,4 +363,129 @@ mod tests {
assert_eq!(merge_batch_input_cap_bytes(GIB), MIN_BATCH_CAP_BYTES);
assert_eq!(merge_batch_input_cap_bytes(2 * GIB), MIN_BATCH_CAP_BYTES);
}
+
+ #[test]
+ fn merge_admission_is_open_when_the_breaker_is_off_or_rss_is_unknown() {
+ assert!(decide_merge_admission(
+ 0,
+ 10_000 * MIB,
+ 0,
+ 5,
+ true,
+ 4_000 * MIB,
+ false
+ ));
+ assert!(decide_merge_admission(
+ 2_850 * MIB,
+ 0,
+ 0,
+ 5,
+ true,
+ 4_000 * MIB,
+ false
+ ));
+ }
+
+ #[test]
+ fn merge_admission_always_lets_one_merge_run_when_not_engaged() {
+ // Even an estimate that alone exceeds the watermark starts when no
+ // other merge is in flight: merging must make progress.
+ assert!(decide_merge_admission(
+ 2_850 * MIB,
+ 2_700 * MIB,
+ 0,
+ 0,
+ false,
+ 400 * MIB,
+ false
+ ));
+ }
+
+ #[test]
+ fn merge_admission_counts_other_in_flight_reservations() {
+ let wm = 2_850 * MIB;
+ // One 47 MiB batch (376 MiB working set) already reserved.
+ let reserved = merge_working_set_bytes(47 * MIB);
+ assert!(decide_merge_admission(
+ wm,
+ 2_000 * MIB,
+ reserved,
+ 1,
+ false,
+ reserved,
+ false
+ ));
+ // At 2,200 MiB RSS a second one would cross the watermark.
+ assert!(!decide_merge_admission(
+ wm,
+ 2_200 * MIB,
+ reserved,
+ 1,
+ false,
+ reserved,
+ false
+ ));
+ // And a third does not fit at all once two are reserved.
+ assert!(!decide_merge_admission(
+ wm,
+ 2_000 * MIB,
+ 2 * reserved,
+ 2,
+ false,
+ reserved,
+ false
+ ));
+ }
+
+ #[test]
+ fn merge_admission_defers_background_merges_while_engaged() {
+ let wm = 2_850 * MIB;
+ // Engaged: no background merge starts, not even the first.
+ assert!(!decide_merge_admission(
+ wm,
+ 2_900 * MIB,
+ 0,
+ 0,
+ true,
+ 8 * MIB,
+ false
+ ));
+ assert!(!decide_merge_admission(
+ wm,
+ 1_000 * MIB,
+ 0,
+ 1,
+ true,
+ 8 * MIB,
+ false
+ ));
+ // A forced merge may run alone while engaged, but not beside another.
+ assert!(decide_merge_admission(
+ wm,
+ 2_900 * MIB,
+ 0,
+ 0,
+ true,
+ 8 * MIB,
+ true
+ ));
+ assert!(!decide_merge_admission(
+ wm,
+ 2_900 * MIB,
+ 64 * MIB,
+ 1,
+ true,
+ 8 * MIB,
+ true
+ ));
+ }
+
+ #[test]
+ fn merge_working_set_matches_the_batch_decision() {
+ assert_eq!(
+ merge_working_set_bytes(10),
+ 10 * MERGE_WORKING_SET_NUMERATOR
+ );
+ assert_eq!(merge_working_set_bytes(u64::MAX), u64::MAX);
+ }
}
diff --git a/landing/docs/env.html b/landing/docs/env.html
index 6c71c4389..8a363b621 100644
--- a/landing/docs/env.html
+++ b/landing/docs/env.html
@@ -286,6 +286,14 @@
<div class="cell d">—</div>
<div class="cell dsc">Segment hydration cache budget, same grammar. Wins over <code class="inline">limits.max_segment_hydration_cache_mb</code>; clamped to 50% of memory.</div>
+ <div class="cell k">XERJ_MEMORY_BREAKER_RELEASE_PCT</div>
+ <div class="cell d">85</div>
+ <div class="cell dsc">Percent of the memory watermark RSS must fall below before the parent memory breaker releases (50&ndash;99).</div>
+
+ <div class="cell k">XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS</div>
+ <div class="cell d">2000</div>
+ <div class="cell dsc">Minimum time the parent memory breaker stays engaged before it releases below the watermark; <code class="inline">0</code> turns it off, capped at 60000.</div>
+
<div class="cell k">XERJ_MERGE_PARALLELISM</div>
<div class="cell d">1</div>
<div class="cell dsc">Concurrent merges (<code class="inline">merge.max_concurrent</code> is not wired).</div>