AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(memory): bound merge/FST buffers under the governor and add breaker release hysteresis (#2427) (#2452)
xerj-org045aada989e2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1bc26037f..d04e93123 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -79,6 +79,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- The parent memory breaker no longer flaps, and background merges and flush+ finalizes are bounded while it is engaged (#2427). A 3,000 MB autoindex run+ engaged the breaker 20 times in 16 minutes, often releasing 100-500 ms+ later, and RSS still reached 3,808 MB because merges, FST builds and flush+ buffers kept allocating after writes were getting 429s. The breaker now+ releases below 85% of the watermark (`XERJ_MEMORY_BREAKER_RELEASE_PCT`,+ 50-99) and only after it has been engaged for 2 s+ (`XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS`). RSS that stays flat inside that+ band for 30 s still releases, so a live set no drain can shrink does not+ block writes forever. The breaker never releases while RSS is at or above+ the watermark: the 1 GB release gate (watermark 2,918 MB) released at+ 5,121 MB through the old flat-footprint release (#1981), re-engaged 100 ms+ later and was OOM-killed at 8.4 GB. A footprint that climbs while the+ breaker is engaged is drained again within 2 s instead of waiting out the+ drain backoff. While the breaker is engaged, searches run one at a time,+ and at or above the process memory limit new searches get a 429+ `circuit_breaking_exception`. Every segment merge reserves its estimated working+ set (8x its inputs) against the watermark in one process-wide ledger.+ Background merges wait while the breaker is engaged or when the+ reservation does not fit, and run on the next merge check. One merge+ always runs when nothing else is merging and the breaker is not engaged.+ `_forcemerge` may run alone while engaged. While the breaker is engaged,+ flush finalizes (segment, FST and doc-values builds) run one at a time.+ A background merge batch that is already running stops once the breaker is+ engaged and RSS reaches the process limit, keeps its inputs, and halves the+ input cap of later batches until batches complete again. On the 1 GB gate+ corpus one admitted batch (JSON segments that compress 10-20x) grew a+ 3,072 MB `xerj autoindex` node to 8,122 MB in 18 s while writes were+ getting 429s; plain `_bulk` into one index peaked at 4,967 MB.+- `xerj autoindex` no longer aborts a long run with `Resource temporarilyunavailable (os error 11)` when the machine refuses to start a thread. Thesync replay used to start a new thread for every operation (hundreds ofdiff --git a/engine/crates/xerj-engine/src/governor.rs b/engine/crates/xerj-engine/src/governor.rsindex ee07fa2e8..6d9c7c6cd 100644--- a/engine/crates/xerj-engine/src/governor.rs+++ b/engine/crates/xerj-engine/src/governor.rs@@ -46,7 +46,8 @@ static GOVERNOR: OnceLock<Arc<ResourceGovernor>> = OnceLock::new();/// The engage edge fires immediately. While the breaker stays engaged, the/// next request waits `interval`, which starts at the 2 s floor and doubles/// up to [`MEMORY_DRAIN_MAX_INTERVAL`] whenever a request did not lower RSS.-/// A drop resets the spacing. Disengaging resets everything, so the next+/// A drop resets the spacing, and a climb past the noise band requests a+/// drain as soon as the minimum spacing allows (#2427). Disengaging resets everything, so the next/// real engage is immediate again.#[derive(Debug)]pub(crate) struct MemoryDrainSchedule {@@ -74,13 +75,30 @@ impl MemoryDrainSchedule {self.rss_at_request = None;return false;}+ // #2427: the latch no longer releases above the watermark, so a+ // footprint that CLIMBS while engaged must not sit out a backed-off+ // gap (up to 15 min) — the release gate climbed from 5.1 GB to an+ // 8.4 GB OOM in 84 s. A climb past the noise band since the last+ // request is new pressure: drain again at the minimum spacing.+ let climbed = self+ .rss_at_request+ .is_some_and(|prev| rss.saturating_sub(prev) > MEMORY_RSS_NOISE_BYTES);let due = match self.last_request {None => true,- Some(at) => now.saturating_duration_since(at) >= self.interval,+ Some(at) => {+ let since = now.saturating_duration_since(at);+ since >= self.interval || (climbed && since >= self.min)+ }};if !due {return false;}+ if climbed {+ self.interval = self.min;+ self.rss_at_request = Some(rss);+ self.last_request = Some(now);+ return true;+ }if let Some(prev) = self.rss_at_request {// A one-page dip is compressor noise, not a drain that worked.// Resetting on it would pin the spacing at 2 s for a footprint@@ -108,21 +126,6 @@ impl MemoryDrainSchedule {/// tokio pool is saturated.pub const SAMPLE_INTERVAL_MS: u64 = 100;-/// How long RSS may sit flat at or above the watermark before the breaker-/// releases (#1981).-///-/// One memory-pressure drain is 2 s apart from the next. That is enough time-/// for the drain to reclaim caches and purge arenas. If RSS still has not-/// fallen, another drain will not either — on macOS jemalloc has no-/// background thread, so a footprint the purge cannot shrink (compressed-/// pages in `ri_phys_footprint`, or a live set the caches are not) used to-/// keep the sampler requesting a drain every 2 s for the rest of the-/// process. Each request flushes every index and walks the caches. That is-/// the idle-CPU climb: the node is quiet until RSS crosses the watermark,-/// then one core stays busy with drains that change nothing, and a restart-/// (RSS back under the watermark) looks like a cure.-pub const MEMORY_FLAT_RELEASE: Duration = Duration::from_secs(2);-/// Ceiling on the automatic drain spacing once a drain stops lowering RSS./// The floor is the sampler's 2 s minimum; doubling from there reaches this/// in a few minutes instead of staying at 2 s for an 18-hour uptime.@@ -133,11 +136,94 @@ pub const MEMORY_DRAIN_MAX_INTERVAL: Duration = Duration::from_secs(15 * 60);////// macOS reports `ri_phys_footprint`, which moves by pages while the/// compressor runs even when the process is idle. An exact-byte clock never-/// stays flat for [`MEMORY_FLAT_RELEASE`] on that signal, and a one-byte dip-/// would also reset the drain backoff, so the 2 s flush loop would survive-/// the latch fix. A climb past this band is a real working-set change.+/// stays flat on that signal, and a one-byte dip would also reset the drain+/// backoff, so the 2 s flush loop would survive the latch fix. A move past+/// this band is a real working-set change.pub const MEMORY_RSS_NOISE_BYTES: u64 = 4 * 1024 * 1024;+/// Default release point of the RSS breaker, as a percentage of the+/// watermark that tripped it (#2427).+///+/// The old band was 95% and released on the first sample that did not fall.+/// A 3,000 MB autoindex run engaged at 2,918 MB and flapped 20 times in 16+/// minutes, often releasing 100-500 ms after it engaged: each release+/// re-admitted the write burst that tripped it. Releasing at 85% leaves a+/// band the drain has to clear before writes come back.+/// `XERJ_MEMORY_BREAKER_RELEASE_PCT` overrides it (50..=99).+pub const DEFAULT_MEMORY_RELEASE_PERCENT: u8 = 85;++/// Lowest accepted `XERJ_MEMORY_BREAKER_RELEASE_PCT`. Below this, a footprint+/// whose live set is over half the watermark could never release.+pub const MIN_MEMORY_RELEASE_PERCENT: u8 = 50;++/// Highest accepted `XERJ_MEMORY_BREAKER_RELEASE_PCT`.+pub const MAX_MEMORY_RELEASE_PERCENT: u8 = 99;++/// Default minimum time the RSS breaker stays engaged before a release below+/// the watermark (#2427). `XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS` overrides it.+///+/// This stops a 100 ms engage/release cycle: a release needs the drain to+/// have run at least once. A live retune (#1323) or Windows commit recovery+/// are not subject to it. No release ever happens at or above the watermark.+pub const DEFAULT_MEMORY_MIN_ENGAGED: Duration = Duration::from_secs(2);++/// Ceiling on `XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS`.+pub const MAX_MEMORY_MIN_ENGAGED: Duration = Duration::from_secs(60);++/// How long RSS may sit without falling past [`MEMORY_RSS_NOISE_BYTES`]+/// inside the release band `[release, watermark)` before the breaker stops+/// waiting for the drain and releases (#2427).+///+/// Without this the band could hold writes forever on a live set between+/// 85% and 100% of the watermark that no drain can shrink (the #1250+/// concern). Thirty seconds is fifteen drain intervals, and bounds a+/// band-stall re-engage cycle to at most one every 30 s.+pub const MEMORY_BAND_STALL_RELEASE: Duration = Duration::from_secs(30);++/// Bytes at which an engaged RSS breaker releases: `release_pct` percent of+/// `watermark`. `0` for a disabled watermark.+pub fn memory_release_bytes(watermark: u64, release_pct: u8) -> u64 {+ ((watermark as u128 * u128::from(release_pct.min(100))) / 100) as u64+}++/// Parse `XERJ_MEMORY_BREAKER_RELEASE_PCT`. Unset, unparsable, or out of+/// `MIN..=MAX` falls back to [`DEFAULT_MEMORY_RELEASE_PERCENT`].+pub fn resolve_memory_release_percent(raw: Option<&str>) -> u8 {+ let Some(raw) = raw.map(str::trim).filter(|s| !s.is_empty()) else {+ return DEFAULT_MEMORY_RELEASE_PERCENT;+ };+ match raw.trim_end_matches('%').parse::<u8>() {+ Ok(pct) if (MIN_MEMORY_RELEASE_PERCENT..=MAX_MEMORY_RELEASE_PERCENT).contains(&pct) => pct,+ _ => {+ tracing::warn!(+ value = raw,+ default = DEFAULT_MEMORY_RELEASE_PERCENT,+ "XERJ_MEMORY_BREAKER_RELEASE_PCT must be {MIN_MEMORY_RELEASE_PERCENT}..={MAX_MEMORY_RELEASE_PERCENT}; using the default"+ );+ DEFAULT_MEMORY_RELEASE_PERCENT+ }+ }+}++/// Parse `XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS`. Unset or unparsable falls back+/// to [`DEFAULT_MEMORY_MIN_ENGAGED`]; larger values are clamped to+/// [`MAX_MEMORY_MIN_ENGAGED`]. `0` turns the minimum off.+pub fn resolve_memory_min_engaged(raw: Option<&str>) -> Duration {+ let Some(raw) = raw.map(str::trim).filter(|s| !s.is_empty()) else {+ return DEFAULT_MEMORY_MIN_ENGAGED;+ };+ match raw.parse::<u64>() {+ Ok(ms) => Duration::from_millis(ms).min(MAX_MEMORY_MIN_ENGAGED),+ Err(_) => {+ tracing::warn!(+ value = raw,+ "XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS is not a whole number of milliseconds; using the default"+ );+ DEFAULT_MEMORY_MIN_ENGAGED+ }+ }+}+/// Bytes of analyze-time reservation charged per raw JSON input byte (#1122).////// `document_limit` measured about 56 bytes of RSS per token for a@@ -223,17 +309,28 @@ pub struct ResourceGovernor {/// tripped; when an operator re-tunes the watermark live/// (`indices.breaker.total.limit`, #1323) the latch is re-evaluated/// against the new value without the band, so a live raise releases at- /// once instead of waiting for RSS to fall 5% below the NEW watermark.+ /// once instead of waiting for RSS to fall below the release point of+ /// the NEW watermark.memory_trip_watermark: AtomicU64,- /// When RSS first stopped falling while the breaker was engaged at or- /// above the watermark. `None` while RSS is falling or the breaker is+ /// When RSS last stopped falling inside the release band while the+ /// breaker was engaged (#2427). `None` outside the band or while/// disengaged. Aged by tests; the sampler's wall clock fills it in- /// production (#1981).+ /// production.memory_flat_since: Mutex<Option<Instant>>,- /// RSS at which a flat footprint was released. Later samples at or below- /// this stay disengaged, so the latch cannot flap back on at the same- /// reading one sample later. `0` means no floor.- memory_stall_floor: AtomicU64,+ /// Release point of the RSS breaker, percent of the tripped watermark+ /// (#2427). See [`DEFAULT_MEMORY_RELEASE_PERCENT`].+ memory_release_percent: u8,+ /// Minimum time engaged before an in-band or below-band release (#2427).+ memory_min_engaged: Duration,+ /// When the RSS breaker last engaged. `None` while released.+ memory_engaged_since: Mutex<Option<Instant>>,+ /// RSS at which the in-band stall clock started (#2427). A sample more+ /// than [`MEMORY_RSS_NOISE_BYTES`] under it is a falling drain and+ /// restarts the clock. `0` = no band clock.+ memory_band_anchor: AtomicU64,+ /// Process-wide ledger of in-flight segment merges and their estimated+ /// working set (#2427). See [`ResourceGovernor::try_reserve_merge`].+ merge_ledger: Arc<Mutex<MergeLedger>>,// ── item 1: per-query memory guard (`max_query_memory_mb`) ──────────/// Maximum bytes a single query may be estimated to allocate. `0` = off.@@ -249,6 +346,10 @@ pub struct ResourceGovernor {search_inflight: Arc<AtomicU64>,/// High-water mark of concurrent searches observed (proof of the cap).search_inflight_peak: Arc<AtomicU64>,+ /// #2427: one permit. While the RSS breaker is engaged every search+ /// also holds this, so searches run one at a time instead of+ /// `max_concurrent_searches` wide.+ search_pressure: Arc<Semaphore>,// ── item 2b: global bulk pool (concurrent bulk parse cap) ─────────/// Global bulk-concurrency permits. Caps the number of concurrent@@ -487,8 +588,8 @@ impl ResourceGovernor {.unwrap_or_else(|e| e.into_inner()) = None;}- /// #1981 test hook: age the flat-footprint clock without sleeping, so a- /// test can cross [`MEMORY_FLAT_RELEASE`] on a paused schedule.+ /// Test hook: age the flat-footprint clock without sleeping, so a test+ /// can cross [`MEMORY_BAND_STALL_RELEASE`] on a paused schedule.#[cfg(test)]fn age_memory_flat_for_test(&self, by: Duration) {let mut slot = self@@ -500,6 +601,139 @@ impl ResourceGovernor {}}+ /// Release point of the RSS breaker as a percentage of the watermark.+ pub fn memory_release_percent(&self) -> u8 {+ self.memory_release_percent+ }++ /// Minimum time the RSS breaker stays engaged before an in-band release.+ pub fn memory_min_engaged(&self) -> Duration {+ self.memory_min_engaged+ }++ /// How long the RSS breaker has been engaged. A latch with no recorded+ /// edge (the startup commit trip) counts as having met any minimum.+ fn memory_engaged_for(&self) -> Duration {+ match *self+ .memory_engaged_since+ .lock()+ .unwrap_or_else(|e| e.into_inner())+ {+ Some(at) => at.elapsed(),+ None => Duration::MAX,+ }+ }++ fn set_memory_engaged_since(&self, at: Option<Instant>) {+ *self+ .memory_engaged_since+ .lock()+ .unwrap_or_else(|e| e.into_inner()) = at;+ }++ /// #2427 test hook: age the engaged clock without sleeping.+ #[cfg(test)]+ fn age_memory_engaged_for_test(&self, by: Duration) {+ let mut slot = self+ .memory_engaged_since+ .lock()+ .unwrap_or_else(|e| e.into_inner());+ if let Some(at) = *slot {+ *slot = Some(at.checked_sub(by).unwrap_or(at));+ }+ }++ /// Bytes reserved by in-flight segment merges (#2427).+ pub fn merge_reserved_bytes(&self) -> u64 {+ self.merge_ledger+ .lock()+ .unwrap_or_else(|e| e.into_inner())+ .reserved+ }++ /// Segment merges currently holding a [`MergeReservation`] (#2427).+ pub fn merges_in_flight(&self) -> u64 {+ self.merge_ledger+ .lock()+ .unwrap_or_else(|e| e.into_inner())+ .in_flight+ }++ /// Reserve the estimated working set of one merge batch whose inputs+ /// total `input_bytes` against the RSS watermark (#2427).+ ///+ /// Before this, every index started its first merge batch whatever the+ /// process footprint, and each batch was checked against an RSS snapshot+ /// that did not see other indices' merges that had started but not yet+ /// allocated. Under autoindex (hundreds of indices) that put many merge+ /// and FST working sets on top of a node already at the watermark: RSS+ /// climbed about 1 GB past a 3,000 MB cap while writes were getting 429s.+ ///+ /// The reservation is `input_bytes * MERGE_WORKING_SET_NUMERATOR` and is+ /// held until the guard drops. See `merge_budget::decide_merge_admission`+ /// for the rule.+ /// `None` means defer: the caller leaves the batch for a later pass.+ pub fn try_reserve_merge(&self, input_bytes: u64, forced: bool) -> Option<MergeReservation> {+ let bytes = crate::merge_budget::merge_working_set_bytes(input_bytes);+ let watermark = self.memory_watermark_bytes();+ let observed = self.observed_process_memory();+ let engaged = self.memory_breaker_engaged();+ let mut ledger = self.merge_ledger.lock().unwrap_or_else(|e| e.into_inner());+ if !crate::merge_budget::decide_merge_admission(+ watermark,+ observed,+ ledger.reserved,+ ledger.in_flight,+ engaged,+ bytes,+ forced,+ ) {+ return None;+ }+ ledger.reserved = ledger.reserved.saturating_add(bytes);+ ledger.in_flight += 1;+ Some(MergeReservation {+ ledger: Arc::clone(&self.merge_ledger),+ bytes,+ })+ }++ /// Whether an in-flight background merge batch must stop now (#2427).+ /// See `merge_budget::merge_must_yield`.+ pub fn merge_must_yield(&self, forced: bool) -> bool {+ crate::merge_budget::merge_must_yield(+ self.observed_process_memory(),+ self.memory_limit_bytes,+ self.memory_breaker_engaged(),+ forced,+ )+ }++ /// Record a background batch of `input_bytes` that stopped under memory+ /// pressure: the next batches are planned under half of it (#2427).+ pub fn note_merge_yield(&self, input_bytes: u64) -> u64 {+ let mut ledger = self.merge_ledger.lock().unwrap_or_else(|e| e.into_inner());+ ledger.pressure_cap =+ crate::merge_budget::pressure_cap_after_yield(ledger.pressure_cap, input_bytes);+ ledger.pressure_cap+ }++ /// Record a background batch of `input_bytes` that completed: a batch+ /// that used the adaptive cap widens it again (#2427).+ pub fn note_merge_completed(&self, input_bytes: u64) {+ let mut ledger = self.merge_ledger.lock().unwrap_or_else(|e| e.into_inner());+ ledger.pressure_cap =+ crate::merge_budget::pressure_cap_after_success(ledger.pressure_cap, input_bytes);+ }++ /// The adaptive merge batch input cap, `0` when none is set (#2427).+ pub fn merge_pressure_cap_bytes(&self) -> u64 {+ self.merge_ledger+ .lock()+ .unwrap_or_else(|e| e.into_inner())+ .pressure_cap+ }+/// Whether the summed-memtable budget breaker is currently engaged.pub fn memtable_breaker_engaged(&self) -> bool {self.memtable_tripped.load(Ordering::Relaxed)@@ -671,19 +905,65 @@ impl ResourceGovernor {/// the permit (and decrements the in-flight gauge) on drop. Excess/// searches queue on the semaphore, exactly like ES's search thread/// pool bounds active workers.+ ///+ /// Under memory pressure (#2427) searches are non-write memory the RSS+ /// breaker never saw: the 1 GB release gate kept climbing from 5.1 GB to+ /// an 8.4 GB OOM with writes blocked, and its search phase ran 30 s+ /// multi-index searches against a node already over its watermark.+ /// While the breaker is engaged each search also takes the single+ /// `search_pressure` permit, so they run one at a time. At or above the+ /// hard process limit a new search is rejected with a 429+ /// `circuit_breaking_exception` instead of queueing.pub async fn acquire_search(&self) -> Result<SearchPermit, XerjError> {+ self.check_search_hard_limit()?;let permit = Arc::clone(&self.search_pool).acquire_owned().await.map_err(|_| XerjError::internal("global search pool closed — shutting down"))?;+ let pressure = if self.memory_breaker_engaged() {+ let p = Arc::clone(&self.search_pressure)+ .acquire_owned()+ .await+ .map_err(|_| XerjError::internal("search pressure gate closed — shutting down"))?;+ // RSS may have crossed the hard limit while this search queued.+ self.check_search_hard_limit()?;+ Some(p)+ } else {+ None+ };let now = self.search_inflight.fetch_add(1, Ordering::Relaxed) + 1;self.search_inflight_peak.fetch_max(now, Ordering::Relaxed);Ok(SearchPermit {_permit: permit,+ _pressure: pressure,inflight: Arc::clone(&self.search_inflight),})}+ /// RSS line at which new searches are rejected (#2427): the process+ /// memory limit, or the live watermark if an operator raised it past+ /// that (#1323). `0` when no limit is known.+ fn search_hard_limit_bytes(&self) -> u64 {+ let watermark = self.memory_watermark_bytes.load(Ordering::Relaxed);+ if watermark == 0 {+ return 0;+ }+ self.memory_limit_bytes.max(watermark)+ }++ fn check_search_hard_limit(&self) -> Result<(), XerjError> {+ let limit = self.search_hard_limit_bytes();+ let rss = self.rss_bytes.load(Ordering::Relaxed);+ if limit != 0 && rss >= limit {+ return Err(XerjError::circuit_breaking(format!(+ "[parent] Data too large: process memory [{rss}/{}] is at or above the memory limit [{limit}/{}]; search rejected until the memory breaker drains",+ human_bytes(rss),+ human_bytes(limit)+ )));+ }+ Ok(())+ }+/// Current in-flight search count (for stats / proof).pub fn search_inflight(&self) -> u64 {self.search_inflight.load(Ordering::Relaxed)@@ -748,10 +1028,9 @@ impl ResourceGovernor {/// [`Self::refresh_memory_disk`] plus an optional Windows available-commit/// sample. `None` leaves the commit latch as it was (Linux, and tests/// that only move RSS). A `Some` below the commit reserve engages the- /// breaker even when `rss` is under the watermark (#1122). That hold is- /// independent of the flat-footprint release (#1981): a footprint that- /// does not fall still releases once commit is back above the reserve,- /// and commit starvation does not clear the flat clock.+ /// breaker even when `rss` is under the watermark (#1122). Once commit is+ /// back above the reserve the RSS latch alone decides, and that latch+ /// never releases at or above the watermark (#2427).pub fn refresh_memory_observing(&self,rss: u64,@@ -759,43 +1038,32 @@ impl ResourceGovernor {disk_avail_bytes: u64,available_commit: Option<u64>,) {- // ── RSS watermark, with a bounded release hysteresis (#1250) ──+ // ── RSS watermark, with release hysteresis (#1250, #2427) ──//- // Engage at `rss >= watermark`. Once engaged, hold the latch while RSS- // sits in the band `[95% of watermark, watermark)` only for as long as- // RSS is still FALLING — the memory-pressure drain is making progress,- // and releasing now would re-admit the write burst that re-trips the- // breaker a sample later (the rss=15,596MB vs watermark=15,564MB- // chatter in #1250). If RSS stops falling inside the band the drain- // has nothing more to free: holding would only reject writes the node- // can take and keep the sampler re-draining every- // `MEMORY_DRAIN_MIN_INTERVAL` forever, so the latch releases exactly- // as it did before the band existed.+ // Engage at `rss >= watermark`. Once engaged, the latch releases below+ // `memory_release_percent` of the watermark (85% by default) after the+ // breaker has been engaged for `memory_min_engaged`. Inside the band+ // `[release, watermark)` it holds while the drain is still lowering+ // RSS. A 95% band that released on the first sample that did not fall+ // flapped 20 times in 16 minutes under autoindex (#2427), each release+ // re-admitting the write burst that tripped it. If RSS does not fall+ // past [`MEMORY_RSS_NOISE_BYTES`] inside the band for+ // [`MEMORY_BAND_STALL_RELEASE`], the drain has nothing more to free:+ // holding would only reject writes the node can take and keep the+ // sampler re-draining forever, so the latch releases.//- // The same "stopped falling" rule applies AT or ABOVE the watermark- // (#1981), after [`MEMORY_FLAT_RELEASE`], with a- // [`MEMORY_RSS_NOISE_BYTES`] band so page-sized footprint noise is- // not a climb or a recovery. A footprint that does not fall past- // that band is not a drain the sampler should repeat for the life- // of the process. The release records `memory_stall_floor` so a- // later sample inside the band does not engage again — without that- // floor `rss >= watermark` would re-trip one sample later and the- // drain edge would fire forever. A climb past the band is new- // pressure and engages. A live watermark change (#1323) drops the- // band and the floor for that sample: the operator's new value- // decides.- let prev_rss = self.rss_bytes.swap(rss, Ordering::Relaxed);+ // At or above the watermark the latch never releases (#2427). The+ // #1981 flat-footprint release that used to apply there is gone: it+ // released a node at 5,121MB against a 2,918MB watermark. A live+ // watermark change (#1323) is judged against the operator's new+ // value on that sample.+ self.rss_bytes.store(rss, Ordering::Relaxed);let watermark = self.memory_watermark_bytes.load(Ordering::Relaxed);let cur_mem = self.memory_tripped.load(Ordering::Relaxed);let retuned = self.memory_trip_watermark.load(Ordering::Relaxed) != watermark;- let release_bytes = watermark - (watermark / 20);- let stall_floor = self.memory_stall_floor.load(Ordering::Relaxed);- // In-band hysteresis (#1250) still uses an exact comparison: one- // sample that does not fall releases. Above the watermark the same- // comparison would treat compressor noise as a climb or a recovery.- let falling = rss < prev_rss;- let dropped = prev_rss.saturating_sub(rss) > MEMORY_RSS_NOISE_BYTES;- let climbed = rss.saturating_sub(prev_rss) > MEMORY_RSS_NOISE_BYTES;+ // #2427: the release point is `memory_release_percent` of the+ // watermark (85% by default, was a fixed 95%).+ let release_bytes = memory_release_bytes(watermark, self.memory_release_percent);// `None` keeps the previous latch so an RSS-only refresh (Linux, and// the test helper) does not clear a commit trip it did not sample.// Reserve 0 means this platform has no commit probe: starved stays@@ -810,61 +1078,74 @@ impl ResourceGovernor {self.commit_starved.load(Ordering::Relaxed)};self.commit_starved.store(starved, Ordering::Relaxed);- // RSS latch (#1250, #1981). Commit starvation is applied after this- // so a flat footprint can release, and a pagefileless box still- // stays engaged while available commit is under the reserve.+ // RSS latch (#1250, #2427). Commit starvation is applied after this+ // so a pagefileless box stays engaged while available commit is+ // under the reserve.let rss_next = watermark != 0&& if rss >= watermark {- let within_floor =- stall_floor != 0 && rss <= stall_floor.saturating_add(MEMORY_RSS_NOISE_BYTES);- if !cur_mem && within_floor && !retuned {- false- } else if cur_mem && !retuned && !dropped {- if climbed {- // Still climbing past noise: the drain has not- // caught the working set. Forget the flat clock.+ // #2427: at or above the watermark the RSS latch ALWAYS holds+ // (or engages). There is no release up here, whatever the+ // footprint is doing. The #1981 flat-footprint release that+ // used to live in this arm let a 1 GB release-gate node+ // release at rss=5,121MB against a 2,918MB watermark, re-engage+ // 100 ms later at 5,125MB, and get OOM-killed at 8.4 GB. The+ // drain backoff (`MemoryDrainSchedule`) is what keeps a+ // footprint that will not fall from re-draining every 2 s.+ self.memory_band_anchor.store(0, Ordering::Relaxed);+ self.clear_memory_flat();+ true+ } else if cur_mem && !retuned {+ // #2427 release hysteresis, strictly below the watermark.+ // Below the release point, release once the minimum engaged+ // time has passed. Inside the band, hold while the drain is+ // still lowering RSS; if RSS has not fallen past the noise+ // band for MEMORY_BAND_STALL_RELEASE, the drain has nothing+ // left to free, so release rather than reject writes forever+ // (#1250).+ let min_met = self.memory_engaged_for() >= self.memory_min_engaged;+ if rss < release_bytes {+ self.clear_memory_flat();+ self.memory_band_anchor.store(0, Ordering::Relaxed);+ !min_met+ } else {+ let anchor = self.memory_band_anchor.load(Ordering::Relaxed);+ if anchor == 0 || rss.saturating_add(MEMORY_RSS_NOISE_BYTES) < anchor {+ // Entered the band, or fell past the noise band:+ // the drain is working. Restart the stall clock.+ self.memory_band_anchor.store(rss, Ordering::Relaxed);self.clear_memory_flat();- self.memory_stall_floor.store(0, Ordering::Relaxed);+ self.note_memory_flat();true} else {- let flat_for = self.note_memory_flat();- // Starvation must not record a stall floor: that- // floor means "released at this footprint", and the- // breaker has not released. The clock keeps running- // so the release happens on the first sample after- // commit recovers, once the window has elapsed.- if !starved && flat_for >= MEMORY_FLAT_RELEASE {- self.memory_stall_floor.store(rss, Ordering::Relaxed);+ let stalled = self.note_memory_flat() >= MEMORY_BAND_STALL_RELEASE;+ if stalled && min_met {self.clear_memory_flat();+ self.memory_band_anchor.store(0, Ordering::Relaxed);false} else {true}}- } else {- // Dropped past the noise band, a live retune, or the- // engage edge. Reaching this arm means one of those is- // true, so the flat clock and the floor are stale.- self.clear_memory_flat();- self.memory_stall_floor.store(0, Ordering::Relaxed);- true}- } else if cur_mem && !retuned {- self.clear_memory_flat();- rss >= release_bytes && falling} else {+ // Disengaged under the watermark, or a live retune (#1323)+ // that put RSS under the operator's new value.self.clear_memory_flat();- if !cur_mem || retuned {- self.memory_stall_floor.store(0, Ordering::Relaxed);- }+ self.memory_band_anchor.store(0, Ordering::Relaxed);false};+ // #2427 invariant: the RSS latch never reports released while RSS is+ // at or above the live watermark.+ debug_assert!(rss_next || watermark == 0 || rss < watermark);let mem_next = starved || rss_next;if mem_next {self.memory_trip_watermark.store(watermark, Ordering::Relaxed);}if mem_next != self.memory_tripped.swap(mem_next, Ordering::Relaxed) {+ // Only an RSS engage starts the minimum-engaged clock: a Windows+ // commit trip releases as soon as commit recovers.+ self.set_memory_engaged_since((mem_next && rss_next).then(Instant::now));if mem_next && starved {tracing::warn!(available_commit_mb = self.available_commit_bytes.load(Ordering::Relaxed)@@ -882,6 +1163,7 @@ impl ResourceGovernor {} else {tracing::info!(rss_mb = rss / (1024 * 1024),+ release_mb = release_bytes / (1024 * 1024),"RSS back below the memory watermark — releasing the memory circuit breaker");}@@ -984,6 +1266,9 @@ impl ResourceGovernor {/// gauge on drop; the underlying semaphore permit is released with it.pub struct SearchPermit {_permit: OwnedSemaphorePermit,+ /// The memory-pressure gate's single permit, held when the search was+ /// admitted while the RSS breaker was engaged (#2427).+ _pressure: Option<OwnedSemaphorePermit>,inflight: Arc<AtomicU64>,}@@ -1483,8 +1768,20 @@ fn build(config: &Config) -> ResourceGovernor {);}+ let memory_release_percent = resolve_memory_release_percent(+ std::env::var("XERJ_MEMORY_BREAKER_RELEASE_PCT")+ .ok()+ .as_deref(),+ );+ let memory_min_engaged = resolve_memory_min_engaged(+ std::env::var("XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS")+ .ok()+ .as_deref(),+ );tracing::info!(memtable_budget_mb = memtable_budget_bytes / (1024 * 1024),+ memory_release_pct = memory_release_percent,+ memory_min_engaged_ms = memory_min_engaged.as_millis() as u64,memory_limit_mb = memory_limit_bytes / (1024 * 1024),memory_limit_source = ?resolved_process_cap.source,memory_watermark_mb = memory_watermark_bytes / (1024 * 1024),@@ -1521,12 +1818,17 @@ fn build(config: &Config) -> ResourceGovernor {0}),memory_flat_since: Mutex::new(None),- memory_stall_floor: AtomicU64::new(0),+ memory_release_percent,+ memory_min_engaged,+ memory_engaged_since: Mutex::new(None),+ memory_band_anchor: AtomicU64::new(0),+ merge_ledger: Arc::new(Mutex::new(MergeLedger::default())),max_query_memory_bytes,search_pool: Arc::new(Semaphore::new(max_concurrent_searches)),max_concurrent_searches,search_inflight: Arc::new(AtomicU64::new(0)),search_inflight_peak: Arc::new(AtomicU64::new(0)),+ search_pressure: Arc::new(Semaphore::new(1)),bulk_pool: Arc::new(Semaphore::new(max_concurrent_bulks)),max_concurrent_bulks,disk_flood_pct_default: limits.disk_flood_stage_percent.min(100),@@ -1938,6 +2240,49 @@ fn parse_byte_size(s: &str) -> Option<u64> {// Small formatting helpers// ─────────────────────────────────────────────────────────────────────────+/// Process-wide merge accounting behind [`MergeReservation`] (#2427).+#[derive(Debug, Default)]+struct MergeLedger {+ reserved: u64,+ in_flight: u64,+ /// Adaptive batch input cap learned from batches that had to yield+ /// under memory pressure. `0` = none. See+ /// `merge_budget::pressure_cap_after_yield`.+ pressure_cap: u64,+}++/// Holds one merge batch's estimated working set against the RSS watermark+/// (#2427). Returned by [`ResourceGovernor::try_reserve_merge`]; dropping it+/// returns the bytes and the in-flight slot.+#[must_use = "dropping the reservation releases the merge budget at once"]+pub struct MergeReservation {+ ledger: Arc<Mutex<MergeLedger>>,+ bytes: u64,+}++impl MergeReservation {+ /// Bytes this reservation holds.+ pub fn bytes(&self) -> u64 {+ self.bytes+ }+}++impl std::fmt::Debug for MergeReservation {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ f.debug_struct("MergeReservation")+ .field("bytes", &self.bytes)+ .finish()+ }+}++impl Drop for MergeReservation {+ fn drop(&mut self) {+ let mut ledger = self.ledger.lock().unwrap_or_else(|e| e.into_inner());+ ledger.reserved = ledger.reserved.saturating_sub(self.bytes);+ ledger.in_flight = ledger.in_flight.saturating_sub(1);+ }+}+/// Releases an analyze reservation when the batch's tokens exist (#1122).#[derive(Debug)]pub(crate) struct AnalyzeReservation {@@ -2800,6 +3145,8 @@ mod tests {g.memory_breaker_engaged(),"RSS at the watermark engages the breaker");+ // #2427: past the minimum engaged time.+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));g.refresh(0, watermark / 2, 0);assert!(!g.memory_breaker_engaged(),@@ -2831,6 +3178,8 @@ mod tests {!g.thread_spawn_safe(),"breaker engaged: spawning would risk #1122's abort");+ // #2427: past the minimum engaged time.+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));g.refresh(0, watermark / 2, 0);assert!(g.thread_spawn_safe(),@@ -2948,14 +3297,13 @@ mod tests {assert!(g.thread_spawn_safe());}- /// #1122 x #1981: commit starvation and a flat footprint are different- /// holds. A footprint that sits on the watermark must not release the- /// breaker while available commit is still under the reserve — that- /// release is what lets the next allocation abort. The flat clock keeps- /// running, so the first sample after commit recovers releases instead- /// of starting the 2 s window over, and that footprint does not re-arm.+ /// #1122 x #2427: commit starvation and RSS above the watermark are+ /// independent holds, and neither releases while RSS is at or above the+ /// watermark. Commit coming back does not release a footprint that is+ /// still over the line, however long it has sat flat; RSS falling under+ /// the release point (after the minimum engaged time) does.#[test]- fn commit_starvation_holds_a_flat_footprint_until_commit_returns() {+ fn commit_recovery_does_not_release_rss_above_the_watermark() {const MIB: u64 = 1024 * 1024;const GIB: u64 = 1024 * MIB;let mut cfg = Config::default();@@ -2975,29 +3323,23 @@ mod tests {g.refresh_memory_observing(stuck, 0, 0, Some(128 * MIB));assert!(g.memory_breaker_engaged(), "crossing the watermark engages");g.refresh_memory_observing(stuck, 0, 0, Some(128 * MIB));- g.age_memory_flat_for_test(MEMORY_FLAT_RELEASE + Duration::from_millis(1));+ g.age_memory_engaged_for_test(Duration::from_secs(3600));g.refresh_memory_observing(stuck, 0, 0, Some(128 * MIB));- assert!(- g.memory_breaker_engaged(),- "a flat footprint must not release a breaker commit starvation is holding"- );+ assert!(g.memory_breaker_engaged());let msg = format!("{}", g.check_ingest_admission().unwrap_err());assert!(msg.contains("pagefile"), "{msg}");- g.refresh_memory_observing(stuck, 0, 0, Some(2 * GIB));- assert!(- !g.memory_breaker_engaged(),- "once commit is back, a footprint that already sat flat releases"- );- g.refresh_memory_observing(stuck, 0, 0, Some(2 * GIB));+ for _ in 0..50 {+ g.refresh_memory_observing(stuck, 0, 0, Some(2 * GIB));+ assert!(+ g.memory_breaker_engaged(),+ "commit recovery must not release RSS that is still above the watermark"+ );+ }+ g.refresh_memory_observing(wm / 2, 0, 0, Some(2 * GIB));assert!(!g.memory_breaker_engaged(),- "the same footprint must not re-arm"- );- g.refresh_memory_observing(stuck + MEMORY_RSS_NOISE_BYTES + 1, 0, 0, Some(2 * GIB));- assert!(- g.memory_breaker_engaged(),- "a climb past the noise band is new pressure"+ "RSS under the release point with commit back releases");}@@ -3287,6 +3629,80 @@ mod tests {drop(p2);}+ /// #2427 (b): while the RSS breaker is engaged, searches run one at a+ /// time; released, the full pool is available again.+ #[tokio::test]+ async fn engaged_memory_breaker_serializes_searches() {+ let mut cfg = Config::default();+ cfg.limits.max_concurrent_searches = 8;+ cfg.limits.max_total_memtable_mb = 0;+ cfg.limits.memory_watermark_percent = 95;+ let g = Arc::new(build(&cfg));+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);+ assert!(wm > 0, "this test needs a live RSS watermark");++ // Engaged, but under the hard limit: admitted, one at a time.+ g.refresh(0, wm, 0);+ assert!(g.memory_breaker_engaged());+ assert!(+ wm < g.search_hard_limit_bytes(),+ "95% watermark sits under the limit"+ );+ let p1 = g.acquire_search().await.unwrap();+ let g2 = Arc::clone(&g);+ let second = tokio::spawn(async move { g2.acquire_search().await.map(|_p| ()) });+ assert!(+ tokio::time::timeout(Duration::from_millis(150), second)+ .await+ .is_err(),+ "a second search must wait while the breaker is engaged"+ );+ drop(p1);+ let p2 = tokio::time::timeout(Duration::from_millis(500), g.acquire_search())+ .await+ .expect("the gate frees with the first search")+ .unwrap();+ drop(p2);++ // Released: concurrency is back to the pool size.+ g.age_memory_engaged_for_test(Duration::from_secs(60));+ g.refresh(0, 0, 0);+ assert!(!g.memory_breaker_engaged());+ let a = g.acquire_search().await.unwrap();+ let b = tokio::time::timeout(Duration::from_millis(500), g.acquire_search())+ .await+ .expect("a released breaker does not serialize")+ .unwrap();+ assert_eq!(g.search_inflight(), 2);+ drop((a, b));+ }++ /// #2427 (b): at or above the hard process limit a new search is a 429,+ /// not another allocation on a node the kernel is about to kill.+ #[tokio::test]+ async fn search_is_rejected_at_the_hard_memory_limit() {+ let mut cfg = Config::default();+ cfg.limits.max_total_memtable_mb = 0;+ cfg.limits.memory_watermark_percent = 95;+ let g = build(&cfg);+ let limit = g.search_hard_limit_bytes();+ assert!(limit > 0, "this test needs a live memory limit");++ g.refresh(0, limit, 0);+ let err = match g.acquire_search().await {+ Ok(_) => panic!("a search at the hard limit must be rejected"),+ Err(e) => e,+ };+ assert_eq!(err.http_status(), 429, "{err}");+ assert_eq!(g.search_inflight(), 0, "a rejected search holds no permit");++ g.refresh(0, limit - 1, 0);+ assert!(+ g.acquire_search().await.is_ok(),+ "just under the hard limit is admitted (serialized)"+ );+ }+#[test]fn memory_watermark_releases_with_hysteresis() {let mut cfg = Config::default();@@ -3299,7 +3715,7 @@ mod tests {g.refresh(0, wm, 0);assert!(g.check_ingest_admission().is_err(), "trips at watermark");- // 98% of watermark is above the 95% release threshold and RSS is+ // 98% of watermark is above the 85% release threshold and RSS is// still falling (the drain is working): stays tripped.g.refresh(0, wm * 98 / 100, 0);let err = g.check_ingest_admission().unwrap_err().to_string();@@ -3324,19 +3740,40 @@ mod tests {"must remain tripped while RSS drains through the band");- // 94% of watermark is below the 95% release threshold: releases.+ // #2427: 94% and 86% are inside the 85% release band now, and RSS+ // is still falling: stays tripped.g.refresh(0, wm * 94 / 100, 0);+ assert!(+ g.check_ingest_admission().is_err(),+ "94% of watermark is inside the 85% release band"+ );+ g.refresh(0, wm * 86 / 100, 0);+ assert!(+ g.check_ingest_admission().is_err(),+ "86% is still in the band"+ );++ // Below 85%, but the breaker engaged moments ago: the minimum+ // engaged time holds it so a drain can run.+ g.refresh(0, wm * 84 / 100, 0);+ assert!(+ g.memory_breaker_engaged(),+ "a release inside the minimum engaged time is the 100 ms flap"+ );+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));+ g.refresh(0, wm * 84 / 100, 0);assert!(g.check_ingest_admission().is_ok(),- "must release once RSS drops below 95% of watermark"+ "must release once RSS is below 85% of watermark and the minimum has passed");}- /// #1250 review: a node whose irreducible resident set settles INSIDE the- /// band after a drain must not stay write-blocked (and re-draining every- /// `MEMORY_DRAIN_MIN_INTERVAL`) forever. Once RSS stops falling the band- /// has nothing left to wait for and the latch releases, which is exactly- /// the pre-band behaviour.+ /// #1250 review, re-tuned for #2427: a node whose irreducible resident+ /// set settles INSIDE the band after a drain must not stay write-blocked+ /// (and re-draining every `MEMORY_DRAIN_MIN_INTERVAL`) forever. One flat+ /// sample is no longer enough (that released 100-500 ms after engaging+ /// under autoindex); RSS has to stay flat for+ /// [`MEMORY_BAND_STALL_RELEASE`].#[test]fn memory_hysteresis_releases_when_rss_stalls_inside_the_band() {let mut cfg = Config::default();@@ -3352,6 +3789,16 @@ mod tests {assert!(g.memory_breaker_engaged(), "still draining");// The drain freed nothing more: RSS is flat inside the band.g.refresh(0, wm * 97 / 100, 0);+ assert!(+ g.memory_breaker_engaged(),+ "one flat in-band sample must not release (#2427 flap)"+ );+ // Page-sized wander is still flat.+ g.refresh(0, wm * 97 / 100 + 4096, 0);+ assert!(g.memory_breaker_engaged());+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));+ g.age_memory_flat_for_test(MEMORY_BAND_STALL_RELEASE + Duration::from_millis(1));+ g.refresh(0, wm * 97 / 100, 0);assert!(!g.memory_breaker_engaged(),"a stalled in-band RSS must release, not latch forever"@@ -3359,87 +3806,284 @@ mod tests {assert!(g.check_ingest_admission().is_ok());}- /// #1981: a footprint that sits flat at or above the watermark is not a- /// drain the sampler should repeat until restart. After- /// [`MEMORY_FLAT_RELEASE`] the latch releases, and the same RSS does not- /// re-engage (that flap would request a drain on every engage edge). A- /// higher RSS is new pressure and engages again. One flat sample is not- /// enough — the in-flight drain has not had its window.+ /// #2427: a drain that keeps lowering RSS through the band restarts the+ /// stall clock, so a slow but real drain is not cut short.#[test]- fn flat_rss_above_the_watermark_releases_and_does_not_rearm() {+ fn falling_rss_inside_the_band_restarts_the_stall_clock() {let mut cfg = Config::default();cfg.limits.max_total_memtable_mb = 0;cfg.limits.memory_watermark_percent = 95;let g = build(&cfg);let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);- assert!(wm > 0, "this test needs a live RSS watermark");- let stuck = wm + 100;+ assert!(+ wm > 64 * MEMORY_RSS_NOISE_BYTES,+ "this test needs a live RSS watermark"+ );- g.refresh(0, stuck, 0);- assert!(g.memory_breaker_engaged(), "crossing the watermark engages");- g.refresh(0, stuck, 0);+ g.refresh(0, wm, 0);+ assert!(g.memory_breaker_engaged());+ g.age_memory_engaged_for_test(g.memory_min_engaged() + Duration::from_millis(1));+ let mut rss = wm * 97 / 100;+ g.refresh(0, rss, 0);+ // Stall for most of the window, then fall past the noise band.+ g.age_memory_flat_for_test(MEMORY_BAND_STALL_RELEASE - Duration::from_millis(100));+ rss -= 2 * MEMORY_RSS_NOISE_BYTES;+ g.refresh(0, rss, 0);assert!(g.memory_breaker_engaged(),- "one flat sample must stay engaged so a drain can run"+ "a falling drain holds the latch");- g.age_memory_flat_for_test(MEMORY_FLAT_RELEASE + Duration::from_millis(1));- g.refresh(0, stuck, 0);+ // The clock restarted: the old window has not elapsed for the new+ // anchor, so another flat sample still holds.+ g.age_memory_flat_for_test(Duration::from_millis(200));+ g.refresh(0, rss, 0);assert!(- !g.memory_breaker_engaged(),- "RSS that does not fall must release instead of draining forever"+ g.memory_breaker_engaged(),+ "the stall clock must restart when RSS falls past noise");- assert!(g.check_ingest_admission().is_ok());+ }- // The old latch re-tripped here: rss >= watermark on a disengaged- // governor is an engage edge, and the sampler would drain again.- g.refresh(0, stuck, 0);- assert!(- !g.memory_breaker_engaged(),- "the same footprint must not re-arm the drain"+ /// #2427: the measured flap. Engage at the watermark, dip just under it,+ /// climb back: the latch must stay engaged through the whole cycle+ /// instead of releasing and re-engaging every 100 ms sample.+ #[test]+ fn breaker_does_not_flap_around_the_watermark() {+ let mut cfg = Config::default();+ cfg.limits.max_total_memtable_mb = 0;+ cfg.limits.memory_watermark_percent = 95;+ let g = build(&cfg);+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);+ assert!(wm > 0, "this test needs a live RSS watermark");++ let mut transitions = 0;+ let mut last = g.memory_breaker_engaged();+ for i in 0..200u64 {+ // 2,918 MB / 2,850 MB style oscillation: above, then 1-3% under.+ let rss = if i % 2 == 0 {+ wm + wm / 50+ } else {+ wm - wm / 100 * (1 + i % 3)+ };+ g.refresh(0, rss, 0);+ let now = g.memory_breaker_engaged();+ if now != last {+ transitions += 1;+ last = now;+ }+ }+ assert_eq!(+ transitions, 1,+ "only the first engage; no release/re-engage flap");- // Page-sized noise around the floor is the macOS signal, not a new- // working set. Re-arming on it would engage, drain, and release on- // every sample.- g.refresh(0, stuck + 1, 0);- assert!(- !g.memory_breaker_engaged(),- "a one-byte bump around the floor must not re-arm the drain"+ }++ #[test]+ fn release_ratio_and_min_engaged_parse_with_safe_defaults() {+ assert_eq!(+ resolve_memory_release_percent(None),+ DEFAULT_MEMORY_RELEASE_PERCENT+ );+ assert_eq!(+ resolve_memory_release_percent(Some("")),+ DEFAULT_MEMORY_RELEASE_PERCENT+ );+ assert_eq!(resolve_memory_release_percent(Some("90")), 90);+ assert_eq!(resolve_memory_release_percent(Some(" 80% ")), 80);+ assert_eq!(+ resolve_memory_release_percent(Some("100")),+ DEFAULT_MEMORY_RELEASE_PERCENT+ );+ assert_eq!(+ resolve_memory_release_percent(Some("10")),+ DEFAULT_MEMORY_RELEASE_PERCENT+ );+ assert_eq!(+ resolve_memory_release_percent(Some("x")),+ DEFAULT_MEMORY_RELEASE_PERCENT);- g.refresh(0, stuck + MEMORY_RSS_NOISE_BYTES + 1, 0);- assert!(- g.memory_breaker_engaged(),- "a climb past the noise band is new pressure"+ assert_eq!(resolve_memory_min_engaged(None), DEFAULT_MEMORY_MIN_ENGAGED);+ assert_eq!(resolve_memory_min_engaged(Some("0")), Duration::ZERO);+ assert_eq!(+ resolve_memory_min_engaged(Some("5000")),+ Duration::from_secs(5)+ );+ assert_eq!(+ resolve_memory_min_engaged(Some("999999999")),+ MAX_MEMORY_MIN_ENGAGED+ );+ assert_eq!(+ resolve_memory_min_engaged(Some("2s")),+ DEFAULT_MEMORY_MIN_ENGAGED+ );++ assert_eq!(memory_release_bytes(2_850, 85), 2_422);+ assert_eq!(memory_release_bytes(0, 85), 0);+ assert_eq!(+ memory_release_bytes(u64::MAX, 85),+ (u64::MAX as u128 * 85 / 100) as u64);}- /// #1981: `ri_phys_footprint` does not sit on one byte. A quiet node- /// whose samples wander inside [`MEMORY_RSS_NOISE_BYTES`] must still- /// release once that wander has lasted [`MEMORY_FLAT_RELEASE`].+ /// #2427: merges reserve their working set process-wide, and background+ /// merges wait while the breaker is engaged.#[test]- fn rss_noise_around_a_flat_footprint_still_releases() {+ fn merge_reservations_are_process_wide_and_respect_the_breaker() {let mut cfg = Config::default();cfg.limits.max_total_memtable_mb = 0;cfg.limits.memory_watermark_percent = 95;let g = build(&cfg);let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);- assert!(wm > MEMORY_RSS_NOISE_BYTES);- let stuck = wm + MEMORY_RSS_NOISE_BYTES;+ assert!(wm > 0, "this test needs a live RSS watermark");- g.refresh(0, stuck, 0);+ // RSS at half the watermark. The first merge always starts.+ g.refresh(0, wm / 2 + 1, 0);+ // Each estimate is a little over a quarter of the watermark.+ let per = wm / 4 / crate::merge_budget::MERGE_WORKING_SET_NUMERATOR + 1;+ let first = g.try_reserve_merge(per, false).expect("first merge starts");+ assert_eq!(g.merges_in_flight(), 1);+ assert_eq!(g.merge_reserved_bytes(), first.bytes());+ // 50% + 25% + 25% reaches the watermark: the second must wait.+ assert!(g.try_reserve_merge(per, false).is_none());+ // A smaller one still fits.+ let second = g.try_reserve_merge(per / 4, false).expect("fits");+ assert_eq!(g.merges_in_flight(), 2);+ drop(first);+ drop(second);+ assert_eq!(g.merges_in_flight(), 0);+ assert_eq!(g.merge_reserved_bytes(), 0);++ // Breaker engaged: no background merge, a forced one may run alone.+ g.refresh(0, wm, 0);assert!(g.memory_breaker_engaged());- g.refresh(0, stuck + 4096, 0);- assert!(g.memory_breaker_engaged(), "noise must not release early");- g.refresh(0, stuck - 4096, 0);+ assert!(g.try_reserve_merge(1, false).is_none());+ let forced = g+ .try_reserve_merge(1, true)+ .expect("forcemerge progresses alone");+ assert!(g.try_reserve_merge(1, true).is_none());+ drop(forced);+ }++ /// #2427: on the 1 GB gate an admitted merge batch grew a 3,072 MB node+ /// to 8,122 MB while the breaker was engaged. A background batch yields+ /// once RSS reaches the process limit with the breaker engaged, never in+ /// the watermark band, never when forced; the yield narrows later batches+ /// and completed batches widen them back.+ #[test]+ fn a_background_merge_yields_at_the_limit_and_narrows_later_batches() {+ const MIB: u64 = 1024 * 1024;+ let mut cfg = Config::default();+ cfg.limits.max_total_memtable_mb = 0;+ cfg.limits.memory_watermark_percent = 95;+ let g = build(&cfg);+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);+ let limit = g.process_memory_limit_bytes();+ assert!(wm > 0 && limit > wm, "this test needs a live RSS watermark");++ // Engaged inside the watermark band: keep merging.+ g.refresh(0, wm, 0);assert!(g.memory_breaker_engaged());- g.age_memory_flat_for_test(MEMORY_FLAT_RELEASE + Duration::from_millis(1));- g.refresh(0, stuck + 8192, 0);+ assert!(!g.merge_must_yield(false));+ // Engaged at the limit: a background batch yields, a forced one not.+ g.refresh(0, limit, 0);+ assert!(g.memory_breaker_engaged());+ assert!(g.merge_must_yield(false));+ assert!(!g.merge_must_yield(true));++ assert_eq!(g.merge_pressure_cap_bytes(), 0);+ assert_eq!(g.note_merge_yield(66 * MIB), 33 * MIB);+ assert_eq!(g.merge_pressure_cap_bytes(), 33 * MIB);+ g.note_merge_completed(MIB);+ assert_eq!(g.merge_pressure_cap_bytes(), 33 * MIB);+ g.note_merge_completed(33 * MIB);+ assert_eq!(g.merge_pressure_cap_bytes(), 66 * MIB);+ }++ /// #2427: the 1 GB release gate released at rss=5,121MB against a+ /// 2,918MB watermark (the old #1981 flat-footprint release), re-engaged+ /// 100 ms later at 5,125MB, and was OOM-killed at 8.4 GB. A footprint at+ /// or above the watermark must hold the latch however long it sits flat,+ /// whatever page-sized noise it shows, and however long the breaker has+ /// been engaged.+ #[test]+ fn flat_rss_above_the_watermark_never_releases() {+ let mut cfg = Config::default();+ cfg.limits.max_total_memtable_mb = 0;+ cfg.limits.memory_watermark_percent = 95;+ let g = build(&cfg);+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);+ assert!(+ wm > MEMORY_RSS_NOISE_BYTES,+ "this test needs a live RSS watermark"+ );+ let stuck = wm + wm / 4;++ g.refresh(0, stuck, 0);+ assert!(g.memory_breaker_engaged(), "crossing the watermark engages");+ for i in 0..200u64 {+ g.age_memory_flat_for_test(Duration::from_secs(3600));+ g.age_memory_engaged_for_test(Duration::from_secs(3600));+ let rss = match i % 4 {+ 0 => stuck,+ 1 => stuck + 4096,+ 2 => stuck - 8192,+ _ => wm,+ };+ g.refresh(0, rss, 0);+ assert!(+ g.memory_breaker_engaged(),+ "rss={rss} >= watermark={wm} must never release (sample {i})"+ );+ assert!(g.check_ingest_admission().is_err());+ }+ g.refresh(0, wm - 1, 0);+ assert!(+ g.memory_breaker_engaged(),+ "one byte under the watermark is inside the release band"+ );+ g.refresh(0, wm / 2, 0);assert!(!g.memory_breaker_engaged(),- "page-sized wander is a flat footprint"+ "under the release point releases");- g.refresh(0, stuck - 8192, 0);- assert!(!g.memory_breaker_engaged(), "the same band must not re-arm");+ }++ /// #2427: the latch's release decision is a pure function of the+ /// sample, so walk a long random-ish RSS trace (the gate's 2,750-5,125MB+ /// shape scaled to the watermark) and check every release lands strictly+ /// under the watermark and every engage edge at or above it.+ #[test]+ fn no_release_edge_at_or_above_the_watermark_on_a_noisy_trace() {+ let mut cfg = Config::default();+ cfg.limits.max_total_memtable_mb = 0;+ cfg.limits.memory_watermark_percent = 95;+ let g = build(&cfg);+ let wm = g.memory_watermark_bytes.load(Ordering::Relaxed);+ assert!(wm > 0, "this test needs a live RSS watermark");+ let mut x: u64 = 0x2427;+ let mut was = false;+ let mut releases = 0;+ for _ in 0..5_000 {+ // xorshift: deterministic, no rand dependency.+ x ^= x << 13;+ x ^= x >> 7;+ x ^= x << 17;+ // 80%..180% of the watermark.+ let rss = wm / 100 * (80 + x % 101);+ g.age_memory_flat_for_test(Duration::from_secs(60));+ g.age_memory_engaged_for_test(Duration::from_secs(60));+ g.refresh(0, rss, 0);+ let now = g.memory_breaker_engaged();+ if was && !now {+ releases += 1;+ assert!(rss < wm, "released at rss={rss} >= watermark={wm}");+ }+ if !was && now {+ assert!(rss >= wm, "engaged at rss={rss} < watermark={wm}");+ }+ was = now;+ }+ assert!(releases > 0, "the trace should exercise releases");}/// #1981: while the breaker stays engaged and RSS does not fall, drain@@ -3498,9 +4142,42 @@ mod tests {assert!(schedule.observe(now + min + min, dropped, true));}+ /// #2427: the latch holds above the watermark now, so a climbing+ /// footprint must not wait out a backed-off gap. Back the schedule off+ /// on a flat footprint, then climb: the next sample past the minimum+ /// spacing drains, and a further flat sample is back on the floor.+ #[test]+ fn a_climb_while_engaged_drains_inside_a_backed_off_gap() {+ let min = Duration::from_secs(2);+ let mut schedule = MemoryDrainSchedule::new(min);+ let t0 = Instant::now();+ let stuck = 64 * MEMORY_RSS_NOISE_BYTES;+ let mut t = Duration::ZERO;+ while t < Duration::from_secs(600) {+ schedule.observe(t0 + t, stuck, true);+ t += min;+ }+ assert!(schedule.interval > min, "a flat footprint backs off");+ let last = schedule.last_request.expect("requested at least once");+ let climbed = stuck + MEMORY_RSS_NOISE_BYTES + 1;+ assert!(+ !schedule.observe(last + min / 2, climbed, true),+ "the minimum spacing still applies to a climb"+ );+ assert!(+ schedule.observe(last + min, climbed, true),+ "a climb drains at the minimum spacing, not after the backed-off gap"+ );+ assert_eq!(schedule.interval, min);+ assert!(+ !schedule.observe(last + min + Duration::from_millis(100), climbed, true),+ "one climb is one request"+ );+ }+/// #1250 x #1323: the watermark is live-tunable. Raising it while the/// breaker is engaged must release at once whenever RSS is under the NEW- /// watermark — not wait for RSS to fall 5% below it — or the live fix+ /// watermark — not wait for RSS to fall below the release point — or the live fix/// #1323 shipped would be undone by the band.#[test]fn live_watermark_raise_releases_without_the_hysteresis_band() {@@ -3516,7 +4193,7 @@ mod tests {assert!(g.check_ingest_admission().is_err());// Raise just above the current RSS: RSS is now inside the new- // watermark's 5% band, and still (barely) falling.+ // watermark's release band, and still (barely) falling.let raised = (rss + rss / 50).min(g.machine_memory_bytes);assert!(raised > rss, "test box must have headroom over 1 GiB");g.set_memory_watermark_override(Some(raised));diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 40390de23..cd6642ffc 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -13798,7 +13798,21 @@ impl Index {/// 16 GiB host does not merge in smaller pieces. A 4 GiB budget/// (the AUTO cap on an 8 GiB host) is 64 MiB./// `XERJ_MERGE_BATCH_MAX_INPUT_MB` overrides (0 disables the bound).+ ///+ /// #2427: a batch that had to yield under memory pressure narrows this+ /// for every index through the governor's adaptive cap+ /// (`merge_budget::effective_batch_cap_bytes`).fn merge_batch_cap_bytes() -> u64 {+ let pressure_cap = crate::governor::global()+ .map(|governor| governor.merge_pressure_cap_bytes())+ .unwrap_or(0);+ crate::merge_budget::effective_batch_cap_bytes(+ Self::merge_static_batch_cap_bytes(),+ pressure_cap,+ )+ }++ fn merge_static_batch_cap_bytes() -> u64 {const MB: u64 = 1024 * 1024;if let Ok(raw) = std::env::var("XERJ_MERGE_BATCH_MAX_INPUT_MB") {if let Ok(mb) = raw.trim().parse::<u64>() {@@ -14087,9 +14101,31 @@ impl Index {// pass returns an ERROR so callers (forcemerge HTTP → 500, background// task → error log) hear about it loudly instead of losing data.let failed_batches = Arc::new(std::sync::atomic::AtomicUsize::new(0));+ // #2427 — background batches that stopped because the breaker was+ // engaged with RSS at the process limit. Not a failure: inputs stay,+ // the pass is flagged deferred, and the governor narrows the next+ // batches (`merge_budget::merge_must_yield`).+ let yielded_batches = Arc::new(std::sync::atomic::AtomicUsize::new(0));++ // #2427 — process-wide admission. Every launch reserves the batch's+ // estimated working set against the RSS watermark through the+ // governor's merge ledger, so concurrent merges across indices see+ // each other before they allocate, and background merges wait while+ // the breaker is engaged. `None` = defer; `Some(None)` = no governor.+ let forced_merge = force_max_segments.is_some();+ let admit_merge = |metas: &[xerj_storage::segment::SegmentMeta]|+ -> Option<Option<crate::governor::MergeReservation>> {+ let Some(governor) = crate::governor::global() else {+ return Some(None);+ };+ let input: u64 = metas.iter().map(|meta| meta.size_bytes).sum();+ governor.try_reserve_merge(input, forced_merge).map(Some)+ };+ let mut governor_deferred = 0usize;let spawn_one = |batch: Vec<xerj_storage::segment::SegmentId>,- metas: Vec<xerj_storage::segment::SegmentMeta>|+ metas: Vec<xerj_storage::segment::SegmentMeta>,+ reservation: Option<crate::governor::MergeReservation>|-> JoinHandle<Option<MergeOutput>> {let store_for_task = Arc::clone(&self.store);let registry_for_task = Arc::clone(&self.registry);@@ -14121,6 +14157,30 @@ impl Index {let batch_for_task = batch;let metas_for_task = metas;let failed_for_task = Arc::clone(&failed_batches);+ let yielded_for_task = Arc::clone(&yielded_batches);+ let forced_for_task = force_max_segments.is_some();+ let input_bytes_for_task: u64 = metas_for_task.iter().map(|meta| meta.size_bytes).sum();+ // Checked at each input segment, every 8192 surviving docs, and+ // before the output segment is created — never after, so a+ // yielded batch leaves no partial output behind.+ let must_yield = move || -> bool {+ let Some(governor) = crate::governor::global() else {+ return false;+ };+ if !governor.merge_must_yield(forced_for_task) {+ return false;+ }+ let cap = governor.note_merge_yield(input_bytes_for_task);+ tracing::warn!(+ input_mb = input_bytes_for_task / (1024 * 1024),+ rss_mb = governor.observed_rss_bytes() / (1024 * 1024),+ limit_mb = governor.process_memory_limit_bytes() / (1024 * 1024),+ next_batch_cap_mb = cap / (1024 * 1024),+ "merge batch yielded: breaker engaged with RSS at the process limit; \+ inputs kept, later batches narrowed"+ );+ true+ };#[cfg(test)]let hold_merge_encode = Arc::clone(&self.test_hold_merge_encode);#[cfg(test)]@@ -14338,6 +14398,10 @@ impl Index {let mut tomb_union: HashMap<String, u64> = HashMap::new();for (source_index, meta) in metas_for_task.iter().enumerate() {+ if must_yield() {+ yielded_for_task.fetch_add(1, Ordering::Relaxed);+ return None;+ }min_seq = min_seq.min(meta.min_seq_no);max_seq = max_seq.max(meta.max_seq_no);let seg_path = segments_dir_for_task.join(&meta.seg_path);@@ -14664,6 +14728,10 @@ impl Index {// stored-side peak.drain_checkpoint += 1;if drain_checkpoint.is_multiple_of(8192) {+ if must_yield() {+ yielded_for_task.fetch_add(1, Ordering::Relaxed);+ return None;+ }crate::ingest_memory::observe_checkpoint(crate::ingest_memory::Category::MergeJsonBuffer,merged_json_buf.len(),@@ -14699,6 +14767,10 @@ impl Index {fts_input_bytes,);+ if must_yield() {+ yielded_for_task.fetch_add(1, Ordering::Relaxed);+ return None;+ }// Write merged stored section using the columnar v2 codec.let mut writer = match SegmentWriter::new(&segments_dir_for_task, 1, 0, 0) {Ok(w) => w,@@ -14998,8 +15070,18 @@ impl Index {let _ = encode_tx.send(encoded);});tokio::spawn(async move {+ // Held until the encode (the allocation-heavy part: stored+ // decode, FST and doc-values build) has finished.+ let _reservation = reservation;match encode_rx.await {- Ok(Ok(value)) => value,+ Ok(Ok(value)) => {+ if value.is_some() && !forced_for_task {+ if let Some(governor) = crate::governor::global() {+ governor.note_merge_completed(input_bytes_for_task);+ }+ }+ value+ }Ok(Err(payload)) => std::panic::resume_unwind(payload),// The encode task vanished without a result. Panic so the// driver counts a failed batch (the `JoinError` arm)@@ -15010,15 +15092,33 @@ impl Index {})};+ // Launch the pending batch if the governor admits it (#2427). A+ // refusal leaves it and every later batch of this pass unmerged and+ // flags the pass deferred, so the scheduler retries after the drain.+ macro_rules! launch_next {+ () => {+ match pending.take() {+ Some((batch, metas)) => match admit_merge(&metas) {+ Some(reservation) => {+ in_flight.push(spawn_one(batch, metas, reservation));+ pending = queue_iter.next();+ true+ }+ None => {+ governor_deferred += 1 + queue_iter.by_ref().count();+ false+ }+ },+ None => false,+ }+ };+ }+// Fill in_flight up to the cap, then drain one-by-one as they// complete, launching replacements from `queue_iter`.while in_flight.len() < merge_parallelism {- match pending.take() {- Some((batch, metas)) => {- in_flight.push(spawn_one(batch, metas));- pending = queue_iter.next();- }- None => break,+ if !launch_next!() {+ break;}}while !in_flight.is_empty() {@@ -15225,10 +15325,7 @@ impl Index {// `continue` bypasses the common top-up below. Launch// the next disjoint batch first so one failed apply// does not silently suppress the rest of this pass.- if let Some((batch, metas)) = pending.take() {- in_flight.push(spawn_one(batch, metas));- pending = queue_iter.next();- }+ launch_next!();continue;}// `apply_merge` has made the output authoritative. No@@ -15290,10 +15387,7 @@ impl Index {// Same pipeline rule as the failed-apply path: the// `continue` below bypasses the loop's bottom// top-up, so launch the next disjoint batch here.- if let Some((batch, metas)) = pending.take() {- in_flight.push(spawn_one(batch, metas));- pending = queue_iter.next();- }+ launch_next!();continue;}// Disk-space fix (2026-07): the input segments are@@ -15359,9 +15453,32 @@ impl Index {}}// Top up in_flight.- if let Some((batch, metas)) = pending.take() {- in_flight.push(spawn_one(batch, metas));- pending = queue_iter.next();+ launch_next!();+ }+ let yielded = yielded_batches.load(AtomicOrdering::Relaxed);+ if yielded > 0 {+ self.merge_batches_deferred+ .store(true, AtomicOrdering::Relaxed);+ tracing::info!(+ index = %self.name,+ yielded,+ "merge batches yielded under memory pressure; inputs kept"+ );+ }+ if governor_deferred > 0 {+ self.merge_batches_deferred+ .store(true, AtomicOrdering::Relaxed);+ if let Some(governor) = crate::governor::global() {+ tracing::info!(+ index = %self.name,+ deferred = governor_deferred,+ rss_mb = governor.observed_rss_bytes() / (1024 * 1024),+ watermark_mb = governor.memory_watermark_bytes() / (1024 * 1024),+ merge_reserved_mb = governor.merge_reserved_bytes() / (1024 * 1024),+ merges_in_flight = governor.merges_in_flight(),+ breaker_engaged = governor.memory_breaker_engaged(),+ "merge batches deferred by the memory governor; inputs kept"+ );}}@@ -35918,6 +36035,30 @@ async fn do_flush_shard(}}};+ // #2427 — while the parent RSS breaker is engaged, finalizes run one at+ // a time (see `flush_pressure_gate`). Same deadline and same+ // before-the-freeze failure as the finalize gate above.+ let pressure_permit = if !bypass_finalize_gate+ && crate::governor::global().is_some_and(|g| g.memory_breaker_engaged())+ {+ match wait_for_flush_permit(+ &index_name,+ "the memory-pressure flush gate",+ admit_timeout,+ crate::flush_pressure_gate().acquire(),+ )+ .await+ {+ Ok(permit) => permit,+ Err(error) => {+ drop(fin_permit);+ on_drained();+ return Err(error);+ }+ }+ } else {+ None+ };// #1015 — NO collection-publication guard here. The drain below is a// FREEZE, not a removal: the claimed docs stay reader-visible in the// shard's frozen generations for the whole segment build, so there is@@ -36312,6 +36453,7 @@ async fn do_flush_shard(}let joined = tokio::task::spawn_blocking(move || {let _fin_permit = fin_permit;+ let _pressure_permit = pressure_permit;// This lease lives in the blocking worker, not either async waiter.// It spans build→publish exactly as before: it tracks the caches// `build_fts` warms so a failed publication can evict them.diff --git a/engine/crates/xerj-engine/src/lib.rs b/engine/crates/xerj-engine/src/lib.rsindex 9159caf93..3b93c1a44 100644--- a/engine/crates/xerj-engine/src/lib.rs+++ b/engine/crates/xerj-engine/src/lib.rs@@ -267,6 +267,22 @@ pub(crate) fn flush_finalize_gate() -> &'static tokio::sync::Semaphore {GATE.get_or_init(|| tokio::sync::Semaphore::new(crate::pools::sizing().flush_finalize))}+/// One permit: while the parent RSS breaker is engaged, flush finalizes+/// (segment build, FST build, doc-values encode) also take this, so only one+/// runs at a time (#2427).+///+/// [`flush_finalize_gate`] allows `max(2, ncores/8)` concurrent finalizes.+/// Each holds a frozen memtable plus its FST and column build buffers. Under+/// the breaker, those buffers stacked on a node that was already at the+/// watermark: RSS climbed about 1 GB past a 3,000 MB cap while writes were+/// already getting 429s. Serializing them while the breaker is engaged bounds+/// the overshoot to one build. The drain still makes progress; it just does+/// not multiply the peak.+pub(crate) fn flush_pressure_gate() -> &'static tokio::sync::Semaphore {+ static GATE: std::sync::OnceLock<tokio::sync::Semaphore> = std::sync::OnceLock::new();+ GATE.get_or_init(|| tokio::sync::Semaphore::new(1))+}+// ── Analyzer helpers ──────────────────────────────────────────────────────────use std::sync::Arc;diff --git a/engine/crates/xerj-engine/src/merge_budget.rs b/engine/crates/xerj-engine/src/merge_budget.rsindex 7a2d73c2f..81b34b0b3 100644--- a/engine/crates/xerj-engine/src/merge_budget.rs+++ b/engine/crates/xerj-engine/src/merge_budget.rs@@ -125,10 +125,171 @@ pub fn throttle_merge_batches(start}+/// Estimated peak working set of a merge batch whose inputs total+/// `input_bytes`: the same `input * MERGE_WORKING_SET_NUMERATOR` that+/// [`decide_merge_batch`] uses.+pub fn merge_working_set_bytes(input_bytes: u64) -> u64 {+ input_bytes.saturating_mul(MERGE_WORKING_SET_NUMERATOR)+}++/// Process-wide merge admission against the RSS watermark (#2427).+///+/// [`throttle_merge_batches`] plans one index's pass from an RSS snapshot+/// and always lets that index's first batch start. With hundreds of indices+/// (autoindex) every index started one, and none of them saw the others'+/// working sets until they had allocated. This is checked under the+/// governor's merge ledger at launch time instead:+///+/// * `watermark == 0` (breaker off) or `observed_rss == 0` (no probe):+/// start.+/// * No merge in flight anywhere in the process: start one, so merging can+/// always make progress, unless the breaker is engaged and the merge is a+/// background one. A `forced` merge (`_forcemerge`) is still allowed as+/// the single in-flight merge while engaged; it is a deliberate operator+/// request and it waits on its own.+/// * Breaker engaged: defer. Background merges are the one memory consumer+/// the node can postpone, so they wait for the drain.+/// * Otherwise start only if `observed_rss + reserved + request` stays under+/// the watermark.+pub fn decide_merge_admission(+ watermark: u64,+ observed_rss: u64,+ reserved: u64,+ in_flight: u64,+ breaker_engaged: bool,+ request: u64,+ forced: bool,+) -> bool {+ if watermark == 0 || observed_rss == 0 {+ return true;+ }+ if in_flight == 0 {+ return !breaker_engaged || forced;+ }+ if breaker_engaged {+ return false;+ }+ observed_rss+ .saturating_add(reserved)+ .saturating_add(request)+ < watermark+}++/// Whether an in-flight background merge batch must stop (#2427).+///+/// Admission checks a batch against `input * MERGE_WORKING_SET_NUMERATOR`+/// before it starts. On the 1 GB gate corpus that estimate was several+/// times too small: JSON-heavy segments compress 10–20×, and a merge holds+/// the decoded survivors plus a parsed `Value` per document. One admitted+/// batch took a 3,072 MB node to 8,122 MB in 18 s while the breaker was+/// engaged and writes were already getting 429s. Nothing else could free+/// that memory, because the merge was the thing allocating it.+///+/// A batch stops when the breaker is engaged and RSS has reached the full+/// process limit (past the 95% watermark). A stopped batch commits nothing+/// and keeps its inputs, like any aborted merge. `forced` (`_forcemerge`)+/// batches are not stopped. `limit == 0` or `rss == 0` (unknown) never+/// stops a batch.+pub fn merge_must_yield(rss: u64, limit: u64, breaker_engaged: bool, forced: bool) -> bool {+ !forced && breaker_engaged && limit > 0 && rss > 0 && rss >= limit+}++/// The adaptive batch input cap after a batch of `input_bytes` stopped+/// under memory pressure (#2427). `current == 0` means no adaptive cap yet.+/// The next batch may use at most half of what did not fit. Never below 1.+pub fn pressure_cap_after_yield(current: u64, input_bytes: u64) -> u64 {+ let halved = (input_bytes / 2).max(1);+ if current == 0 {+ halved+ } else {+ current.min(halved)+ }+}++/// The adaptive batch input cap after a background batch of `input_bytes`+/// completed (#2427). A batch that used at least half of the cap doubles+/// it. Once the cap reaches [`HISTORICAL_BATCH_CAP_BYTES`] it is removed+/// (`0`), and the static cap applies again.+pub fn pressure_cap_after_success(current: u64, input_bytes: u64) -> u64 {+ if current == 0 {+ return 0;+ }+ if input_bytes.saturating_mul(2) < current {+ return current;+ }+ let doubled = current.saturating_mul(2);+ if doubled >= HISTORICAL_BATCH_CAP_BYTES {+ 0+ } else {+ doubled+ }+}++/// The batch input cap to plan with: the static cap, narrowed by the+/// adaptive pressure cap when one is set. A static cap of `0` (the+/// `XERJ_MERGE_BATCH_MAX_INPUT_MB=0` opt-out) still takes a pressure cap.+pub fn effective_batch_cap_bytes(static_cap: u64, pressure_cap: u64) -> u64 {+ match (static_cap, pressure_cap) {+ (s, 0) => s,+ (0, p) => p,+ (s, p) => s.min(p),+ }+}+#[cfg(test)]mod tests {use super::*;+ #[test]+ fn a_background_merge_yields_only_when_engaged_at_the_limit() {+ const MIB: u64 = 1024 * 1024;+ let limit = 1024 * MIB;+ assert!(merge_must_yield(1024 * MIB, limit, true, false));+ assert!(merge_must_yield(1600 * MIB, limit, true, false));+ // Watermark band (95%..100%) and below: keep going.+ assert!(!merge_must_yield(1000 * MIB, limit, true, false));+ // Not engaged: keep going (the sampler will engage first).+ assert!(!merge_must_yield(1600 * MIB, limit, false, false));+ // `_forcemerge` is never stopped.+ assert!(!merge_must_yield(1600 * MIB, limit, true, true));+ // Unknown limit or RSS.+ assert!(!merge_must_yield(1600 * MIB, 0, true, false));+ assert!(!merge_must_yield(0, limit, true, false));+ }++ #[test]+ fn the_pressure_cap_halves_on_yield_and_recovers_on_success() {+ const MIB: u64 = 1024 * 1024;+ let cap = pressure_cap_after_yield(0, 66 * MIB);+ assert_eq!(cap, 33 * MIB);+ // A second, larger yield never widens it.+ assert_eq!(pressure_cap_after_yield(cap, 200 * MIB), 33 * MIB);+ let cap = pressure_cap_after_yield(cap, 30 * MIB);+ assert_eq!(cap, 15 * MIB);+ // A tiny batch's success does not widen it.+ assert_eq!(pressure_cap_after_success(cap, MIB), cap);+ // Batches that use the cap double it until it is removed.+ let mut cap = cap;+ let mut rounds = 0;+ while cap != 0 {+ cap = pressure_cap_after_success(cap, cap);+ rounds += 1;+ assert!(rounds < 16);+ }+ assert_eq!(pressure_cap_after_success(0, 512 * MIB), 0);+ assert_eq!(pressure_cap_after_yield(0, 0), 1);+ }++ #[test]+ fn the_pressure_cap_narrows_the_static_cap() {+ const MIB: u64 = 1024 * 1024;+ assert_eq!(effective_batch_cap_bytes(32 * MIB, 0), 32 * MIB);+ assert_eq!(effective_batch_cap_bytes(32 * MIB, 8 * MIB), 8 * MIB);+ assert_eq!(effective_batch_cap_bytes(32 * MIB, 64 * MIB), 32 * MIB);+ assert_eq!(effective_batch_cap_bytes(0, 8 * MIB), 8 * MIB);+ assert_eq!(effective_batch_cap_bytes(0, 0), 0);+ }+const GIB: u64 = 1024 * 1024 * 1024;const MIB: u64 = 1024 * 1024;@@ -202,4 +363,129 @@ mod tests {assert_eq!(merge_batch_input_cap_bytes(GIB), MIN_BATCH_CAP_BYTES);assert_eq!(merge_batch_input_cap_bytes(2 * GIB), MIN_BATCH_CAP_BYTES);}++ #[test]+ fn merge_admission_is_open_when_the_breaker_is_off_or_rss_is_unknown() {+ assert!(decide_merge_admission(+ 0,+ 10_000 * MIB,+ 0,+ 5,+ true,+ 4_000 * MIB,+ false+ ));+ assert!(decide_merge_admission(+ 2_850 * MIB,+ 0,+ 0,+ 5,+ true,+ 4_000 * MIB,+ false+ ));+ }++ #[test]+ fn merge_admission_always_lets_one_merge_run_when_not_engaged() {+ // Even an estimate that alone exceeds the watermark starts when no+ // other merge is in flight: merging must make progress.+ assert!(decide_merge_admission(+ 2_850 * MIB,+ 2_700 * MIB,+ 0,+ 0,+ false,+ 400 * MIB,+ false+ ));+ }++ #[test]+ fn merge_admission_counts_other_in_flight_reservations() {+ let wm = 2_850 * MIB;+ // One 47 MiB batch (376 MiB working set) already reserved.+ let reserved = merge_working_set_bytes(47 * MIB);+ assert!(decide_merge_admission(+ wm,+ 2_000 * MIB,+ reserved,+ 1,+ false,+ reserved,+ false+ ));+ // At 2,200 MiB RSS a second one would cross the watermark.+ assert!(!decide_merge_admission(+ wm,+ 2_200 * MIB,+ reserved,+ 1,+ false,+ reserved,+ false+ ));+ // And a third does not fit at all once two are reserved.+ assert!(!decide_merge_admission(+ wm,+ 2_000 * MIB,+ 2 * reserved,+ 2,+ false,+ reserved,+ false+ ));+ }++ #[test]+ fn merge_admission_defers_background_merges_while_engaged() {+ let wm = 2_850 * MIB;+ // Engaged: no background merge starts, not even the first.+ assert!(!decide_merge_admission(+ wm,+ 2_900 * MIB,+ 0,+ 0,+ true,+ 8 * MIB,+ false+ ));+ assert!(!decide_merge_admission(+ wm,+ 1_000 * MIB,+ 0,+ 1,+ true,+ 8 * MIB,+ false+ ));+ // A forced merge may run alone while engaged, but not beside another.+ assert!(decide_merge_admission(+ wm,+ 2_900 * MIB,+ 0,+ 0,+ true,+ 8 * MIB,+ true+ ));+ assert!(!decide_merge_admission(+ wm,+ 2_900 * MIB,+ 64 * MIB,+ 1,+ true,+ 8 * MIB,+ true+ ));+ }++ #[test]+ fn merge_working_set_matches_the_batch_decision() {+ assert_eq!(+ merge_working_set_bytes(10),+ 10 * MERGE_WORKING_SET_NUMERATOR+ );+ assert_eq!(merge_working_set_bytes(u64::MAX), u64::MAX);+ }}diff --git a/landing/docs/env.html b/landing/docs/env.htmlindex 6c71c4389..8a363b621 100644--- a/landing/docs/env.html+++ b/landing/docs/env.html@@ -286,6 +286,14 @@<div class="cell d">—</div><div class="cell dsc">Segment hydration cache budget, same grammar. Wins over <code class="inline">limits.max_segment_hydration_cache_mb</code>; clamped to 50% of memory.</div>+ <div class="cell k">XERJ_MEMORY_BREAKER_RELEASE_PCT</div>+ <div class="cell d">85</div>+ <div class="cell dsc">Percent of the memory watermark RSS must fall below before the parent memory breaker releases (50–99).</div>++ <div class="cell k">XERJ_MEMORY_BREAKER_MIN_ENGAGED_MS</div>+ <div class="cell d">2000</div>+ <div class="cell dsc">Minimum time the parent memory breaker stays engaged before it releases below the watermark; <code class="inline">0</code> turns it off, capped at 60000.</div>+<div class="cell k">XERJ_MERGE_PARALLELISM</div><div class="cell d">1</div><div class="cell dsc">Concurrent merges (<code class="inline">merge.max_concurrent</code> is not wired).</div>