AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(governor): count only memory the node can release on Linux (#2574) (#2656)
xerj-org8ba8c2e6bc02
diff --git a/docs/recipes/production-deployment.md b/docs/recipes/production-deployment.mdindex 4a1d16d81..9cb2cd65a 100644--- a/docs/recipes/production-deployment.md+++ b/docs/recipes/production-deployment.md@@ -396,6 +396,32 @@ transport encryption; your platform provides the rest.| Secrets written `0600` (admin key, auto-generated TLS key) | ✅ || Client identity from the TCP peer, not `X-Forwarded-For` | ✅ (`server.trusted_proxies`, empty by default) |+### Memory limits and what "process memory" means++The memory breaker compares the node's own memory against one cap. These+`[limits]` keys set it (all MiB, `0` = automatic):++| Key | Env override | Default | What it does |+|---|---|---|---|+| `max_process_memory_mb` | `XERJ_MAX_PROCESS_MEMORY_MB` | `0` (tier from total RAM or the cgroup limit) | The cap. Writes get `429` above the watermark below; searches get `429` only after waiting for a drain that did not bring memory under the cap. |+| `memory_watermark_percent` | – | `95` | Percent of the cap where writes start to get `429`. |+| `max_total_memtable_mb` | – | `0` (25% of the limit, floor 2 GiB, at most 50%) | Unflushed write buffers across all indices. |+| `max_segment_hydration_cache_mb` | `XERJ_SEGMENT_HYDRATION_CACHE_MB` | `0` (20% of the limit) | See the next section. |++"Process memory" (`rss_mb` in breaker log lines) is memory the node itself+allocated and that only it can give back. On Linux that is `RssAnon ++RssShmem` from `/proc/self/status`, not the `RSS` column `top` shows: segment+files are memory-mapped, and the pages of those files a search reads stay in+the process's RSS as page cache until the kernel needs the RAM back. The node+cannot release those pages itself, so counting them made a node whose+mapped segments were larger than its cap answer every search with `429`+until it was restarted (#2574). Expect `top` to show more than `rss_mb`.++The automatic cap is sized from total RAM, not from what is free. If other+processes already hold most of the RAM, the startup log adds a+`memory: only … is available right now` warning. In that case set+`max_process_memory_mb` to fit what is left.+### Bound retained segment hydrationEvery index shares one process-wide admission authority for the sevendiff --git a/engine/crates/xerj-common/src/resource.rs b/engine/crates/xerj-common/src/resource.rsindex 937506112..24bb2c27d 100644--- a/engine/crates/xerj-common/src/resource.rs+++ b/engine/crates/xerj-common/src/resource.rs@@ -402,9 +402,21 @@ pub fn available_memory_bytes() -> Option<u64> {probe_available_memory()}-/// Resident set size of this process (on macOS its physical footprint, see-/// the macOS `probe_rss`), or `None` where this platform has no probe.-/// `None` must disable an RSS-driven decision, never read as zero.+/// Memory this process holds that the kernel cannot drop on its own, or+/// `None` where this platform has no probe. On Linux that is the anonymous+/// and shared-memory resident pages (`RssAnon + RssShmem`), not `VmRSS`; on+/// macOS the physical footprint (see the macOS `probe_rss`); on Windows the+/// working set. `None` must disable an RSS-driven decision, never read as+/// zero.+///+/// Why not `VmRSS` on Linux (#2574): it also counts clean file-backed pages+/// of every mmapped segment, term dictionary and postings file a search has+/// touched. The kernel reclaims those whenever another process needs RAM,+/// so they cannot exhaust memory, but no drain inside the process can+/// release them while the mapping is open. On a node whose indices are+/// larger than its memory cap, searches faulted those pages in until RSS sat+/// above the cap, and from then on every search was rejected with+/// `circuit_breaking_exception` until the node restarted.pub fn current_rss_bytes() -> Option<u64> {probe_rss()}@@ -554,14 +566,45 @@ fn meminfo_field(meminfo: &str, key: &str) -> Option<u64> {None}+/// Linux: anonymous plus shared-memory resident bytes from+/// `/proc/self/status` (#2574). Clean file-backed pages (mmapped segments)+/// are left out because the kernel drops them under pressure; see+/// [`current_rss_bytes`]. Kernels older than 4.5 have no `RssAnon` line, so+/// fall back to `/proc/self/statm` resident minus its shared (file-backed and+/// shmem) pages, which is the same anonymous figure without the shmem part.#[cfg(target_os = "linux")]fn probe_rss() -> Option<u64> {+ if let Some(bytes) = std::fs::read_to_string("/proc/self/status")+ .ok()+ .and_then(|status| unreclaimable_resident_bytes(&status))+ {+ return Some(bytes);+ }let statm = std::fs::read_to_string("/proc/self/statm").ok()?;- let pages: u64 = statm.split_whitespace().nth(1)?.parse().ok()?;// SAFETY: `sysconf` reads a system constant.let page = unsafe { libc::sysconf(libc::_SC_PAGESIZE) };let page = if page > 0 { page as u64 } else { 4096 };- Some(pages.saturating_mul(page))+ statm_anonymous_bytes(&statm, page)+}++/// `RssAnon + RssShmem` out of a `/proc/<pid>/status` body, in bytes. `None`+/// when the kernel has no `RssAnon` line (pre-4.5). A missing `RssShmem`+/// counts as zero.+#[cfg(any(target_os = "linux", test))]+fn unreclaimable_resident_bytes(status: &str) -> Option<u64> {+ let anon = meminfo_field(status, "RssAnon:")?;+ let shmem = meminfo_field(status, "RssShmem:").unwrap_or(0);+ Some(anon.saturating_add(shmem))+}++/// Resident minus shared pages out of a `/proc/<pid>/statm` body, in bytes:+/// the anonymous resident set on a kernel without `RssAnon`.+#[cfg(any(target_os = "linux", test))]+fn statm_anonymous_bytes(statm: &str, page: u64) -> Option<u64> {+ let mut fields = statm.split_whitespace().skip(1);+ let resident: u64 = fields.next()?.parse().ok()?;+ let shared: u64 = fields.next()?.parse().ok()?;+ Some(resident.saturating_sub(shared).saturating_mul(page))}/// macOS: the process's physical footprint (`ri_phys_footprint`), not its@@ -984,6 +1027,79 @@ mod tests {}}+ #[test]+ fn process_memory_leaves_out_clean_file_backed_pages() {+ // #2574: a node at rest on a 400 MiB cap, after a burst of searches+ // over 670 MB of mmapped segments. VmRSS (427 MB) was over the cap,+ // so every search was rejected; the heap was 178 MB.+ let status = "Name:\txerj\nVmRSS:\t 427640 kB\nRssAnon:\t 181880 kB\n\+ RssFile:\t 245760 kB\nRssShmem:\t 0 kB\nThreads:\t40\n";+ assert_eq!(unreclaimable_resident_bytes(status), Some(181_880 * 1024));+ let with_shmem = "RssAnon:\t 100 kB\nRssFile:\t 900 kB\nRssShmem:\t 20 kB\n";+ assert_eq!(unreclaimable_resident_bytes(with_shmem), Some(120 * 1024));+ // Pre-4.5 kernels have no RssAnon: let the statm fallback answer.+ assert_eq!(unreclaimable_resident_bytes("VmRSS:\t 427640 kB\n"), None);+ assert_eq!(+ unreclaimable_resident_bytes("RssAnon:\t 7 kB\n"),+ Some(7 * 1024)+ );+ // statm: size resident shared text lib data dt (pages).+ assert_eq!(+ statm_anonymous_bytes("500000 106910 61440 900 0 90000 0\n", 4096),+ Some((106_910 - 61_440) * 4096)+ );+ assert_eq!(statm_anonymous_bytes("1 2\n", 4096), None);+ assert_eq!(statm_anonymous_bytes("1 5 9 0 0 0 0", 4096), Some(0));+ }++ #[cfg(target_os = "linux")]+ #[test]+ fn process_memory_does_not_grow_with_a_touched_file_mapping() {+ // Map and touch 64 MiB of a file: VmRSS grows by about that much,+ // the probe the breaker reads must not (#2574).+ use std::io::Write;+ let dir = std::env::temp_dir().join(format!("xerj-rss-probe-{}", std::process::id()));+ std::fs::create_dir_all(&dir).unwrap();+ let path = dir.join("segment.bin");+ let len = 64 * MIB as usize;+ {+ let mut f = std::fs::File::create(&path).unwrap();+ let chunk = vec![7u8; MIB as usize];+ for _ in 0..64 {+ f.write_all(&chunk).unwrap();+ }+ f.sync_all().unwrap();+ }+ let file = std::fs::File::open(&path).unwrap();+ // SAFETY: a private read-only mapping of a file this test owns.+ let addr = unsafe {+ libc::mmap(+ std::ptr::null_mut(),+ len,+ libc::PROT_READ,+ libc::MAP_PRIVATE,+ std::os::unix::io::AsRawFd::as_raw_fd(&file),+ 0,+ )+ };+ assert_ne!(addr, libc::MAP_FAILED);+ let before = current_rss_bytes().unwrap();+ let mut sum = 0u64;+ for off in (0..len).step_by(4096) {+ // SAFETY: `off < len` inside the mapping above.+ sum += unsafe { std::ptr::read_volatile((addr as *const u8).add(off)) } as u64;+ }+ assert_eq!(sum, (len / 4096) as u64 * 7);+ let after = current_rss_bytes().unwrap();+ // SAFETY: unmapping exactly the mapping created above.+ unsafe { libc::munmap(addr, len) };+ let _ = std::fs::remove_dir_all(&dir);+ assert!(+ after < before + 16 * MIB,+ "touching a 64 MiB file mapping moved the process-memory probe from {before} to {after}"+ );+ }+#[test]fn this_machine_probes_are_sane() {// Not a fixed expectation — a probe that returns something absurd isdiff --git a/engine/crates/xerj-engine/src/governor.rs b/engine/crates/xerj-engine/src/governor.rsindex 972c34abe..46a8639b8 100644--- a/engine/crates/xerj-engine/src/governor.rs+++ b/engine/crates/xerj-engine/src/governor.rs@@ -1594,6 +1594,32 @@ fn tiered_cap_mb(effective_bytes: u64) -> u64 {}}+/// Startup warning when the machine has less memory free right now than+/// the cap this node just chose (#2574). The cap is sized from total RAM (or+/// the cgroup limit), so the friendly "detected N usable" line reads as if+/// availability had been checked. On a box where another process already+/// holds most of the RAM (a local LLM engine, say) the node will be swapped+/// or OOM-killed before its own breaker trips. `None` when availability is+/// unknown, when there is no cap, or when there is room for it.+fn available_memory_startup_line(+ available: Option<u64>,+ effective: u64,+ limit: u64,+) -> Option<String> {+ let available = available?;+ if limit == 0 || available >= limit {+ return None;+ }+ Some(format!(+ "memory: only {} of {} is available right now — other processes hold the rest — \+ which is less than the {} cap; if they keep it, lower limits.max_process_memory_mb \+ or XERJ_MAX_PROCESS_MEMORY_MB to fit what is left",+ format_gib(available),+ format_gib(effective),+ format_gib(limit),+ ))+}+/// Human-friendly binary size for the startup line: whole GiB when exact, one/// decimal when not, and MiB below 1 GiB. Presentation only.fn format_gib(bytes: u64) -> String {@@ -1964,6 +1990,13 @@ fn build(config: &Config) -> ResourceGovernor {cap_reason,);}+ if let Some(line) = available_memory_startup_line(+ xerj_common::resource::available_memory_bytes(),+ effective_memory_bytes,+ memory_limit_bytes,+ ) {+ tracing::warn!("{line}");+ }if startup_starved {tracing::warn!("Windows has no commit left under the reserve — writes will be rejected until \@@ -2629,6 +2662,31 @@ fn human_bytes(b: u64) -> String {mod tests {use super::*;+ /// #2574: a 125 GiB box with ~40 GiB free took the 16 GiB auto tier and+ /// said only "detected 125.4 GiB usable". Warn when the cap is above+ /// what is free; stay quiet when there is room or nothing is known.+ #[test]+ fn startup_warns_when_free_memory_is_below_the_cap() {+ const GIB: u64 = 1024 * 1024 * 1024;+ let line = available_memory_startup_line(Some(10 * GIB), 100 * GIB, 16 * GIB)+ .expect("10 GiB free under a 16 GiB cap must warn");+ assert!(+ line.contains("only 10 GiB of 100 GiB is available"),+ "{line}"+ );+ assert!(line.contains("16 GiB cap"), "{line}");+ assert!(line.contains("limits.max_process_memory_mb"), "{line}");+ assert_eq!(+ available_memory_startup_line(Some(40 * GIB), 125 * GIB, 16 * GIB),+ None+ );+ assert_eq!(+ available_memory_startup_line(None, 125 * GIB, 16 * GIB),+ None+ );+ assert_eq!(available_memory_startup_line(Some(GIB), 125 * GIB, 0), None);+ }+#[test]fn analyze_reservation_fits_under_the_watermark_and_rejects_at_it() {assert!(analyze_reservation_fits(0, 100, 100, 100));