Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(governor): count only memory the node can release on Linux (#2574) (#2656)

xerj-org8ba8c2e6bc02
diff --git a/docs/recipes/production-deployment.md b/docs/recipes/production-deployment.md
index 4a1d16d81..9cb2cd65a 100644
--- a/docs/recipes/production-deployment.md
+++ b/docs/recipes/production-deployment.md
@@ -396,6 +396,32 @@ transport encryption; your platform provides the rest.
| Secrets written `0600` (admin key, auto-generated TLS key) | ✅ |
| Client identity from the TCP peer, not `X-Forwarded-For` | ✅ (`server.trusted_proxies`, empty by default) |
+### Memory limits and what "process memory" means
+
+The memory breaker compares the node's own memory against one cap. These
+`[limits]` keys set it (all MiB, `0` = automatic):
+
+| Key | Env override | Default | What it does |
+|---|---|---|---|
+| `max_process_memory_mb` | `XERJ_MAX_PROCESS_MEMORY_MB` | `0` (tier from total RAM or the cgroup limit) | The cap. Writes get `429` above the watermark below; searches get `429` only after waiting for a drain that did not bring memory under the cap. |
+| `memory_watermark_percent` | – | `95` | Percent of the cap where writes start to get `429`. |
+| `max_total_memtable_mb` | – | `0` (25% of the limit, floor 2 GiB, at most 50%) | Unflushed write buffers across all indices. |
+| `max_segment_hydration_cache_mb` | `XERJ_SEGMENT_HYDRATION_CACHE_MB` | `0` (20% of the limit) | See the next section. |
+
+"Process memory" (`rss_mb` in breaker log lines) is memory the node itself
+allocated and that only it can give back. On Linux that is `RssAnon +
+RssShmem` from `/proc/self/status`, not the `RSS` column `top` shows: segment
+files are memory-mapped, and the pages of those files a search reads stay in
+the process's RSS as page cache until the kernel needs the RAM back. The node
+cannot release those pages itself, so counting them made a node whose
+mapped segments were larger than its cap answer every search with `429`
+until it was restarted (#2574). Expect `top` to show more than `rss_mb`.
+
+The automatic cap is sized from total RAM, not from what is free. If other
+processes already hold most of the RAM, the startup log adds a
+`memory: only … is available right now` warning. In that case set
+`max_process_memory_mb` to fit what is left.
+
### Bound retained segment hydration
Every index shares one process-wide admission authority for the seven
diff --git a/engine/crates/xerj-common/src/resource.rs b/engine/crates/xerj-common/src/resource.rs
index 937506112..24bb2c27d 100644
--- a/engine/crates/xerj-common/src/resource.rs
+++ b/engine/crates/xerj-common/src/resource.rs
@@ -402,9 +402,21 @@ pub fn available_memory_bytes() -> Option<u64> {
probe_available_memory()
}
-/// Resident set size of this process (on macOS its physical footprint, see
-/// the macOS `probe_rss`), or `None` where this platform has no probe.
-/// `None` must disable an RSS-driven decision, never read as zero.
+/// Memory this process holds that the kernel cannot drop on its own, or
+/// `None` where this platform has no probe. On Linux that is the anonymous
+/// and shared-memory resident pages (`RssAnon + RssShmem`), not `VmRSS`; on
+/// macOS the physical footprint (see the macOS `probe_rss`); on Windows the
+/// working set. `None` must disable an RSS-driven decision, never read as
+/// zero.
+///
+/// Why not `VmRSS` on Linux (#2574): it also counts clean file-backed pages
+/// of every mmapped segment, term dictionary and postings file a search has
+/// touched. The kernel reclaims those whenever another process needs RAM,
+/// so they cannot exhaust memory, but no drain inside the process can
+/// release them while the mapping is open. On a node whose indices are
+/// larger than its memory cap, searches faulted those pages in until RSS sat
+/// above the cap, and from then on every search was rejected with
+/// `circuit_breaking_exception` until the node restarted.
pub fn current_rss_bytes() -> Option<u64> {
probe_rss()
}
@@ -554,14 +566,45 @@ fn meminfo_field(meminfo: &str, key: &str) -> Option<u64> {
None
}
+/// Linux: anonymous plus shared-memory resident bytes from
+/// `/proc/self/status` (#2574). Clean file-backed pages (mmapped segments)
+/// are left out because the kernel drops them under pressure; see
+/// [`current_rss_bytes`]. Kernels older than 4.5 have no `RssAnon` line, so
+/// fall back to `/proc/self/statm` resident minus its shared (file-backed and
+/// shmem) pages, which is the same anonymous figure without the shmem part.
#[cfg(target_os = "linux")]
fn probe_rss() -> Option<u64> {
+ if let Some(bytes) = std::fs::read_to_string("/proc/self/status")
+ .ok()
+ .and_then(|status| unreclaimable_resident_bytes(&status))
+ {
+ return Some(bytes);
+ }
let statm = std::fs::read_to_string("/proc/self/statm").ok()?;
- let pages: u64 = statm.split_whitespace().nth(1)?.parse().ok()?;
// SAFETY: `sysconf` reads a system constant.
let page = unsafe { libc::sysconf(libc::_SC_PAGESIZE) };
let page = if page > 0 { page as u64 } else { 4096 };
- Some(pages.saturating_mul(page))
+ statm_anonymous_bytes(&statm, page)
+}
+
+/// `RssAnon + RssShmem` out of a `/proc/<pid>/status` body, in bytes. `None`
+/// when the kernel has no `RssAnon` line (pre-4.5). A missing `RssShmem`
+/// counts as zero.
+#[cfg(any(target_os = "linux", test))]
+fn unreclaimable_resident_bytes(status: &str) -> Option<u64> {
+ let anon = meminfo_field(status, "RssAnon:")?;
+ let shmem = meminfo_field(status, "RssShmem:").unwrap_or(0);
+ Some(anon.saturating_add(shmem))
+}
+
+/// Resident minus shared pages out of a `/proc/<pid>/statm` body, in bytes:
+/// the anonymous resident set on a kernel without `RssAnon`.
+#[cfg(any(target_os = "linux", test))]
+fn statm_anonymous_bytes(statm: &str, page: u64) -> Option<u64> {
+ let mut fields = statm.split_whitespace().skip(1);
+ let resident: u64 = fields.next()?.parse().ok()?;
+ let shared: u64 = fields.next()?.parse().ok()?;
+ Some(resident.saturating_sub(shared).saturating_mul(page))
}
/// macOS: the process's physical footprint (`ri_phys_footprint`), not its
@@ -984,6 +1027,79 @@ mod tests {
}
}
+ #[test]
+ fn process_memory_leaves_out_clean_file_backed_pages() {
+ // #2574: a node at rest on a 400 MiB cap, after a burst of searches
+ // over 670 MB of mmapped segments. VmRSS (427 MB) was over the cap,
+ // so every search was rejected; the heap was 178 MB.
+ let status = "Name:\txerj\nVmRSS:\t 427640 kB\nRssAnon:\t 181880 kB\n\
+ RssFile:\t 245760 kB\nRssShmem:\t 0 kB\nThreads:\t40\n";
+ assert_eq!(unreclaimable_resident_bytes(status), Some(181_880 * 1024));
+ let with_shmem = "RssAnon:\t 100 kB\nRssFile:\t 900 kB\nRssShmem:\t 20 kB\n";
+ assert_eq!(unreclaimable_resident_bytes(with_shmem), Some(120 * 1024));
+ // Pre-4.5 kernels have no RssAnon: let the statm fallback answer.
+ assert_eq!(unreclaimable_resident_bytes("VmRSS:\t 427640 kB\n"), None);
+ assert_eq!(
+ unreclaimable_resident_bytes("RssAnon:\t 7 kB\n"),
+ Some(7 * 1024)
+ );
+ // statm: size resident shared text lib data dt (pages).
+ assert_eq!(
+ statm_anonymous_bytes("500000 106910 61440 900 0 90000 0\n", 4096),
+ Some((106_910 - 61_440) * 4096)
+ );
+ assert_eq!(statm_anonymous_bytes("1 2\n", 4096), None);
+ assert_eq!(statm_anonymous_bytes("1 5 9 0 0 0 0", 4096), Some(0));
+ }
+
+ #[cfg(target_os = "linux")]
+ #[test]
+ fn process_memory_does_not_grow_with_a_touched_file_mapping() {
+ // Map and touch 64 MiB of a file: VmRSS grows by about that much,
+ // the probe the breaker reads must not (#2574).
+ use std::io::Write;
+ let dir = std::env::temp_dir().join(format!("xerj-rss-probe-{}", std::process::id()));
+ std::fs::create_dir_all(&dir).unwrap();
+ let path = dir.join("segment.bin");
+ let len = 64 * MIB as usize;
+ {
+ let mut f = std::fs::File::create(&path).unwrap();
+ let chunk = vec![7u8; MIB as usize];
+ for _ in 0..64 {
+ f.write_all(&chunk).unwrap();
+ }
+ f.sync_all().unwrap();
+ }
+ let file = std::fs::File::open(&path).unwrap();
+ // SAFETY: a private read-only mapping of a file this test owns.
+ let addr = unsafe {
+ libc::mmap(
+ std::ptr::null_mut(),
+ len,
+ libc::PROT_READ,
+ libc::MAP_PRIVATE,
+ std::os::unix::io::AsRawFd::as_raw_fd(&file),
+ 0,
+ )
+ };
+ assert_ne!(addr, libc::MAP_FAILED);
+ let before = current_rss_bytes().unwrap();
+ let mut sum = 0u64;
+ for off in (0..len).step_by(4096) {
+ // SAFETY: `off < len` inside the mapping above.
+ sum += unsafe { std::ptr::read_volatile((addr as *const u8).add(off)) } as u64;
+ }
+ assert_eq!(sum, (len / 4096) as u64 * 7);
+ let after = current_rss_bytes().unwrap();
+ // SAFETY: unmapping exactly the mapping created above.
+ unsafe { libc::munmap(addr, len) };
+ let _ = std::fs::remove_dir_all(&dir);
+ assert!(
+ after < before + 16 * MIB,
+ "touching a 64 MiB file mapping moved the process-memory probe from {before} to {after}"
+ );
+ }
+
#[test]
fn this_machine_probes_are_sane() {
// Not a fixed expectation — a probe that returns something absurd is
diff --git a/engine/crates/xerj-engine/src/governor.rs b/engine/crates/xerj-engine/src/governor.rs
index 972c34abe..46a8639b8 100644
--- a/engine/crates/xerj-engine/src/governor.rs
+++ b/engine/crates/xerj-engine/src/governor.rs
@@ -1594,6 +1594,32 @@ fn tiered_cap_mb(effective_bytes: u64) -> u64 {
}
}
+/// Startup warning when the machine has less memory free right now than
+/// the cap this node just chose (#2574). The cap is sized from total RAM (or
+/// the cgroup limit), so the friendly "detected N usable" line reads as if
+/// availability had been checked. On a box where another process already
+/// holds most of the RAM (a local LLM engine, say) the node will be swapped
+/// or OOM-killed before its own breaker trips. `None` when availability is
+/// unknown, when there is no cap, or when there is room for it.
+fn available_memory_startup_line(
+ available: Option<u64>,
+ effective: u64,
+ limit: u64,
+) -> Option<String> {
+ let available = available?;
+ if limit == 0 || available >= limit {
+ return None;
+ }
+ Some(format!(
+ "memory: only {} of {} is available right now — other processes hold the rest — \
+ which is less than the {} cap; if they keep it, lower limits.max_process_memory_mb \
+ or XERJ_MAX_PROCESS_MEMORY_MB to fit what is left",
+ format_gib(available),
+ format_gib(effective),
+ format_gib(limit),
+ ))
+}
+
/// Human-friendly binary size for the startup line: whole GiB when exact, one
/// decimal when not, and MiB below 1 GiB. Presentation only.
fn format_gib(bytes: u64) -> String {
@@ -1964,6 +1990,13 @@ fn build(config: &Config) -> ResourceGovernor {
cap_reason,
);
}
+ if let Some(line) = available_memory_startup_line(
+ xerj_common::resource::available_memory_bytes(),
+ effective_memory_bytes,
+ memory_limit_bytes,
+ ) {
+ tracing::warn!("{line}");
+ }
if startup_starved {
tracing::warn!(
"Windows has no commit left under the reserve — writes will be rejected until \
@@ -2629,6 +2662,31 @@ fn human_bytes(b: u64) -> String {
mod tests {
use super::*;
+ /// #2574: a 125 GiB box with ~40 GiB free took the 16 GiB auto tier and
+ /// said only "detected 125.4 GiB usable". Warn when the cap is above
+ /// what is free; stay quiet when there is room or nothing is known.
+ #[test]
+ fn startup_warns_when_free_memory_is_below_the_cap() {
+ const GIB: u64 = 1024 * 1024 * 1024;
+ let line = available_memory_startup_line(Some(10 * GIB), 100 * GIB, 16 * GIB)
+ .expect("10 GiB free under a 16 GiB cap must warn");
+ assert!(
+ line.contains("only 10 GiB of 100 GiB is available"),
+ "{line}"
+ );
+ assert!(line.contains("16 GiB cap"), "{line}");
+ assert!(line.contains("limits.max_process_memory_mb"), "{line}");
+ assert_eq!(
+ available_memory_startup_line(Some(40 * GIB), 125 * GIB, 16 * GIB),
+ None
+ );
+ assert_eq!(
+ available_memory_startup_line(None, 125 * GIB, 16 * GIB),
+ None
+ );
+ assert_eq!(available_memory_startup_line(Some(GIB), 125 * GIB, 0), None);
+ }
+
#[test]
fn analyze_reservation_fits_under_the_watermark_and_rejects_at_it() {
assert!(analyze_reservation_fits(0, 100, 100, 100));