AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(resource): an unprobeable machine is not a 1 GiB machine, and an explicit --workers is not a suggestion
xerj-org295cd50ab8be
diff --git a/demo/usecases/autoindex/RECIPE_DRAFT.md b/demo/usecases/autoindex/RECIPE_DRAFT.mdindex e8035dc80..927c88aa1 100644--- a/demo/usecases/autoindex/RECIPE_DRAFT.md+++ b/demo/usecases/autoindex/RECIPE_DRAFT.md@@ -222,8 +222,10 @@ curl localhost:9200/ax-*/_search -H 'Content-Type: application/json' \```Useful knobs (all optional): `--url` for a remote endpoint, `--workers N`-(default: every core, reduced when the memory safe zone cannot pay for that-many in-flight bulk buffers; bounds both the scan and the indexing phase), `--prefix` to namespace the indices (default `ax`),+(the *default* is every core, reduced when the memory safe zone cannot pay for+that many in-flight bulk buffers; a value you pass is honoured as typed and the+disagreement is printed, not applied; bounds both the scan and the indexing+phase), `--prefix` to namespace the indices (default `ax`),`--no-semantic` for pure BM25+keyword without vector fields, `--dry-run` toprint the inferred plan without indexing anything, `--follow-symlinks`(loop-safe), `--sample N` records per file for inference (default 500).diff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.mdindex 87376c998..b14b5bb95 100644--- a/docs/recipes/zero-config-indexing.md+++ b/docs/recipes/zero-config-indexing.md@@ -246,8 +246,10 @@ curl localhost:9200/ax-*/_search -H 'Content-Type: application/json' \```Useful knobs (all optional): `--url` for a non-default endpoint, `--workers N`-(default: every core, reduced when the memory safe zone cannot pay for that-many in-flight bulk buffers; bounds both the scan and the indexing phase), `--prefix` to namespace the indices (default `ax`),+(the *default* is every core, reduced when the memory safe zone cannot pay for+that many in-flight bulk buffers; a value you pass is honoured as typed and the+disagreement is printed, not applied; bounds both the scan and the indexing+phase), `--prefix` to namespace the indices (default `ax`),`--no-semantic` for pure BM25+keyword without vector fields, `--dry-run` toprint the inferred plan without indexing anything, `--follow-symlinks`(loop-safe), `--sample N` records per file for inference (default 500).diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex 9af0f748b..748159478 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -78,10 +78,12 @@ pub fn print_help() {--api-key <K> Authorization header (or env XERJ_API_KEY)\n\--workers <N> workers for BOTH phases — content hashing/sniffing and\n\indexing (default: every core, reduced if the memory\n\- safe zone cannot pay for that many bulk buffers;\n\+ safe zone cannot pay for that many bulk buffers; a\n\+ value you pass is honoured and only warned about;\n\valid range 1..=1024)\n\--pdf-workers <N> concurrent PDF parser processes (default min(cores,4); max 4,\n\- reduced further on a machine with little free memory)\n\+ the default reduced further on a machine with little\n\+ free memory)\n\--pdf-timeout-secs <N> per-PDF parser timeout (default 120; max 3600)\n\--bulk-mb <N> bulk cut size in MB (default 8)\n\--bulk-timeout-secs <N>\n\diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 4d31f4f99..e4d255704 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -2409,7 +2409,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {"pdf_workers": cfg.pdf_workers,"bulk_mb": cfg.bulk_mb,"cores_available": xerj_common::resource::cores(),- "memory_safe_zone_mb": xerj_common::resource::memory_safe_zone_bytes() / (1024 * 1024),+ // `null` on a platform with no RAM probe — the summary reports what the+ // run actually knew, never a stand-in number (#240).+ "memory_safe_zone_mb": xerj_common::resource::memory_safe_zone_bytes()+ .map(|b| b / (1024 * 1024)),// What the server's backpressure did to the offered load. A final// limit below `workers` means this run met real congestion and// answered it, which is the difference between a slow run and a rundiff --git a/engine/crates/xerj-autoindex/src/resources.rs b/engine/crates/xerj-autoindex/src/resources.rsindex 987e45e07..41fd64722 100644--- a/engine/crates/xerj-autoindex/src/resources.rs+++ b/engine/crates/xerj-autoindex/src/resources.rs@@ -21,6 +21,26 @@//! * **PDF workers** — separate processes, each allowed 1536 MiB of address//! space (`extract::pdf::WORKER_ADDRESS_SPACE`), so the cap is whatever the//! safe zone can actually pay for, never more than the measured 4.+//!+//! Two rules keep the memory arithmetic from doing damage of its own:+//!+//! * **A memory cap the machine cannot substantiate is not applied.** The safe+//! zone is an `Option`; `None` (Windows, FreeBSD — no RAM probe) means "no+//! memory-derived cap", not "a 1 GiB machine". Getting this wrong cut PDF+//! extraction from `min(cores, 4)` to 1 on every Windows run, charged against+//! an `RLIMIT_AS` that is `#[cfg(unix)]`-only and therefore never enforced+//! there (`extract::pdf::containment_description`). Same shape meilisearch+//! uses: `Option<u64>` for the probe (`meilisearch/src/option.rs:1135`),+//! `Option` for the derived budget, `None` = unbounded+//! (`milli/.../grenad_helpers.rs:122`).+//! * **The cap applies to the default, not to the operator.** A number the user+//! typed is honoured and, if the safe zone disagrees, said out loud — the+//! safe zone is a heuristic built on a live `MemAvailable` reading, and on a+//! large box a transient dip can drive it to its 256 MiB floor. Silently+//! turning `--workers 32` into 3 is the accepted-and-quietly-replaced class+//! from #204. Hard range errors are still refused outright at the CLI, which+//! is the split tantivy draws too (`index_writer.rs:2831` rejects an+//! out-of-range writer option instead of clamping it).use xerj_common::resource;@@ -57,8 +77,9 @@ pub struct Plan {pub index_workers: usize,/// Concurrent isolated PDF extraction subprocesses.pub pdf_workers: usize,- /// Bytes this run considers itself entitled to.- pub safe_zone_bytes: u64,+ /// Bytes this run considers itself entitled to, or `None` on a machine+ /// whose RAM this platform cannot probe.+ pub safe_zone_bytes: Option<u64>,/// Anything the machine forced on the run, for the operator to see.pub notes: Vec<String>,}@@ -82,15 +103,30 @@ pub fn plan(}/// Pure planning rule — the tested half of [`plan`].+///+/// `safe_zone_bytes` is `None` when this platform cannot read the machine's+/// RAM; that disables every memory-derived cap below rather than standing in+/// for a small machine.pub fn plan_for(cores: usize,- safe_zone_bytes: u64,+ safe_zone_bytes: Option<u64>,requested_workers: Option<usize>,requested_pdf_workers: Option<usize>,bulk_mb: usize,) -> Plan {let cores = cores.max(1);let mut notes = Vec::new();+ if safe_zone_bytes.is_none() {+ notes.push(format!(+ "cannot read this machine's RAM on {}; worker counts are core-derived and not \+ memory-capped — pass --workers/--pdf-workers if that is too much for this box",+ std::env::consts::OS+ ));+ }+ let zone_mib = || match safe_zone_bytes {+ Some(b) => format!("{} MiB", b / MIB),+ None => "unknown".to_string(),+ };// Phase A: every core by default. "Use the machine" belongs here — this is// the phase that is actually CPU-bound.@@ -101,29 +137,53 @@ pub fn plan_for(let per_worker = (bulk_mb.max(1) as u64).saturating_mul(WORKER_BYTES_PER_BULK_MIB).saturating_mul(MIB);- let affordable = (safe_zone_bytes / per_worker).max(1) as usize;+ let affordable = safe_zone_bytes.map(|zone| (zone / per_worker).max(1) as usize);let mut index_workers = scan_threads;- if index_workers > affordable {- notes.push(format!(- "memory safe zone {} MiB allows {affordable} index workers at --bulk-mb {bulk_mb}, \- not {index_workers}",- safe_zone_bytes / MIB- ));- index_workers = affordable;+ if let Some(affordable) = affordable {+ if index_workers > affordable {+ if requested_workers.is_some() {+ // The operator named a number. Say what the machine thinks and+ // then do as asked (#204: no silent substitution).+ notes.push(format!(+ "memory safe zone {} allows {affordable} index workers at --bulk-mb \+ {bulk_mb}; honouring --workers {index_workers} anyway — lower --workers or \+ --bulk-mb if this run starts swapping",+ zone_mib()+ ));+ } else {+ notes.push(format!(+ "memory safe zone {} allows {affordable} index workers at --bulk-mb \+ {bulk_mb}, not {index_workers}",+ zone_mib()+ ));+ index_workers = affordable;+ }+ }}// PDF workers: bounded by the measured ceiling, by the cores available,- // and by what the safe zone can pay for at 1536 MiB of address space each.- let pdf_affordable = (safe_zone_bytes / PDF_WORKER_BYTES).max(1) as usize;+ // and — only where the safe zone is a real reading — by what it can pay for+ // at 1536 MiB of address space each.+ let pdf_affordable = safe_zone_bytes.map(|zone| (zone / PDF_WORKER_BYTES).max(1) as usize);let pdf_default = cores.min(MAX_PDF_WORKERS);let mut pdf_workers = requested_pdf_workers.unwrap_or(pdf_default).max(1);- if pdf_workers > pdf_affordable {- notes.push(format!(- "memory safe zone {} MiB allows {pdf_affordable} PDF worker(s) at 1536 MiB each, \- not {pdf_workers}",- safe_zone_bytes / MIB- ));- pdf_workers = pdf_affordable;+ if let Some(pdf_affordable) = pdf_affordable {+ if pdf_workers > pdf_affordable {+ if requested_pdf_workers.is_some() {+ notes.push(format!(+ "memory safe zone {} allows {pdf_affordable} PDF worker(s) at 1536 MiB each; \+ honouring --pdf-workers {pdf_workers} anyway",+ zone_mib()+ ));+ } else {+ notes.push(format!(+ "memory safe zone {} allows {pdf_affordable} PDF worker(s) at 1536 MiB each, \+ not {pdf_workers}",+ zone_mib()+ ));+ pdf_workers = pdf_affordable;+ }+ }}Plan {@@ -138,12 +198,15 @@ pub fn plan_for(impl Plan {/// One line for the run header and the `--json` summary.pub fn describe(&self) -> String {+ let zone = match self.safe_zone_bytes {+ Some(b) => format!("{} MiB", b / MIB),+ None => "unknown".to_string(),+ };format!(- "resources: {} scan threads, {} index workers, {} pdf workers, memory safe zone {} MiB ({})",+ "resources: {} scan threads, {} index workers, {} pdf workers, memory safe zone {zone} ({})",self.scan_threads,self.index_workers,self.pdf_workers,- self.safe_zone_bytes / MIB,resource::describe(),)}@@ -157,18 +220,18 @@ mod tests {#[test]fn the_default_plan_uses_the_whole_machine_for_the_cpu_bound_phase() {- let plan = plan_for(12, 8 * GIB, None, None, 8);+ let plan = plan_for(12, Some(8 * GIB), None, None, 8);assert_eq!(plan.scan_threads, 12, "phase A is the CPU-bound phase");assert_eq!(plan.index_workers, 12);assert_eq!(plan.pdf_workers, 4);assert!(plan.notes.is_empty());// The pre-#240 default capped both at 8 on any machine with more cores.- assert_eq!(plan_for(32, 32 * GIB, None, None, 8).scan_threads, 32);+ assert_eq!(plan_for(32, Some(32 * GIB), None, None, 8).scan_threads, 32);}#[test]fn an_explicit_request_is_honoured_on_both_phases() {- let plan = plan_for(32, 32 * GIB, Some(2), Some(1), 8);+ let plan = plan_for(32, Some(32 * GIB), Some(2), Some(1), 8);assert_eq!(plan.scan_threads, 2, "--workers must bound phase A too");assert_eq!(plan.index_workers, 2);assert_eq!(plan.pdf_workers, 1);@@ -178,7 +241,7 @@ mod tests {fn a_small_machine_gets_fewer_workers_and_is_told_why() {// A 32-core box squeezed into a 512 MiB safe zone (small container),// asked for the biggest bulk size: 72 MiB per worker buys 7 of them.- let plan = plan_for(32, 512 * MIB, None, None, 24);+ let plan = plan_for(32, Some(512 * MIB), None, None, 24);assert_eq!(plan.index_workers, 7);assert_eq!(plan.scan_threads, 32,@@ -190,15 +253,69 @@ mod tests {assert!(plan.notes.iter().any(|n| n.contains("PDF worker")));}+ /// A machine whose RAM this platform cannot read (Windows, FreeBSD) must be+ /// planned as *unknown*, not as the smallest machine imaginable. The+ /// revision this replaces substituted a 1 GiB safe zone here, which bought+ /// exactly one PDF worker at 1536 MiB each — a 4× cut to PDF extraction on+ /// every Windows run, charged against an `RLIMIT_AS` that is `cfg(unix)`+ /// and so is never applied on that platform.+ #[test]+ fn an_unprobeable_machine_is_not_treated_as_a_tiny_one() {+ let plan = plan_for(16, None, None, None, 8);+ assert_eq!(plan.pdf_workers, 4, "min(cores, MAX_PDF_WORKERS), not 1");+ assert_eq!(plan.index_workers, 16, "no memory-derived cap to apply");+ assert_eq!(plan.scan_threads, 16);+ assert_eq!(plan.safe_zone_bytes, None);+ assert!(+ plan.notes.iter().any(|n| n.contains("cannot read")),+ "the run must say the budget is core-derived: {:?}",+ plan.notes+ );+ // …and the operator's own numbers still win on that platform.+ let asked = plan_for(16, None, Some(2), Some(1), 8);+ assert_eq!((asked.index_workers, asked.pdf_workers), (2, 1));+ // A two-core Windows box still gets the core-derived PDF default.+ assert_eq!(plan_for(2, None, None, None, 8).pdf_workers, 2);+ }++ /// The memory heuristic bounds the *default*. A number the operator typed+ /// is honoured and argued with out loud — silently replacing it is the+ /// accepted-and-quietly-ignored class from #204, and the trigger is easy to+ /// hit: the safe zone is computed from a live `MemAvailable` reading, so a+ /// transient dip on a large box can drop it to its 256 MiB floor.+ #[test]+ fn an_explicit_count_survives_a_small_safe_zone_but_is_argued_with() {+ let plan = plan_for(32, Some(256 * MIB), Some(32), Some(4), 24);+ assert_eq!(plan.index_workers, 32, "--workers 32 must stay 32");+ assert_eq!(plan.pdf_workers, 4, "--pdf-workers 4 must stay 4");+ assert!(+ plan.notes+ .iter()+ .any(|n| n.contains("honouring --workers 32")),+ "the disagreement must be stated: {:?}",+ plan.notes+ );+ assert!(plan+ .notes+ .iter()+ .any(|n| n.contains("honouring --pdf-workers 4")));+ // The same machine with no flags still gets the cap, and is told.+ let defaulted = plan_for(32, Some(256 * MIB), None, None, 24);+ assert_eq!((defaulted.index_workers, defaulted.pdf_workers), (3, 1));+ assert!(defaulted.notes.iter().all(|n| !n.contains("honouring")));+ }+#[test]fn the_plan_is_never_zero_or_absurd() {for cores in [0usize, 1, 2, 256] {- for safe in [0u64, 64 * MIB, GIB, 512 * GIB] {+ for safe in [None, Some(0u64), Some(64 * MIB), Some(GIB), Some(512 * GIB)] {for bulk in [1usize, 8, 24] {- let plan = plan_for(cores, safe, None, None, bulk);- assert!(plan.scan_threads >= 1);- assert!(plan.index_workers >= 1);- assert!((1..=MAX_PDF_WORKERS).contains(&plan.pdf_workers));+ for asked in [None, Some(1usize), Some(MAX_WORKERS)] {+ let plan = plan_for(cores, safe, asked, None, bulk);+ assert!(plan.scan_threads >= 1);+ assert!(plan.index_workers >= 1);+ assert!((1..=MAX_PDF_WORKERS).contains(&plan.pdf_workers));+ }}}}diff --git a/engine/crates/xerj-common/src/resource.rs b/engine/crates/xerj-common/src/resource.rsindex a6d6d6658..b1ab124cc 100644--- a/engine/crates/xerj-common/src/resource.rs+++ b/engine/crates/xerj-common/src/resource.rs@@ -472,20 +472,23 @@ fn sysctl_u64(name: &std::ffi::CStr) -> Option<u64> {/// Half of RAM as the ceiling follows the same reasoning meilisearch gives for/// its two-thirds indexing budget (`option.rs:1103-1106`) — a machine that is/// also serving its owner should not have its page cache evicted by a-/// background indexer. When `total` is unknown the caller gets-/// [`FALLBACK_SAFE_ZONE_BYTES`], which is deliberately small: an unknown-/// machine is not an excuse to take an unknown amount of it.-pub fn memory_safe_zone_bytes() -> u64 {- match total_memory_bytes() {- Some(total) => safe_zone(total, available_memory_bytes()),- None => FALLBACK_SAFE_ZONE_BYTES,- }+/// background indexer.+///+/// `None` means **this machine's RAM is not knowable here** (Windows, FreeBSD,+/// …: [`total_memory_bytes`] has no probe), and callers must read it as "no+/// memory-derived limit", never as "a small limit". An earlier revision of this+/// function returned a 1 GiB fallback instead, and the result was that a+/// Windows box with 64 GB of RAM was budgeted as a 1 GiB machine — a phantom+/// budget that no code on that platform enforces. meilisearch keeps the same+/// shape for the same reason: `total_memory_bytes() -> Option<u64>` is `None`+/// on an unsupported system (`crates/meilisearch/src/option.rs:1135`) and the+/// derived per-thread budget stays an `Option` all the way down, where `None`+/// means unbounded rather than minimal+/// (`crates/milli/src/update/index_documents/helpers/grenad_helpers.rs:122`).+pub fn memory_safe_zone_bytes() -> Option<u64> {+ total_memory_bytes().map(|total| safe_zone(total, available_memory_bytes()))}-/// The budget used when the machine's RAM cannot be probed at all: 1 GiB,-/// small enough to be safe on the smallest machine XERJ is expected to run on.-pub const FALLBACK_SAFE_ZONE_BYTES: u64 = GIB;-/// Pure safe-zone rule — the tested half of [`memory_safe_zone_bytes`].fn safe_zone(total: u64, available: Option<u64>) -> u64 {let reserve = (total / 8).max(GIB);@@ -505,7 +508,7 @@ pub fn describe() -> String {cores(),mib(total_memory_bytes()),mib(available_memory_bytes()),- memory_safe_zone_bytes() / MIB,+ mib(memory_safe_zone_bytes()),)}@@ -643,7 +646,11 @@ mod tests {assert!((1..=4096).contains(&cores));if let Some(total) = total_memory_bytes() {assert!(total >= 256 * MIB, "implausible total RAM: {total}");- assert!(memory_safe_zone_bytes() <= total);+ assert!(memory_safe_zone_bytes().expect("probeable RAM has a safe zone") <= total);+ } else {+ // Unknown must stay unknown: a fabricated small budget here is the+ // Windows PDF-throttling bug this API shape exists to prevent.+ assert_eq!(memory_safe_zone_bytes(), None);}#[cfg(any(target_os = "linux", target_os = "macos"))]{