Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(resource): an unprobeable machine is not a 1 GiB machine, and an explicit --workers is not a suggestion

xerj-org295cd50ab8be
diff --git a/demo/usecases/autoindex/RECIPE_DRAFT.md b/demo/usecases/autoindex/RECIPE_DRAFT.md
index e8035dc80..927c88aa1 100644
--- a/demo/usecases/autoindex/RECIPE_DRAFT.md
+++ b/demo/usecases/autoindex/RECIPE_DRAFT.md
@@ -222,8 +222,10 @@ curl localhost:9200/ax-*/_search -H 'Content-Type: application/json' \
```
Useful knobs (all optional): `--url` for a remote endpoint, `--workers N`
-(default: every core, reduced when the memory safe zone cannot pay for that
-many in-flight bulk buffers; bounds both the scan and the indexing phase), `--prefix` to namespace the indices (default `ax`),
+(the *default* is every core, reduced when the memory safe zone cannot pay for
+that many in-flight bulk buffers; a value you pass is honoured as typed and the
+disagreement is printed, not applied; bounds both the scan and the indexing
+phase), `--prefix` to namespace the indices (default `ax`),
`--no-semantic` for pure BM25+keyword without vector fields, `--dry-run` to
print the inferred plan without indexing anything, `--follow-symlinks`
(loop-safe), `--sample N` records per file for inference (default 500).
diff --git a/docs/recipes/zero-config-indexing.md b/docs/recipes/zero-config-indexing.md
index 87376c998..b14b5bb95 100644
--- a/docs/recipes/zero-config-indexing.md
+++ b/docs/recipes/zero-config-indexing.md
@@ -246,8 +246,10 @@ curl localhost:9200/ax-*/_search -H 'Content-Type: application/json' \
```
Useful knobs (all optional): `--url` for a non-default endpoint, `--workers N`
-(default: every core, reduced when the memory safe zone cannot pay for that
-many in-flight bulk buffers; bounds both the scan and the indexing phase), `--prefix` to namespace the indices (default `ax`),
+(the *default* is every core, reduced when the memory safe zone cannot pay for
+that many in-flight bulk buffers; a value you pass is honoured as typed and the
+disagreement is printed, not applied; bounds both the scan and the indexing
+phase), `--prefix` to namespace the indices (default `ax`),
`--no-semantic` for pure BM25+keyword without vector fields, `--dry-run` to
print the inferred plan without indexing anything, `--follow-symlinks`
(loop-safe), `--sample N` records per file for inference (default 500).
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index 9af0f748b..748159478 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -78,10 +78,12 @@ pub fn print_help() {
--api-key <K> Authorization header (or env XERJ_API_KEY)\n\
--workers <N> workers for BOTH phases — content hashing/sniffing and\n\
indexing (default: every core, reduced if the memory\n\
- safe zone cannot pay for that many bulk buffers;\n\
+ safe zone cannot pay for that many bulk buffers; a\n\
+ value you pass is honoured and only warned about;\n\
valid range 1..=1024)\n\
--pdf-workers <N> concurrent PDF parser processes (default min(cores,4); max 4,\n\
- reduced further on a machine with little free memory)\n\
+ the default reduced further on a machine with little\n\
+ free memory)\n\
--pdf-timeout-secs <N> per-PDF parser timeout (default 120; max 3600)\n\
--bulk-mb <N> bulk cut size in MB (default 8)\n\
--bulk-timeout-secs <N>\n\
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index 4d31f4f99..e4d255704 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -2409,7 +2409,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
"pdf_workers": cfg.pdf_workers,
"bulk_mb": cfg.bulk_mb,
"cores_available": xerj_common::resource::cores(),
- "memory_safe_zone_mb": xerj_common::resource::memory_safe_zone_bytes() / (1024 * 1024),
+ // `null` on a platform with no RAM probe — the summary reports what the
+ // run actually knew, never a stand-in number (#240).
+ "memory_safe_zone_mb": xerj_common::resource::memory_safe_zone_bytes()
+ .map(|b| b / (1024 * 1024)),
// What the server's backpressure did to the offered load. A final
// limit below `workers` means this run met real congestion and
// answered it, which is the difference between a slow run and a run
diff --git a/engine/crates/xerj-autoindex/src/resources.rs b/engine/crates/xerj-autoindex/src/resources.rs
index 987e45e07..41fd64722 100644
--- a/engine/crates/xerj-autoindex/src/resources.rs
+++ b/engine/crates/xerj-autoindex/src/resources.rs
@@ -21,6 +21,26 @@
//! * **PDF workers** — separate processes, each allowed 1536 MiB of address
//! space (`extract::pdf::WORKER_ADDRESS_SPACE`), so the cap is whatever the
//! safe zone can actually pay for, never more than the measured 4.
+//!
+//! Two rules keep the memory arithmetic from doing damage of its own:
+//!
+//! * **A memory cap the machine cannot substantiate is not applied.** The safe
+//! zone is an `Option`; `None` (Windows, FreeBSD — no RAM probe) means "no
+//! memory-derived cap", not "a 1 GiB machine". Getting this wrong cut PDF
+//! extraction from `min(cores, 4)` to 1 on every Windows run, charged against
+//! an `RLIMIT_AS` that is `#[cfg(unix)]`-only and therefore never enforced
+//! there (`extract::pdf::containment_description`). Same shape meilisearch
+//! uses: `Option<u64>` for the probe (`meilisearch/src/option.rs:1135`),
+//! `Option` for the derived budget, `None` = unbounded
+//! (`milli/.../grenad_helpers.rs:122`).
+//! * **The cap applies to the default, not to the operator.** A number the user
+//! typed is honoured and, if the safe zone disagrees, said out loud — the
+//! safe zone is a heuristic built on a live `MemAvailable` reading, and on a
+//! large box a transient dip can drive it to its 256 MiB floor. Silently
+//! turning `--workers 32` into 3 is the accepted-and-quietly-replaced class
+//! from #204. Hard range errors are still refused outright at the CLI, which
+//! is the split tantivy draws too (`index_writer.rs:2831` rejects an
+//! out-of-range writer option instead of clamping it).
use xerj_common::resource;
@@ -57,8 +77,9 @@ pub struct Plan {
pub index_workers: usize,
/// Concurrent isolated PDF extraction subprocesses.
pub pdf_workers: usize,
- /// Bytes this run considers itself entitled to.
- pub safe_zone_bytes: u64,
+ /// Bytes this run considers itself entitled to, or `None` on a machine
+ /// whose RAM this platform cannot probe.
+ pub safe_zone_bytes: Option<u64>,
/// Anything the machine forced on the run, for the operator to see.
pub notes: Vec<String>,
}
@@ -82,15 +103,30 @@ pub fn plan(
}
/// Pure planning rule — the tested half of [`plan`].
+///
+/// `safe_zone_bytes` is `None` when this platform cannot read the machine's
+/// RAM; that disables every memory-derived cap below rather than standing in
+/// for a small machine.
pub fn plan_for(
cores: usize,
- safe_zone_bytes: u64,
+ safe_zone_bytes: Option<u64>,
requested_workers: Option<usize>,
requested_pdf_workers: Option<usize>,
bulk_mb: usize,
) -> Plan {
let cores = cores.max(1);
let mut notes = Vec::new();
+ if safe_zone_bytes.is_none() {
+ notes.push(format!(
+ "cannot read this machine's RAM on {}; worker counts are core-derived and not \
+ memory-capped — pass --workers/--pdf-workers if that is too much for this box",
+ std::env::consts::OS
+ ));
+ }
+ let zone_mib = || match safe_zone_bytes {
+ Some(b) => format!("{} MiB", b / MIB),
+ None => "unknown".to_string(),
+ };
// Phase A: every core by default. "Use the machine" belongs here — this is
// the phase that is actually CPU-bound.
@@ -101,29 +137,53 @@ pub fn plan_for(
let per_worker = (bulk_mb.max(1) as u64)
.saturating_mul(WORKER_BYTES_PER_BULK_MIB)
.saturating_mul(MIB);
- let affordable = (safe_zone_bytes / per_worker).max(1) as usize;
+ let affordable = safe_zone_bytes.map(|zone| (zone / per_worker).max(1) as usize);
let mut index_workers = scan_threads;
- if index_workers > affordable {
- notes.push(format!(
- "memory safe zone {} MiB allows {affordable} index workers at --bulk-mb {bulk_mb}, \
- not {index_workers}",
- safe_zone_bytes / MIB
- ));
- index_workers = affordable;
+ if let Some(affordable) = affordable {
+ if index_workers > affordable {
+ if requested_workers.is_some() {
+ // The operator named a number. Say what the machine thinks and
+ // then do as asked (#204: no silent substitution).
+ notes.push(format!(
+ "memory safe zone {} allows {affordable} index workers at --bulk-mb \
+ {bulk_mb}; honouring --workers {index_workers} anyway — lower --workers or \
+ --bulk-mb if this run starts swapping",
+ zone_mib()
+ ));
+ } else {
+ notes.push(format!(
+ "memory safe zone {} allows {affordable} index workers at --bulk-mb \
+ {bulk_mb}, not {index_workers}",
+ zone_mib()
+ ));
+ index_workers = affordable;
+ }
+ }
}
// PDF workers: bounded by the measured ceiling, by the cores available,
- // and by what the safe zone can pay for at 1536 MiB of address space each.
- let pdf_affordable = (safe_zone_bytes / PDF_WORKER_BYTES).max(1) as usize;
+ // and — only where the safe zone is a real reading — by what it can pay for
+ // at 1536 MiB of address space each.
+ let pdf_affordable = safe_zone_bytes.map(|zone| (zone / PDF_WORKER_BYTES).max(1) as usize);
let pdf_default = cores.min(MAX_PDF_WORKERS);
let mut pdf_workers = requested_pdf_workers.unwrap_or(pdf_default).max(1);
- if pdf_workers > pdf_affordable {
- notes.push(format!(
- "memory safe zone {} MiB allows {pdf_affordable} PDF worker(s) at 1536 MiB each, \
- not {pdf_workers}",
- safe_zone_bytes / MIB
- ));
- pdf_workers = pdf_affordable;
+ if let Some(pdf_affordable) = pdf_affordable {
+ if pdf_workers > pdf_affordable {
+ if requested_pdf_workers.is_some() {
+ notes.push(format!(
+ "memory safe zone {} allows {pdf_affordable} PDF worker(s) at 1536 MiB each; \
+ honouring --pdf-workers {pdf_workers} anyway",
+ zone_mib()
+ ));
+ } else {
+ notes.push(format!(
+ "memory safe zone {} allows {pdf_affordable} PDF worker(s) at 1536 MiB each, \
+ not {pdf_workers}",
+ zone_mib()
+ ));
+ pdf_workers = pdf_affordable;
+ }
+ }
}
Plan {
@@ -138,12 +198,15 @@ pub fn plan_for(
impl Plan {
/// One line for the run header and the `--json` summary.
pub fn describe(&self) -> String {
+ let zone = match self.safe_zone_bytes {
+ Some(b) => format!("{} MiB", b / MIB),
+ None => "unknown".to_string(),
+ };
format!(
- "resources: {} scan threads, {} index workers, {} pdf workers, memory safe zone {} MiB ({})",
+ "resources: {} scan threads, {} index workers, {} pdf workers, memory safe zone {zone} ({})",
self.scan_threads,
self.index_workers,
self.pdf_workers,
- self.safe_zone_bytes / MIB,
resource::describe(),
)
}
@@ -157,18 +220,18 @@ mod tests {
#[test]
fn the_default_plan_uses_the_whole_machine_for_the_cpu_bound_phase() {
- let plan = plan_for(12, 8 * GIB, None, None, 8);
+ let plan = plan_for(12, Some(8 * GIB), None, None, 8);
assert_eq!(plan.scan_threads, 12, "phase A is the CPU-bound phase");
assert_eq!(plan.index_workers, 12);
assert_eq!(plan.pdf_workers, 4);
assert!(plan.notes.is_empty());
// The pre-#240 default capped both at 8 on any machine with more cores.
- assert_eq!(plan_for(32, 32 * GIB, None, None, 8).scan_threads, 32);
+ assert_eq!(plan_for(32, Some(32 * GIB), None, None, 8).scan_threads, 32);
}
#[test]
fn an_explicit_request_is_honoured_on_both_phases() {
- let plan = plan_for(32, 32 * GIB, Some(2), Some(1), 8);
+ let plan = plan_for(32, Some(32 * GIB), Some(2), Some(1), 8);
assert_eq!(plan.scan_threads, 2, "--workers must bound phase A too");
assert_eq!(plan.index_workers, 2);
assert_eq!(plan.pdf_workers, 1);
@@ -178,7 +241,7 @@ mod tests {
fn a_small_machine_gets_fewer_workers_and_is_told_why() {
// A 32-core box squeezed into a 512 MiB safe zone (small container),
// asked for the biggest bulk size: 72 MiB per worker buys 7 of them.
- let plan = plan_for(32, 512 * MIB, None, None, 24);
+ let plan = plan_for(32, Some(512 * MIB), None, None, 24);
assert_eq!(plan.index_workers, 7);
assert_eq!(
plan.scan_threads, 32,
@@ -190,15 +253,69 @@ mod tests {
assert!(plan.notes.iter().any(|n| n.contains("PDF worker")));
}
+ /// A machine whose RAM this platform cannot read (Windows, FreeBSD) must be
+ /// planned as *unknown*, not as the smallest machine imaginable. The
+ /// revision this replaces substituted a 1 GiB safe zone here, which bought
+ /// exactly one PDF worker at 1536 MiB each — a 4× cut to PDF extraction on
+ /// every Windows run, charged against an `RLIMIT_AS` that is `cfg(unix)`
+ /// and so is never applied on that platform.
+ #[test]
+ fn an_unprobeable_machine_is_not_treated_as_a_tiny_one() {
+ let plan = plan_for(16, None, None, None, 8);
+ assert_eq!(plan.pdf_workers, 4, "min(cores, MAX_PDF_WORKERS), not 1");
+ assert_eq!(plan.index_workers, 16, "no memory-derived cap to apply");
+ assert_eq!(plan.scan_threads, 16);
+ assert_eq!(plan.safe_zone_bytes, None);
+ assert!(
+ plan.notes.iter().any(|n| n.contains("cannot read")),
+ "the run must say the budget is core-derived: {:?}",
+ plan.notes
+ );
+ // …and the operator's own numbers still win on that platform.
+ let asked = plan_for(16, None, Some(2), Some(1), 8);
+ assert_eq!((asked.index_workers, asked.pdf_workers), (2, 1));
+ // A two-core Windows box still gets the core-derived PDF default.
+ assert_eq!(plan_for(2, None, None, None, 8).pdf_workers, 2);
+ }
+
+ /// The memory heuristic bounds the *default*. A number the operator typed
+ /// is honoured and argued with out loud — silently replacing it is the
+ /// accepted-and-quietly-ignored class from #204, and the trigger is easy to
+ /// hit: the safe zone is computed from a live `MemAvailable` reading, so a
+ /// transient dip on a large box can drop it to its 256 MiB floor.
+ #[test]
+ fn an_explicit_count_survives_a_small_safe_zone_but_is_argued_with() {
+ let plan = plan_for(32, Some(256 * MIB), Some(32), Some(4), 24);
+ assert_eq!(plan.index_workers, 32, "--workers 32 must stay 32");
+ assert_eq!(plan.pdf_workers, 4, "--pdf-workers 4 must stay 4");
+ assert!(
+ plan.notes
+ .iter()
+ .any(|n| n.contains("honouring --workers 32")),
+ "the disagreement must be stated: {:?}",
+ plan.notes
+ );
+ assert!(plan
+ .notes
+ .iter()
+ .any(|n| n.contains("honouring --pdf-workers 4")));
+ // The same machine with no flags still gets the cap, and is told.
+ let defaulted = plan_for(32, Some(256 * MIB), None, None, 24);
+ assert_eq!((defaulted.index_workers, defaulted.pdf_workers), (3, 1));
+ assert!(defaulted.notes.iter().all(|n| !n.contains("honouring")));
+ }
+
#[test]
fn the_plan_is_never_zero_or_absurd() {
for cores in [0usize, 1, 2, 256] {
- for safe in [0u64, 64 * MIB, GIB, 512 * GIB] {
+ for safe in [None, Some(0u64), Some(64 * MIB), Some(GIB), Some(512 * GIB)] {
for bulk in [1usize, 8, 24] {
- let plan = plan_for(cores, safe, None, None, bulk);
- assert!(plan.scan_threads >= 1);
- assert!(plan.index_workers >= 1);
- assert!((1..=MAX_PDF_WORKERS).contains(&plan.pdf_workers));
+ for asked in [None, Some(1usize), Some(MAX_WORKERS)] {
+ let plan = plan_for(cores, safe, asked, None, bulk);
+ assert!(plan.scan_threads >= 1);
+ assert!(plan.index_workers >= 1);
+ assert!((1..=MAX_PDF_WORKERS).contains(&plan.pdf_workers));
+ }
}
}
}
diff --git a/engine/crates/xerj-common/src/resource.rs b/engine/crates/xerj-common/src/resource.rs
index a6d6d6658..b1ab124cc 100644
--- a/engine/crates/xerj-common/src/resource.rs
+++ b/engine/crates/xerj-common/src/resource.rs
@@ -472,20 +472,23 @@ fn sysctl_u64(name: &std::ffi::CStr) -> Option<u64> {
/// Half of RAM as the ceiling follows the same reasoning meilisearch gives for
/// its two-thirds indexing budget (`option.rs:1103-1106`) — a machine that is
/// also serving its owner should not have its page cache evicted by a
-/// background indexer. When `total` is unknown the caller gets
-/// [`FALLBACK_SAFE_ZONE_BYTES`], which is deliberately small: an unknown
-/// machine is not an excuse to take an unknown amount of it.
-pub fn memory_safe_zone_bytes() -> u64 {
- match total_memory_bytes() {
- Some(total) => safe_zone(total, available_memory_bytes()),
- None => FALLBACK_SAFE_ZONE_BYTES,
- }
+/// background indexer.
+///
+/// `None` means **this machine's RAM is not knowable here** (Windows, FreeBSD,
+/// …: [`total_memory_bytes`] has no probe), and callers must read it as "no
+/// memory-derived limit", never as "a small limit". An earlier revision of this
+/// function returned a 1 GiB fallback instead, and the result was that a
+/// Windows box with 64 GB of RAM was budgeted as a 1 GiB machine — a phantom
+/// budget that no code on that platform enforces. meilisearch keeps the same
+/// shape for the same reason: `total_memory_bytes() -> Option<u64>` is `None`
+/// on an unsupported system (`crates/meilisearch/src/option.rs:1135`) and the
+/// derived per-thread budget stays an `Option` all the way down, where `None`
+/// means unbounded rather than minimal
+/// (`crates/milli/src/update/index_documents/helpers/grenad_helpers.rs:122`).
+pub fn memory_safe_zone_bytes() -> Option<u64> {
+ total_memory_bytes().map(|total| safe_zone(total, available_memory_bytes()))
}
-/// The budget used when the machine's RAM cannot be probed at all: 1 GiB,
-/// small enough to be safe on the smallest machine XERJ is expected to run on.
-pub const FALLBACK_SAFE_ZONE_BYTES: u64 = GIB;
-
/// Pure safe-zone rule — the tested half of [`memory_safe_zone_bytes`].
fn safe_zone(total: u64, available: Option<u64>) -> u64 {
let reserve = (total / 8).max(GIB);
@@ -505,7 +508,7 @@ pub fn describe() -> String {
cores(),
mib(total_memory_bytes()),
mib(available_memory_bytes()),
- memory_safe_zone_bytes() / MIB,
+ mib(memory_safe_zone_bytes()),
)
}
@@ -643,7 +646,11 @@ mod tests {
assert!((1..=4096).contains(&cores));
if let Some(total) = total_memory_bytes() {
assert!(total >= 256 * MIB, "implausible total RAM: {total}");
- assert!(memory_safe_zone_bytes() <= total);
+ assert!(memory_safe_zone_bytes().expect("probeable RAM has a safe zone") <= total);
+ } else {
+ // Unknown must stay unknown: a fabricated small budget here is the
+ // Windows PDF-throttling bug this API shape exists to prevent.
+ assert_eq!(memory_safe_zone_bytes(), None);
}
#[cfg(any(target_os = "linux", target_os = "macos"))]
{