Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(autoindex): budget the index before hashing (#2102)

xerj-orgc0091f310502
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9a3d0cdb6..0078c9c56 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -63,6 +63,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
disk. A node that does not report the threshold is unchanged, and
`--dry-run` (which writes nothing) is never refused for disk.
+- `xerj autoindex` counts the walked folder and estimates the index before
+ hashing. The free-space budget reuses that same allocation row (free bytes
+ under the reported flood-stage watermark; the threshold is not guessed),
+ and `--max-index-size` can lower it. When the default plan does not fit,
+ stdout is a JSON plan list and the run exits 4 until `--yes` takes the
+ recommended plan that fits or `--plan` names one; a named plan runs even over
+ the estimate, with a warning, since the estimate errs high. A kind with no measured ratio stays unmeasured and
+ does not refuse the run. A digit-leading `--prefix` is rejected before the
+ walk ([#2057](https://github.com/xerj-org/xerj/issues/2057)). Skipped files
+ are named after the `ok:` line, and a PDF reuse-fallback line says the PDF
+ is still indexed ([#2056](https://github.com/xerj-org/xerj/issues/2056)).
+ A folder already indexed under another prefix is named before the walk
+ ([#2078](https://github.com/xerj-org/xerj/issues/2078)). MCP:
+ `xerj_index_budget` returns the same estimate and does not index.
+
- Playground user dashboards resolve their inherited template for chart data
and panel editing. Cloning a user dashboard retains the underlying template;
missing or cyclic template references keep the blank fallback. Custom
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index 2b7e228c2..86acce186 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -7694,6 +7694,7 @@ dependencies = [
"tokio",
"unicode-segmentation",
"xerj-api",
+ "xerj-autoindex",
"xerj-common",
"xerj-engine",
"xerj-hub",
diff --git a/engine/crates/xerj-autoindex/src/budget.rs b/engine/crates/xerj-autoindex/src/budget.rs
new file mode 100644
index 000000000..2425673a5
--- /dev/null
+++ b/engine/crates/xerj-autoindex/src/budget.rs
@@ -0,0 +1,1553 @@
+//! Disk budget for `xerj autoindex`, decided before hashing.
+//!
+//! The walk has already counted files. Hashing and embedding have not
+//! started. This module estimates how large the index will be, compares that
+//! with the room left under the node's flood-stage watermark, and — when the
+//! default plan does not fit — names a smaller plan and waits for an explicit
+//! yes. It never invents a ratio: a kind with no measured calibration is
+//! reported as unmeasured, and an unmeasured estimate does not refuse a run.
+//!
+//! Free space and the watermark come from [`crate::flood`], the one
+//! `GET /_cat/allocation?format=json` already made before the walk. A second
+//! probe is not issued. A node that omits `disk.flood_stage` does not get a
+//! guessed 95% margin.
+
+use std::collections::BTreeMap;
+use std::io::BufRead;
+use std::path::Path;
+
+use anyhow::Result;
+use serde_json::{json, Value};
+
+use crate::estimate::{count_phrase, human_bytes};
+use crate::extract::code::is_code_ext;
+use crate::flood::{self, Volume};
+use crate::gate::{self, Approval};
+use crate::order;
+use crate::walk::{self, FileEntry};
+
+/// Plans a caller can name with `--plan`. The order is the tie-break: an
+/// earlier plan wins when two that fit keep the same number of source bytes,
+/// so `default` wins whenever it fits.
+pub const PLAN_NAMES: &[&str] = &[
+ "default",
+ "skip-vendored",
+ "skip-data",
+ "code-and-docs",
+ "no-vectors",
+ "cap-1mb",
+ "code-and-docs-no-vectors",
+];
+
+const CAP_1MB: u64 = 1 << 20;
+
+/// What the calibration measured, and where. An empty table is not a zero
+/// estimate. The numbers in [`MEASURED`] are dataset-index store sizes from
+/// one release binary on one host; they are not a promise about this machine.
+const CALIBRATION_NOTE: &str = "per-kind ratios are dataset-index store sizes measured 2026-10-21 with xerj v1.0.0-rc.97 on one 4-core Linux x86_64 host. Code and prose were measured on real repository sources and docs; data, pdf and unindexed kinds on synthetic files. Embeddings were the default lexical feature-hash (the neural embedder was not enabled). Catalog store size is not included. time_s is a rough projection from that host, not this machine. A kind with no row is unmeasured and does not refuse the run";
+
+/// One measured index of one kind.
+#[derive(Debug, Clone, Copy)]
+struct Point {
+ files: u64,
+ source_bytes: u64,
+ index_bytes: u64,
+ wall_ms: u64,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
+enum Model {
+ Code,
+ Prose,
+ Pdf,
+ Data,
+ /// Binary and scientific containers the extractor does not read. Their
+ /// index cost is a catalog row, not a copy of the file.
+ Unindexed,
+ Other,
+}
+
+#[derive(Debug, Clone, Copy)]
+struct Calib {
+ model: Model,
+ semantic: bool,
+ small: Point,
+ large: Point,
+}
+
+/// Release-binary calibration, 2026-10-21.
+///
+/// Binary: `xerj v1.0.0-rc.97`, started `--insecure` (lexical feature-hash
+/// embeddings; `--embed-mode neural` was not passed). Host: Linux 6.12.94+
+/// x86_64, 4 cores, 16387280 KiB RAM.
+///
+/// Code and prose points are real repository content, not synthetic files:
+/// copies of this repository's `.rs` sources (5 files / 291,007 B and
+/// 97 files / 5,096,601 B) and `.md` documents (24 files / 520,410 B and
+/// 169 files / 2,969,566 B). Synthetic distinct-token files under-predicted
+/// those by about 3.5x, so they were replaced. Data, PDF and unindexed points
+/// are still the synthetic runs described below.
+///
+/// `index_bytes` is the sum of `store.size` for that prefix's indices from
+/// `GET /_cat/indices?bytes=b&h=index,store.size` right after the run, before
+/// segment merges settle: that peak is what the disk has to hold. A force-merge
+/// later shrank the real-content stores by roughly 20-40%, so this errs high.
+/// `autoindex-catalog` is omitted: its store size moved by tens of KB in both
+/// directions between runs. For code and prose the per-file intercept comes
+/// out negative, so the fitter uses the steeper of the two measured ratios.
+/// `wall_ms` for code and prose is the wall time of the whole `xerj autoindex`
+/// command, so it still contains the fixed per-run cost.
+///
+/// Synthetic points (data, PDF, unindexed): each is one `xerj autoindex` of
+/// files with distinct tokens. `.npy` files were recorded as junk
+/// (`files_junk` equal to the file count) and their dataset-index store delta
+/// was 0. The two PDF points are about 1 KB and 10 KB; their per-byte slope is
+/// negative, so the fitter uses the steeper of the two measured ratios.
+static MEASURED: &[Calib] = &[
+ Calib {
+ model: Model::Code,
+ semantic: true,
+ small: Point {
+ files: 5,
+ source_bytes: 291_007,
+ index_bytes: 1_337_452,
+ wall_ms: 2_279,
+ },
+ large: Point {
+ files: 97,
+ source_bytes: 5_096_601,
+ index_bytes: 15_252_875,
+ wall_ms: 5_886,
+ },
+ },
+ Calib {
+ model: Model::Code,
+ semantic: false,
+ small: Point {
+ files: 5,
+ source_bytes: 291_007,
+ index_bytes: 574_967,
+ wall_ms: 1_740,
+ },
+ large: Point {
+ files: 97,
+ source_bytes: 5_096_601,
+ index_bytes: 7_367_909,
+ wall_ms: 5_139,
+ },
+ },
+ Calib {
+ model: Model::Prose,
+ semantic: true,
+ small: Point {
+ files: 24,
+ source_bytes: 520_410,
+ index_bytes: 4_656_359,
+ wall_ms: 1_749,
+ },
+ large: Point {
+ files: 169,
+ source_bytes: 2_969_566,
+ index_bytes: 15_859_387,
+ wall_ms: 5_308,
+ },
+ },
+ Calib {
+ model: Model::Prose,
+ semantic: false,
+ small: Point {
+ files: 24,
+ source_bytes: 520_410,
+ index_bytes: 748_083,
+ wall_ms: 3_796,
+ },
+ large: Point {
+ files: 169,
+ source_bytes: 2_969_566,
+ index_bytes: 2_805_069,
+ wall_ms: 3_646,
+ },
+ },
+ Calib {
+ model: Model::Data,
+ semantic: true,
+ small: Point {
+ files: 1,
+ source_bytes: 20_039,
+ index_bytes: 63_682,
+ wall_ms: 600,
+ },
+ large: Point {
+ files: 4,
+ source_bytes: 320_156,
+ index_bytes: 681_267,
+ wall_ms: 900,
+ },
+ },
+ Calib {
+ model: Model::Data,
+ semantic: false,
+ small: Point {
+ files: 1,
+ source_bytes: 20_039,
+ index_bytes: 63_682,
+ wall_ms: 600,
+ },
+ large: Point {
+ files: 4,
+ source_bytes: 320_156,
+ index_bytes: 669_741,
+ wall_ms: 900,
+ },
+ },
+ Calib {
+ model: Model::Other,
+ semantic: true,
+ small: Point {
+ files: 1,
+ source_bytes: 4_000,
+ index_bytes: 19_611,
+ wall_ms: 600,
+ },
+ large: Point {
+ files: 8,
+ source_bytes: 160_000,
+ index_bytes: 169_502,
+ wall_ms: 600,
+ },
+ },
+ Calib {
+ model: Model::Other,
+ semantic: false,
+ small: Point {
+ files: 1,
+ source_bytes: 4_000,
+ index_bytes: 19_611,
+ wall_ms: 600,
+ },
+ large: Point {
+ files: 8,
+ source_bytes: 160_000,
+ index_bytes: 169_134,
+ wall_ms: 600,
+ },
+ },
+ Calib {
+ model: Model::Pdf,
+ semantic: true,
+ small: Point {
+ files: 1,
+ source_bytes: 1_082,
+ index_bytes: 14_542,
+ wall_ms: 0,
+ },
+ large: Point {
+ files: 4,
+ source_bytes: 10_336,
+ index_bytes: 29_218,
+ wall_ms: 100,
+ },
+ },
+ Calib {
+ model: Model::Pdf,
+ semantic: false,
+ small: Point {
+ files: 1,
+ source_bytes: 1_082,
+ index_bytes: 14_542,
+ wall_ms: 0,
+ },
+ large: Point {
+ files: 4,
+ source_bytes: 10_336,
+ index_bytes: 29_180,
+ wall_ms: 100,
+ },
+ },
+ Calib {
+ model: Model::Unindexed,
+ semantic: true,
+ small: Point {
+ files: 1,
+ source_bytes: 4_000,
+ index_bytes: 0,
+ wall_ms: 0,
+ },
+ large: Point {
+ files: 8,
+ source_bytes: 320_000,
+ index_bytes: 0,
+ wall_ms: 0,
+ },
+ },
+ Calib {
+ model: Model::Unindexed,
+ semantic: false,
+ small: Point {
+ files: 1,
+ source_bytes: 4_000,
+ index_bytes: 0,
+ wall_ms: 0,
+ },
+ large: Point {
+ files: 8,
+ source_bytes: 320_000,
+ index_bytes: 0,
+ wall_ms: 0,
+ },
+ },
+];
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
+enum Kind {
+ Code,
+ Prose,
+ Pdf,
+ Data,
+ Binary,
+ Other,
+}
+
+impl Kind {
+ fn name(self) -> &'static str {
+ match self {
+ Kind::Code => "code",
+ Kind::Prose => "prose",
+ Kind::Pdf => "pdf",
+ Kind::Data => "data",
+ Kind::Binary => "binary",
+ Kind::Other => "other",
+ }
+ }
+}
+
+struct Fact {
+ rel: String,
+ size: u64,
+ kind: Kind,
+ model: Model,
+ vendored: bool,
+}
+
+/// What the caller already decided, plus the walked files and the volume from
+/// the flood-stage probe.
+pub(crate) struct Request<'a> {
+ pub root: &'a str,
+ pub files: &'a [FileEntry],
+ pub volume: Option<Volume>,
+ pub max_index_bytes: Option<u64>,
+ pub no_semantic: bool,
+ pub plan: Option<&'a str>,
+ pub approve: Option<Approval>,
+ pub duplicate_prefixes: &'a [String],
+}
+
+/// A line the operator typed at the budget prompt.
+#[derive(Debug, PartialEq, Eq)]
+pub enum Choice {
+ /// Nobody was asked, or the reply was empty.
+ Absent,
+ Plan(String),
+ Cancel,
+}
+
+enum Preselect {
+ None,
+ Plan(String),
+ Cancel,
+ Refuse(&'static str),
+}
+
+struct PlanEst {
+ name: &'static str,
+ files: u64,
+ source_bytes: u64,
+ index_bytes: Option<u64>,
+ time_s: Option<f64>,
+ drops: Vec<String>,
+ no_semantic: bool,
+ keep: Option<Vec<String>>,
+ fits: Option<bool>,
+}
+
+pub struct Assessment {
+ pub body: Value,
+ /// Lines to show before a prompt, or with a refusal.
+ pub lines: Vec<String>,
+ /// One line when the run continues. Empty when there is nothing new to say.
+ pub note: String,
+ pub needs_choice: bool,
+ pub plan_names: Vec<String>,
+ preselect: Preselect,
+ plans: Vec<PlanEst>,
+}
+
+pub enum Outcome {
+ Continue {
+ /// `None` keeps every walked file.
+ keep: Option<Vec<String>>,
+ no_semantic: bool,
+ note: String,
+ },
+ /// `body` is `Null` for a cancel, which prints no JSON.
+ Stop { code: i32, body: Value },
+}
+
+/// Room the index may occupy.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum DiskBudget {
+ /// Bytes until `disk.avail` would cross the flood-stage reserve.
+ UntilFlood {
+ bytes: u64,
+ flood_percent: u8,
+ avail: u64,
+ total: u64,
+ },
+ /// Watermark off (`disk.flood_stage` 0, or the threshold disabled).
+ WatermarkOff { bytes: u64, avail: u64 },
+}
+
+/// Index budget from the allocation row `#2077` already parsed.
+///
+/// `None` when the node did not report a flood threshold, or reported one
+/// but not the total the reserve is a fraction of. The missing number is
+/// not filled in from the default 95.
+pub(crate) fn disk_budget(volume: &Volume) -> Option<DiskBudget> {
+ let avail = volume.avail?;
+ let flood = volume.flood_stage?;
+ if !volume.threshold_enabled || flood == 0 {
+ return Some(DiskBudget::WatermarkOff {
+ bytes: avail,
+ avail,
+ });
+ }
+ let total = volume.total?;
+ let headroom = u64::from(100u8.saturating_sub(flood));
+ let reserve = ((total as u128) * u128::from(headroom) / 100) as u64;
+ Some(DiskBudget::UntilFlood {
+ bytes: avail.saturating_sub(reserve),
+ flood_percent: flood,
+ avail,
+ total,
+ })
+}
+
+/// Parse `--max-index-size`. A bare integer is bytes. `KB`/`MB`/`GB` (and the
+/// one-letter forms) are powers of 1024. Zero is rejected.
+pub fn parse_byte_size(raw: &str) -> Result<u64, String> {
+ let raw = raw.trim();
+ if raw.is_empty() {
+ return Err("--max-index-size needs a size, for example 2GB".into());
+ }
+ let split = raw.find(|c: char| !c.is_ascii_digit());
+ let (number, suffix) = match split {
+ Some(i) => (&raw[..i], raw[i..].trim()),
+ None => (raw, ""),
+ };
+ if number.is_empty() {
+ return Err(format!(
+ "--max-index-size needs a size in bytes or with a KB/MB/GB suffix, got {raw}"
+ ));
+ }
+ let n: u64 = number.parse().map_err(|_| {
+ format!("--max-index-size needs a size in bytes or with a KB/MB/GB suffix, got {raw}")
+ })?;
+ let mult: u64 = match suffix.to_ascii_lowercase().as_str() {
+ "" | "b" => 1,
+ "k" | "kb" | "kib" => 1024,
+ "m" | "mb" | "mib" => 1024 * 1024,
+ "g" | "gb" | "gib" => 1024 * 1024 * 1024,
+ _ => {
+ return Err(format!(
+ "--max-index-size suffix must be B, KB, MB, or GB, got {suffix}"
+ ))
+ }
+ };
+ let bytes = n
+ .checked_mul(mult)
+ .ok_or_else(|| format!("--max-index-size {raw} overflows"))?;
+ if bytes == 0 {
+ return Err("--max-index-size must be greater than 0".into());
+ }
+ Ok(bytes)
+}
+
+pub(crate) fn assess(req: &Request<'_>) -> Assessment {
+ let facts: Vec<Fact> = req.files.iter().map(fact_of).collect();
+ let semantic_default = !req.no_semantic;
+ let (budget_bytes, budget_basis, disk_json) = budget_view(req.volume, req.max_index_bytes);
+ let plans = PLAN_NAMES
+ .iter()
+ .filter_map(|name| estimate_plan(name, &facts, semantic_default, budget_bytes))
+ .collect::<Vec<_>>();
+ let plan_names: Vec<String> = plans.iter().map(|p| p.name.to_string()).collect();
+ let recommended = recommended_plan(&plans).map(|p| p.name);
+ let default_fits = plans
+ .iter()
+ .find(|p| p.name == "default")
+ .and_then(|p| p.fits);
+
+ let mut preselect = Preselect::None;
+ let mut needs_choice = false;
+ if req.approve == Some(Approval::Cancel) {
+ preselect = Preselect::Cancel;
+ } else if let Some(name) = req.plan {
+ match plans.iter().find(|p| p.name == name) {
+ // A plan named on the command line is the override when the
+ // estimate is wrong: it runs even if the estimate says it does
+ // not fit, with a warning. The flood-stage guard still applies.
+ Some(_) => preselect = Preselect::Plan(name.to_string()),
+ // `no-vectors` is omitted when the run is already `--no-semantic`:
+ // it is the same plan as `default`.
+ None if PLAN_NAMES.contains(&name) => {
+ preselect = Preselect::Plan("default".to_string())
+ }
+ None => preselect = Preselect::Refuse("unknown"),
+ }
+ } else if default_fits == Some(false) {
+ let accepted = matches!(req.approve, Some(Approval::Proceed | Approval::Fast));
+ if accepted {
+ match recommended {
+ Some(name) => preselect = Preselect::Plan(name.to_string()),
+ None => preselect = Preselect::Refuse("none"),
+ }
+ } else {
+ needs_choice = true;
+ }
+ }
+
+ let note = if needs_choice || matches!(preselect, Preselect::Refuse(_) | Preselect::Cancel) {
+ String::new()
+ } else if let Preselect::Plan(name) = &preselect {
+ plans
+ .iter()
+ .find(|p| p.name == name)
+ .map(|p| continue_note(p, budget_bytes, &budget_basis))
+ .unwrap_or_default()
+ } else if default_fits == Some(true) {
+ plans
+ .iter()
+ .find(|p| p.name == "default")
+ .map(|p| {
+ format!(
+ "{} under the {} budget ({budget_basis})",
+ plan_size_phrase(p),
+ budget_bytes
+ .map(human_bytes)
+ .unwrap_or_else(|| "unset".into())
+ )
+ })
+ .unwrap_or_default()
+ } else {
+ String::new()
+ };
+
+ let lines = summary_lines(&facts, &plans, budget_bytes, &budget_basis, recommended);
+ let body = json!({
+ "xerj": "autoindex-budget",
+ "root": req.root,
+ "classification": "by extension, before content sniff",
+ "files": facts.len(),
+ "bytes": facts.iter().map(|f| f.size).sum::<u64>(),
+ "by_kind": counts_by(&facts, |f| f.kind.name()),
+ "by_folder": counts_by(&facts, |f| top_folder(&f.rel)),
+ "disk": disk_json,
+ "budget_bytes": budget_bytes,
+ "budget_basis": budget_basis,
+ "max_index_bytes": req.max_index_bytes,
+ "decision_required": needs_choice || matches!(preselect, Preselect::Refuse(_)),
+ "plans": plans.iter().map(plan_json).collect::<Vec<_>>(),
+ "recommended": recommended,
+ "duplicate_prefixes": req.duplicate_prefixes,
+ "accept": "re-run with --plan <name>, or --yes to take the recommended plan that fits. A named plan runs even when the estimate says it does not fit (the estimate errs high). --approve cancel indexes nothing",
+ "calibration": CALIBRATION_NOTE,
+ });
+
+ Assessment {
+ body,
+ lines,
+ note,
+ needs_choice,
+ plan_names,
+ preselect,
+ plans,
+ }
+}
+
+pub fn resolve(assessment: &Assessment, choice: Choice) -> Outcome {
+ if matches!(assessment.preselect, Preselect::Cancel) || matches!(choice, Choice::Cancel) {
+ return Outcome::Stop {
+ code: 0,
+ body: Value::Null,
+ };
+ }
+ if let Preselect::Refuse(why) = assessment.preselect {
+ let mut body = assessment.body.clone();
+ body["exit_code"] = json!(gate::EXIT_NEEDS_DECISION);
+ body["refused"] = json!(match why {
+ "none" => "no plan fits the budget",
+ _ => "unknown plan",
+ });
+ return Outcome::Stop {
+ code: gate::EXIT_NEEDS_DECISION,
+ body,
+ };
+ }
+ let name = match &assessment.preselect {
+ Preselect::Plan(name) => Some(name.as_str()),
+ Preselect::None if assessment.needs_choice => match choice {
+ Choice::Plan(ref name) => Some(name.as_str()),
+ Choice::Absent => {
+ let mut body = assessment.body.clone();
+ body["exit_code"] = json!(gate::EXIT_NEEDS_DECISION);
+ return Outcome::Stop {
+ code: gate::EXIT_NEEDS_DECISION,
+ body,
+ };
+ }
+ Choice::Cancel => {
+ return Outcome::Stop {
+ code: 0,
+ body: Value::Null,
+ };
+ }
+ },
+ _ => None,
+ };
+ let Some(name) = name else {
+ return Outcome::Continue {
+ keep: None,
+ no_semantic: false,
+ note: assessment.note.clone(),
+ };
+ };
+ match assessment.plans.iter().find(|p| p.name == name) {
+ Some(plan) if plan.keep.as_ref().is_some_and(|kept| kept.is_empty()) => {
+ let mut body = assessment.body.clone();
+ body["exit_code"] = json!(gate::EXIT_NEEDS_DECISION);
+ body["refused"] = json!(format!(
+ "--plan {} keeps no files, so nothing was hashed. Pick another plan, or cancel",
+ plan.name
+ ));
+ Outcome::Stop {
+ code: gate::EXIT_NEEDS_DECISION,
+ body,
+ }
+ }
+ Some(plan) => Outcome::Continue {
+ keep: plan.keep.clone(),
+ no_semantic: plan.no_semantic,
+ note: if assessment.note.is_empty() {
+ continue_note(plan, budget_bytes_of(&assessment.body), "")
+ } else {
+ assessment.note.clone()
+ },
+ },
+ None => {
+ let mut body = assessment.body.clone();
+ body["exit_code"] = json!(gate::EXIT_NEEDS_DECISION);
+ body["refused"] = json!(format!(
+ "unknown plan {name}; choices: {}",
+ assessment.plan_names.join(", ")
+ ));
+ Outcome::Stop {
+ code: gate::EXIT_NEEDS_DECISION,
+ body,
+ }
+ }
+ }
+}
+
+fn budget_bytes_of(body: &Value) -> Option<u64> {
+ body.get("budget_bytes").and_then(Value::as_u64)
+}
+
+pub fn read_choice<R: BufRead>(
+ input: &mut R,
+ names: &[String],
+ mut echo: impl FnMut(&str),
+) -> Choice {
+ for _ in 0..3 {
+ echo(" plan name, or cancel: ");
+ let mut line = String::new();
+ match input.read_line(&mut line) {
+ Ok(0) | Err(_) => return Choice::Absent,
+ Ok(_) => {}
+ }
+ let typed = line.trim();
+ if typed.eq_ignore_ascii_case("cancel") || typed.eq_ignore_ascii_case("c") {
+ return Choice::Cancel;
+ }
+ if let Some(name) = names.iter().find(|n| n.eq_ignore_ascii_case(typed)) {
+ return Choice::Plan(name.clone());
+ }
+ echo(&format!(
+ " {typed:?} is not a plan. choices: {}",
+ names.join(", ")
+ ));
+ }
+ Choice::Absent
+}
+
+/// Walk `root`, read the node's allocation once, and return the budget JSON.
+/// Does not index. A node that is down still returns the file counts; `disk`
+/// is null and the basis says the probe did not answer.
+pub fn assess_path(
+ root: &Path,
+ url: &str,
+ auth: Option<&str>,
+ prefix: &str,
+ max_index_bytes: Option<u64>,
+ no_semantic: bool,
+) -> Result<Value> {
+ let (files, _) = walk::walk_reporting_opts(
+ root,
+ false,
+ false,
+ crate::ignore_rules::IgnoreOptions::default(),
+ )?;
+ let root_str = root
+ .canonicalize()
+ .unwrap_or_else(|_| root.to_path_buf())
+ .to_string_lossy()
+ .to_string();
+ let (volume, duplicates) =
+ match crate::esclient::Es::with_bulk_timeout(url, auth.map(str::to_string), 30) {
+ Ok(es) => {
+ let volume = flood::probe(&es).volume;
+ let duplicates = other_prefixes(&es, &root_str, prefix).unwrap_or_default();
+ (volume, duplicates)
+ }
+ Err(_) => (None, Vec::new()),
+ };
+ let assessment = assess(&Request {
+ root: &root_str,
+ files: &files,
+ volume,
+ max_index_bytes,
+ no_semantic,
+ plan: None,
+ approve: None,
+ duplicate_prefixes: &duplicates,
+ });
+ Ok(assessment.body)
+}
+
+fn other_prefixes(es: &crate::esclient::Es, root: &str, prefix: &str) -> Result<Vec<String>> {
+ let body = json!({
+ "size": 500,
+ "query": {"bool": {"filter": [{"term": {"doc_kind": "run"}}]}}
+ });
+ let Some(page) = es.search_present(crate::catalog::CATALOG_INDEX, &body)? else {
+ return Ok(Vec::new());
+ };
+ let runs = page
+ .pointer("/hits/hits")
+ .and_then(Value::as_array)
+ .map(|hits| {
+ hits.iter()
+ .filter_map(|hit| hit.get("_source").cloned())
+ .collect::<Vec<_>>()
+ })
+ .unwrap_or_default();
+ Ok(crate::catalog::other_prefixes_for_root(&runs, root, prefix))
+}
+
+fn budget_view(
+ volume: Option<Volume>,
+ max_index_bytes: Option<u64>,
+) -> (Option<u64>, String, Value) {
+ let disk = volume.as_ref().and_then(disk_budget);
+ let (disk_bytes, basis, disk_json) = match (&volume, &disk) {
+ (Some(vol), Some(DiskBudget::UntilFlood { bytes, flood_percent, avail, total })) => {
+ let basis = format!(
+ "free {} minus the {}% flood-stage reserve on {}",
+ human_bytes(*avail),
+ flood_percent,
+ human_bytes(*total)
+ );
+ (
+ Some(*bytes),
+ basis,
+ json!({
+ "total": total,
+ "avail": avail,
+ "used_percent": vol.used_percent,
+ "flood_stage": flood_percent,
+ "threshold_enabled": vol.threshold_enabled,
+ "reserve_bytes": avail.saturating_sub(*bytes),
+ "budget_bytes": bytes,
+ }),
+ )
+ }
+ (Some(vol), Some(DiskBudget::WatermarkOff { bytes, avail })) => (
+ Some(*bytes),
+ "flood-stage watermark is off; budget is the free space".to_string(),
+ json!({
+ "total": vol.total,
+ "avail": avail,
+ "used_percent": vol.used_percent,
+ "flood_stage": vol.flood_stage,
+ "threshold_enabled": vol.threshold_enabled,
+ "watermark": "off",
+ "budget_bytes": bytes,
+ }),
+ ),
+ (Some(_), None) => (
+ None,
+ "the node did not report a flood-stage threshold and a total, so no disk margin was applied".to_string(),
+ json!(null_disk(volume.as_ref())),
+ ),
+ (None, _) => (
+ None,
+ "the node did not report disk allocation, so no disk margin was applied".to_string(),
+ Value::Null,
+ ),
+ };
+ let (bytes, basis) = match (disk_bytes, max_index_bytes) {
+ (Some(disk), Some(cap)) => (
+ Some(disk.min(cap)),
+ format!(
+ "{basis}; capped at {} by --max-index-size",
+ human_bytes(cap)
+ ),
+ ),
+ (None, Some(cap)) => (
+ Some(cap),
+ format!("--max-index-size {} ({basis})", human_bytes(cap)),
+ ),
+ (disk, None) => (disk, basis),
+ };
+ (bytes, basis, disk_json)
+}
+
+fn null_disk(volume: Option<&Volume>) -> Value {
+ match volume {
+ Some(vol) => json!({
+ "total": vol.total,
+ "avail": vol.avail,
+ "used_percent": vol.used_percent,
+ "flood_stage": vol.flood_stage,
+ "threshold_enabled": vol.threshold_enabled,
+ }),
+ None => Value::Null,
+ }
+}
+
+fn estimate_plan(
+ name: &str,
+ facts: &[Fact],
+ semantic_default: bool,
+ budget_bytes: Option<u64>,
+) -> Option<PlanEst> {
+ let force_no_semantic = matches!(name, "no-vectors" | "code-and-docs-no-vectors");
+ let no_semantic = force_no_semantic || !semantic_default;
+ if force_no_semantic && !semantic_default {
+ // Already indexing without vectors; this plan is the same as another.
+ return None;
+ }
+ let mut kept = Vec::new();
+ let mut dropped_kinds: BTreeMap<&'static str, u64> = BTreeMap::new();
+ let mut dropped_vendored = 0u64;
+ let mut dropped_cap = 0u64;
+ for fact in facts {
+ if keep(name, fact) {
+ kept.push(fact);
+ } else if fact.vendored && name == "skip-vendored" {
+ dropped_vendored += 1;
+ } else if fact.size > CAP_1MB && name == "cap-1mb" {
+ dropped_cap += 1;
+ } else {
+ *dropped_kinds.entry(fact.kind.name()).or_default() += 1;
+ }
+ }
+ let mut drops = Vec::new();
+ if dropped_vendored > 0 {
+ drops.push(format!(
+ "{} vendored",
+ count_phrase(dropped_vendored, "path", "paths")
+ ));
+ }
+ for (kind, n) in &dropped_kinds {
+ drops.push(format!("{n} {kind}"));
+ }
+ if dropped_cap > 0 {
+ drops.push(format!(
+ "{} over 1 MB",
+ count_phrase(dropped_cap, "file", "files")
+ ));
+ }
+ if force_no_semantic {
+ drops.push("vectors (semantic_text)".to_string());
+ }
+ let (index_bytes, time_s) = project_kept(&kept, !no_semantic);
+ let fits = match (index_bytes, budget_bytes) {
+ (Some(index), Some(budget)) => Some(index <= budget),
+ _ => None,
+ };
+ let source_bytes = kept.iter().map(|f| f.size).sum();
+ let keep_rels = if kept.len() == facts.len() && !force_no_semantic {
+ None
+ } else {
+ Some(kept.iter().map(|f| f.rel.clone()).collect())
+ };
+ Some(PlanEst {
+ name: PLAN_NAMES
+ .iter()
+ .copied()
+ .find(|n| *n == name)
+ .unwrap_or("default"),
+ files: kept.len() as u64,
+ source_bytes,
+ index_bytes,
+ time_s,
+ drops,
+ no_semantic,
+ keep: keep_rels,
+ fits,
+ })
+}
+
+fn keep(name: &str, fact: &Fact) -> bool {
+ match name {
+ "skip-vendored" => !fact.vendored,
+ "skip-data" => !matches!(fact.kind, Kind::Data | Kind::Binary),
+ "code-and-docs" | "code-and-docs-no-vectors" => {
+ matches!(fact.kind, Kind::Code | Kind::Prose | Kind::Pdf)
+ }
+ "cap-1mb" => fact.size <= CAP_1MB,
+ _ => true,
+ }
+}
+
+fn project_kept(kept: &[&Fact], semantic: bool) -> (Option<u64>, Option<f64>) {
+ if MEASURED.is_empty() {
+ return (None, None);
+ }
+ let mut index = 0u64;
+ let mut wall_ms = 0u64;
+ let mut timed = false;
+ let mut measured_all = true;
+ let mut groups: BTreeMap<Model, (u64, u64)> = BTreeMap::new();
+ for fact in kept {
+ let entry = groups.entry(fact.model).or_default();
+ entry.0 += 1;
+ entry.1 += fact.size;
+ }
+ for (model, (files, source)) in groups {
+ let Some(cal) = MEASURED
+ .iter()
+ .find(|c| c.model == model && c.semantic == semantic)
+ else {
+ measured_all = false;
+ continue;
+ };
+ index = index.saturating_add(project_points(&cal.small, &cal.large, files, source));
+ let sample = if cal.large.source_bytes >= cal.small.source_bytes {
+ &cal.large
+ } else {
+ &cal.small
+ };
+ if sample.source_bytes > 0 && sample.wall_ms > 0 {
+ wall_ms = wall_ms.saturating_add(
+ ((source as u128) * u128::from(sample.wall_ms) / u128::from(sample.source_bytes))
+ as u64,
+ );
+ timed = true;
+ }
+ }
+ let index = measured_all.then_some(index);
+ let time_s = (timed && measured_all).then_some((wall_ms as f64) / 1000.0);
+ (index, time_s)
+}
+
+/// `index_bytes ≈ per_file * files + source_bytes * num / den`, fitted to the
+/// two measured points. A negative per-file intercept is clamped to 0 and the
+/// slope becomes the steeper of the two measured index/source ratios, so the
+/// projection stays at or above both measurements.
+fn project_points(a: &Point, b: &Point, files: u64, source: u64) -> u64 {
+ let fit = fit_points(a, b);
+ fit.per_file
+ .saturating_mul(files)
+ .saturating_add(mul_div(source, fit.num, fit.den))
+}
+
+struct Fit {
+ per_file: u64,
+ num: u64,
+ den: u64,
+}
+
+fn fit_points(a: &Point, b: &Point) -> Fit {
+ let f1 = a.files as i128;
+ let s1 = a.source_bytes as i128;
+ let i1 = a.index_bytes as i128;
+ let f2 = b.files as i128;
+ let s2 = b.source_bytes as i128;
+ let i2 = b.index_bytes as i128;
+ let pden = f1 * s2 - f2 * s1;
+ let pnum = i1 * s2 - i2 * s1;
+ if pden == 0 {
+ return steeper_ratio(a, b);
+ }
+ // p and pden have opposite signs when the intercept is negative.
+ if pnum.signum() * pden.signum() < 0 {
+ return steeper_ratio(a, b);
+ }
+ let per_file = (pnum / pden) as u64;
+ let rnum = i1 * pden - pnum * f1;
+ let rden = s1 * pden;
+ if rden == 0 || rnum.signum() * rden.signum() < 0 {
+ return steeper_ratio(a, b);
+ }
+ let (num, den) = reduce(rnum.unsigned_abs(), rden.unsigned_abs());
+ Fit { per_file, num, den }
+}
+
+fn steeper_ratio(a: &Point, b: &Point) -> Fit {
+ let pick = |p: &Point| -> Option<(u128, u128)> {
+ if p.source_bytes == 0 {
+ None
+ } else {
+ Some((u128::from(p.index_bytes), u128::from(p.source_bytes)))
+ }
+ };
+ let best = match (pick(a), pick(b)) {
+ (Some((i1, s1)), Some((i2, s2))) => {
+ if i1 * s2 >= i2 * s1 {
+ Some((i1, s1))
+ } else {
+ Some((i2, s2))
+ }
+ }
+ (Some(r), None) | (None, Some(r)) => Some(r),
+ (None, None) => None,
+ };
+ match best {
+ Some((num, den)) => {
+ let (num, den) = reduce(num, den);
+ Fit {
+ per_file: 0,
+ num,
+ den,
+ }
+ }
+ None => {
+ let files = a.files.max(b.files).max(1);
+ let index = a.index_bytes.max(b.index_bytes);
+ Fit {
+ per_file: index / files,
+ num: 0,
+ den: 1,
+ }
+ }
+ }
+}
+
+fn reduce(num: u128, den: u128) -> (u64, u64) {
+ if den == 0 {
+ return (0, 1);
+ }
+ let g = gcd(num, den);
+ ((num / g) as u64, (den / g) as u64)
+}
+
+fn gcd(mut a: u128, mut b: u128) -> u128 {
+ while b != 0 {
+ let r = a % b;
+ a = b;
+ b = r;
+ }
+ a
+}
+
+fn mul_div(value: u64, num: u64, den: u64) -> u64 {
+ if den == 0 {
+ return 0;
+ }
+ ((u128::from(value) * u128::from(num)) / u128::from(den)) as u64
+}
+
+fn recommended_plan(plans: &[PlanEst]) -> Option<&PlanEst> {
+ let mut best: Option<&PlanEst> = None;
+ for plan in plans {
+ if plan.fits != Some(true) {
+ continue;
+ }
+ best = Some(match best {
+ None => plan,
+ Some(cur) if plan.source_bytes > cur.source_bytes => plan,
+ Some(cur) => cur,
+ });
+ }
+ best
+}
+
+fn plan_json(plan: &PlanEst) -> Value {
+ json!({
+ "name": plan.name,
+ "fits": plan.fits,
+ "index_bytes": plan.index_bytes,
+ "source_bytes": plan.source_bytes,
+ "files": plan.files,
+ "drops": plan.drops,
+ "no_semantic": plan.no_semantic,
+ "time_s": plan.time_s,
+ "time_basis": plan.time_s.map(|_| "rough projection from the calibration host, not this machine"),
+ })
+}
+
+fn plan_size_phrase(plan: &PlanEst) -> String {
+ match plan.index_bytes {
+ Some(bytes) => format!(
+ "estimated index {} for {} ({} source)",
+ human_bytes(bytes),
+ count_phrase(plan.files, "file", "files"),
+ human_bytes(plan.source_bytes)
+ ),
+ None => format!(
+ "index size unmeasured for {} ({} source)",
+ count_phrase(plan.files, "file", "files"),
+ human_bytes(plan.source_bytes)
+ ),
+ }
+}
+
+fn continue_note(plan: &PlanEst, budget: Option<u64>, basis: &str) -> String {
+ let budget = budget
+ .map(human_bytes)
+ .unwrap_or_else(|| "an unknown".into());
+ let drops = if plan.drops.is_empty() {
+ String::new()
+ } else {
+ format!(". drops: {}", plan.drops.join(", "))
+ };
+ let basis = if basis.is_empty() {
+ String::new()
+ } else {
+ format!(" ({basis})")
+ };
+ let over = if plan.fits == Some(false) {
+ ". warning: the estimate is over the budget; running because the plan was named. \
+ The node's flood-stage watermark still stops writes"
+ } else {
+ ""
+ };
+ format!(
+ "--plan {} keeps {}{drops}. {} Budget {budget}{basis}{over}",
+ plan.name,
+ count_phrase(plan.files, "file", "files"),
+ plan_size_phrase(plan)
+ )
+}
+
+fn summary_lines(
+ facts: &[Fact],
+ plans: &[PlanEst],
+ budget: Option<u64>,
+ basis: &str,
+ recommended: Option<&str>,
+) -> Vec<String> {
+ let bytes = facts.iter().map(|f| f.size).sum::<u64>();
+ let mut lines = vec![format!(
+ "budget: {} ({}), {}",
+ count_phrase(facts.len() as u64, "file", "files"),
+ human_bytes(bytes),
+ kind_phrase(facts)
+ )];
+ if let Some(default) = plans.iter().find(|p| p.name == "default") {
+ let fit = match default.fits {
+ Some(true) => "fits",
+ Some(false) => "does not fit",
+ None => "is not compared (unmeasured, or no budget)",
+ };
+ lines.push(format!(
+ "budget: default plan {fit}: {}. budget {} ({basis})",
+ plan_size_phrase(default),
+ budget.map(human_bytes).unwrap_or_else(|| "unset".into())
+ ));
+ }
+ for plan in plans
+ .iter()
+ .filter(|p| p.name != "default" && p.fits != Some(false))
+ {
+ let mark = if Some(plan.name) == recommended {
+ "recommended"
+ } else {
+ "option"
+ };
+ let drop = if plan.drops.is_empty() {
+ String::new()
+ } else {
+ format!(" — drops {}", plan.drops.join(", "))
+ };
+ lines.push(format!(
+ " {mark} {}: {}{drop}",
+ plan.name,
+ plan_size_phrase(plan)
+ ));
+ }
+ lines
+}
+
+fn kind_phrase(facts: &[Fact]) -> String {
+ let mut parts = Vec::new();
+ for kind in [
+ Kind::Code,
+ Kind::Prose,
+ Kind::Pdf,
+ Kind::Data,
+ Kind::Binary,
+ Kind::Other,
+ ] {
+ let n = facts.iter().filter(|f| f.kind == kind).count();
+ if n > 0 {
+ parts.push(format!("{n} {}", kind.name()));
+ }
+ }
+ if parts.is_empty() {
+ "empty".to_string()
+ } else {
+ parts.join(", ")
+ }
+}
+
+fn counts_by(facts: &[Fact], key: impl Fn(&Fact) -> &str) -> Value {
+ let mut map: BTreeMap<&str, (u64, u64)> = BTreeMap::new();
+ for fact in facts {
+ let entry = map.entry(key(fact)).or_default();
+ entry.0 += 1;
+ entry.1 += fact.size;
+ }
+ let mut out = serde_json::Map::new();
+ for (k, (files, bytes)) in map {
+ out.insert(k.to_string(), json!({"files": files, "bytes": bytes}));
+ }
+ Value::Object(out)
+}
+
+fn top_folder(rel: &str) -> &str {
+ match rel.split_once('/') {
+ Some((dir, rest)) if !dir.is_empty() && !rest.is_empty() => dir,
+ _ => "(root)",
+ }
+}
+
+fn fact_of(file: &FileEntry) -> Fact {
+ let (kind, model) = classify(&file.rel);
+ Fact {
+ rel: file.rel.clone(),
+ size: file.size,
+ kind,
+ model,
+ vendored: order::vendored_reason(&file.rel).is_some(),
+ }
+}
+
+/// Extension only. Sniff has not run; the census says so.
+fn classify(rel: &str) -> (Kind, Model) {
+ let ext = extension(rel);
+ if is_code_ext(&ext) {
+ return (Kind::Code, Model::Code);
+ }
+ if ext == "pdf" {
+ return (Kind::Pdf, Model::Pdf);
+ }
+ if matches!(
+ ext.as_str(),
+ "csv" | "tsv" | "jsonl" | "ndjson" | "sqlite" | "db" | "xlsx" | "xls" | "sql"
+ ) {
+ return (Kind::Data, Model::Data);
+ }
+ if matches!(
+ ext.as_str(),
+ "hdf"
+ | "h5"
+ | "hdf5"
+ | "parquet"
+ | "npy"
+ | "npz"
+ | "nc"
+ | "cdf"
+ | "feather"
+ | "arrow"
+ | "mat"
+ | "fits"
+ | "bvh"
+ ) {
+ return (Kind::Data, Model::Unindexed);
+ }
+ if matches!(
+ ext.as_str(),
+ "md" | "markdown"
+ | "rst"
+ | "txt"
+ | "adoc"
+ | "asciidoc"
+ | "org"
+ | "html"
+ | "htm"
+ | "tex"
+ | "rtf"
+ | "docx"
+ | "doc"
+ | "pptx"
+ | "epub"
+ ) {
+ return (Kind::Prose, Model::Prose);
+ }
+ if matches!(
+ ext.as_str(),
+ "png"
+ | "jpg"
+ | "jpeg"
+ | "gif"
+ | "webp"
+ | "bmp"
+ | "ico"
+ | "zip"
+ | "gz"
+ | "tgz"
+ | "bz2"
+ | "xz"
+ | "7z"
+ | "rar"
+ | "tar"
+ | "exe"
+ | "dll"
+ | "so"
+ | "dylib"
+ | "bin"
+ | "wasm"
+ | "class"
+ | "jar"
+ | "mp3"
+ | "mp4"
+ | "mov"
+ | "avi"
+ | "pyc"
+ | "woff"
+ | "woff2"
+ | "ttf"
+ | "otf"
+ | "dmg"
+ | "iso"
+ | "img"
+ ) {
+ return (Kind::Binary, Model::Unindexed);
+ }
+ (Kind::Other, Model::Other)
+}
+
+fn extension(rel: &str) -> String {
+ let name = rel.rsplit('/').next().unwrap_or(rel);
+ match name.rsplit_once('.') {
+ Some((stem, ext)) if !stem.is_empty() && !ext.is_empty() => ext.to_ascii_lowercase(),
+ _ => String::new(),
+ }
+}
+
+/// Test seam: project `files`/`source` through two measured points.
+#[cfg(test)]
+fn project_for_test(a: Point, b: Point, files: u64, source: u64) -> u64 {
+ project_points(&a, &b, files, source)
+}
+
+#[cfg(test)]
+fn entry(rel: &str, size: u64) -> FileEntry {
+ FileEntry {
+ path: Path::new(rel).to_path_buf(),
+ rel: rel.to_string(),
+ rel_id: rel.to_string(),
+ is_symlink: false,
+ size,
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn point(files: u64, source: u64, index: u64) -> Point {
+ Point {
+ files,
+ source_bytes: source,
+ index_bytes: index,
+ wall_ms: 0,
+ }
+ }
+
+ #[test]
+ fn two_points_reproduce_within_one_byte() {
+ let a = point(1, 100, 150);
+ let b = point(2, 400, 500);
+ assert_eq!(project_for_test(a, b, 1, 100), 150);
+ assert_eq!(project_for_test(a, b, 2, 400), 500);
+ // Same slope, a third size: 50 per file + 1 per source byte.
+ assert_eq!(project_for_test(a, b, 4, 1000), 4 * 50 + 1000);
+ }
+
+ #[test]
+ fn a_negative_per_file_intercept_uses_the_steeper_measured_ratio() {
+ // Unconstrained per-file term is negative (see fit_points).
+ let a = point(5, 100, 10);
+ let b = point(1, 1000, 900);
+ let fit = fit_points(&a, &b);
+ assert_eq!(fit.per_file, 0);
+ // 900/1000 is steeper than 10/100.
+ assert_eq!(fit.num as u128 * 1000, fit.den as u128 * 900);
+ let projected = project_for_test(a, b, 1, 1000);
+ assert!(projected >= 900, "{projected}");
+ assert!(project_for_test(a, b, 5, 100) >= 10);
+ }
+
+ #[test]
+ fn disk_budget_uses_the_reported_flood_reserve_and_does_not_guess() {
+ let volume = Volume {
+ total: Some(10_000),
+ avail: Some(4_000),
+ used_percent: Some(60),
+ flood_stage: Some(95),
+ threshold_enabled: true,
+ flood_blocked: false,
+ };
+ // Reserve is 5% of 10000 = 500. Budget is 4000 - 500.
+ assert_eq!(
+ disk_budget(&volume),
+ Some(DiskBudget::UntilFlood {
+ bytes: 3_500,
+ flood_percent: 95,
+ avail: 4_000,
+ total: 10_000,
+ })
+ );
+ let mut off = volume;
+ off.flood_stage = Some(0);
+ assert_eq!(
+ disk_budget(&off),
+ Some(DiskBudget::WatermarkOff {
+ bytes: 4_000,
+ avail: 4_000,
+ })
+ );
+ let mut missing = volume;
+ missing.flood_stage = None;
+ assert!(disk_budget(&missing).is_none());
+ let mut no_total = volume;
+ no_total.total = None;
+ assert!(disk_budget(&no_total).is_none());
+ }
+
+ fn sample_files() -> Vec<FileEntry> {
+ vec![
+ entry("src/main.rs", 100),
+ entry("docs/readme.md", 50),
+ entry("data/big.csv", 10_000),
+ entry("vendor/lib/x.rs", 80),
+ entry("pic.png", 500),
+ ]
+ }
+
+ #[test]
+ fn the_measured_points_are_not_projected_below_themselves() {
+ for cal in MEASURED {
+ for point in [&cal.small, &cal.large] {
+ let got = project_points(&cal.small, &cal.large, point.files, point.source_bytes);
+ assert!(
+ got + 8 >= point.index_bytes,
+ "{:?} semantic={} files={} source={} measured={} projected={got}",
+ cal.model,
+ cal.semantic,
+ point.files,
+ point.source_bytes,
+ point.index_bytes
+ );
+ }
+ }
+ }
+
+ #[test]
+ fn a_one_byte_cap_does_not_fit_and_does_not_start() {
+ let files = sample_files();
+ let assessment = assess(&Request {
+ root: "/data",
+ files: &files,
+ volume: None,
+ max_index_bytes: Some(1),
+ no_semantic: true,
+ plan: None,
+ approve: None,
+ duplicate_prefixes: &[],
+ });
+ assert!(assessment.needs_choice, "{}", assessment.body);
+ assert_eq!(
+ assessment.body["plans"][0]["fits"],
+ serde_json::Value::Bool(false)
+ );
+ assert!(matches!(
+ resolve(&assessment, Choice::Absent),
+ Outcome::Stop {
+ code: gate::EXIT_NEEDS_DECISION,
+ ..
+ }
+ ));
+ }
+
+ #[test]
+ fn a_named_plan_over_the_estimate_runs_with_a_warning() {
+ let files = sample_files();
+ let assessment = assess(&Request {
+ root: "/data",
+ files: &files,
+ volume: None,
+ max_index_bytes: Some(1),
+ no_semantic: true,
+ plan: Some("default"),
+ approve: None,
+ duplicate_prefixes: &[],
+ });
+ assert!(!assessment.needs_choice, "{}", assessment.body);
+ match resolve(&assessment, Choice::Absent) {
+ Outcome::Continue { keep, note, .. } => {
+ assert!(keep.is_none(), "default keeps every file");
+ assert!(note.contains("over the budget"), "{note}");
+ }
+ Outcome::Stop { code, body } => panic!("stopped with {code}: {body}"),
+ }
+ }
+
+ #[test]
+ fn kinds_follow_the_extension_before_sniff() {
+ assert_eq!(classify("a.rs").0, Kind::Code);
+ assert_eq!(classify("a.md").0, Kind::Prose);
+ assert_eq!(classify("a.pdf").0, Kind::Pdf);
+ assert_eq!(classify("a.csv").0, Kind::Data);
+ assert_eq!(classify("a.h5").0, Kind::Data);
+ assert_eq!(classify("a.h5").1, Model::Unindexed);
+ assert_eq!(classify("a.png").0, Kind::Binary);
+ assert_eq!(classify("Makefile").0, Kind::Other);
+ assert!(order::vendored_reason("vendor/lib.rs").is_some());
+ }
+
+ #[test]
+ fn byte_sizes_reject_zero_and_accept_a_suffix() {
+ assert_eq!(parse_byte_size("1024").unwrap(), 1024);
+ assert_eq!(parse_byte_size("2MB").unwrap(), 2 * 1024 * 1024);
+ assert_eq!(parse_byte_size("1GB").unwrap(), 1024 * 1024 * 1024);
+ assert!(parse_byte_size("0").is_err());
+ assert!(parse_byte_size("0MB").is_err());
+ assert!(parse_byte_size("soon").is_err());
+ }
+
+ #[test]
+ fn a_choice_of_cancel_or_a_plan_name_is_read_once() {
+ let names = vec!["code-and-docs".to_string()];
+ let mut input = std::io::Cursor::new("code-and-docs\n");
+ assert_eq!(
+ read_choice(&mut input, &names, |_| {}),
+ Choice::Plan("code-and-docs".into())
+ );
+ let mut input = std::io::Cursor::new("cancel\n");
+ assert_eq!(read_choice(&mut input, &names, |_| {}), Choice::Cancel);
+ }
+}
diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rs
index 119a859b7..e748a331a 100644
--- a/engine/crates/xerj-autoindex/src/catalog.rs
+++ b/engine/crates/xerj-autoindex/src/catalog.rs
@@ -2,6 +2,8 @@
//! wildcard so data-wide searches never hit metadata. Its dataset docs ARE
//! the agent-facing data map; `xerj autoindex map` renders them.
+use std::collections::BTreeMap;
+
use crate::correlate::KeyCorr;
use crate::state::PlanDataset;
use serde_json::{json, Value};
@@ -597,6 +599,54 @@ pub fn foreign_prefix_roots(runs: &[Value], prefix: &str, root: &str) -> Vec<Str
foreign.into_iter().collect()
}
+/// Prefixes other than `prefix` whose latest run recorded this same `root`.
+///
+/// #2078: indexing one folder twice, under `ax` and under a named prefix,
+/// does not collide on catalog ids, so it is a warning rather than the
+/// refusal [`foreign_prefix_roots`] produces. `root` is not a mapped catalog
+/// field; callers fetch `doc_kind: run` and filter here.
+pub fn other_prefixes_for_root(runs: &[Value], root: &str, prefix: &str) -> Vec<String> {
+ let mut latest: BTreeMap<&str, &str> = BTreeMap::new();
+ for run in runs {
+ if run.get("doc_kind").and_then(Value::as_str) != Some("run") {
+ continue;
+ }
+ let Some(recorded) = run.get("prefix").and_then(Value::as_str) else {
+ continue;
+ };
+ if recorded == prefix || recorded.is_empty() {
+ continue;
+ }
+ if run.get("root").and_then(Value::as_str) != Some(root) {
+ continue;
+ }
+ let started = run.get("started").and_then(Value::as_str).unwrap_or("");
+ match latest.get(recorded) {
+ Some(prev) if *prev >= started => {}
+ _ => {
+ latest.insert(recorded, started);
+ }
+ }
+ }
+ latest.into_keys().map(str::to_string).collect()
+}
+
+/// One line. `warn` strips newlines, so this stays a single sentence.
+pub fn same_root_other_prefix_line(prefix: &str, others: &[String]) -> String {
+ let listed = others
+ .iter()
+ .map(|other| format!("{other} (indices {other}-*)"))
+ .collect::<Vec<_>>()
+ .join(", ");
+ let first = others.first().map(String::as_str).unwrap_or(prefix);
+ format!(
+ "autoindex: this folder is already indexed under {listed}. This run uses {prefix} and \
+ will index it again. Re-run with --prefix {first} to update that copy instead. Both \
+ copies are kept: a {first}-* index can also hold other folders, so nothing here \
+ suggests deleting it."
+ )
+}
+
/// What a reused prefix will do, for both the refusal and the
/// `--replace-prefix` warning. `replacing` is true only when the operator
/// passed the flag and the run is going to continue.
@@ -1483,6 +1533,29 @@ mod prefix_reuse_tests {
);
}
+ #[test]
+ fn the_same_folder_under_another_prefix_is_named() {
+ let runs = vec![
+ json!({"doc_kind": "run", "prefix": "ax", "root": "/data/proj", "started": "2026-01-01T00:00:00Z"}),
+ json!({"doc_kind": "run", "prefix": "ax", "root": "/data/old", "started": "2026-02-01T00:00:00Z"}),
+ json!({"doc_kind": "run", "prefix": "proj", "root": "/data/proj", "started": "2026-03-01T00:00:00Z"}),
+ ];
+ assert_eq!(
+ other_prefixes_for_root(&runs, "/data/proj", "proj"),
+ vec!["ax".to_string()]
+ );
+ let line = same_root_other_prefix_line("proj", &["ax".into()]);
+ assert!(line.contains("under ax"), "{line}");
+ // `ax` also holds /data/old here, so a delete hint would destroy a
+ // folder this run never touched (#2114).
+ assert!(!line.contains("DELETE"), "{line}");
+ assert!(line.contains("--prefix ax"), "{line}");
+ assert_eq!(
+ other_prefixes_for_root(&runs, "/data/proj", "ax"),
+ vec!["proj".to_string()]
+ );
+ }
+
#[test]
fn replace_prefix_says_the_run_continues() {
let message = prefix_reuse_message("ref", "/data/new", &["/data/old".into()], true);
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index d177e5739..e95c4934e 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -111,6 +111,13 @@ pub struct IndexCfg {
/// An answer to a previous decision request. `None` means "nobody has
/// answered yet", which is what arms the gate.
pub approve: Option<Approval>,
+ /// `--max-index-size`, in bytes. `None` means the only budget is the free
+ /// space under the node's flood-stage watermark, when the node reports one.
+ pub max_index_bytes: Option<u64>,
+ /// `--plan`. `None` means the default plan, which is indexed when it fits
+ /// and refused when it does not. A named plan runs even when the estimate
+ /// says it does not fit (with a warning): that is the override.
+ pub budget_plan: Option<String>,
pub json: bool,
pub quiet: bool,
/// Progress surface. Orthogonal to `json`: `--json` shapes *stdout* (the
@@ -380,7 +387,19 @@ pub fn help_text_with(feedback: bool) -> String {
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20`fast` also applies --no-semantic --no-graph. `narrower`\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20is NOT accepted here — it means re-running against a\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20subdirectory, which this flag cannot do.\n\
- \x20\x20\x20\x20--yes, -y alias for --approve proceed\n\
+ \x20\x20\x20\x20--yes, -y alias for --approve proceed. When the default index\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20plan does not fit the disk budget, --yes takes the\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20recommended plan that does fit. It does not index a\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20plan that is over budget.\n\
+ \x20\x20\x20\x20--max-index-size <N> cap the estimated index (bytes, or a KB/MB/GB suffix).\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20The disk budget is the free space under the node's\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20flood-stage watermark; this flag lowers that cap.\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x200 is rejected\n\
+ \x20\x20\x20\x20--plan <NAME> index with one plan: default, skip-vendored, skip-data,\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20code-and-docs, no-vectors, cap-1mb,\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20code-and-docs-no-vectors. A named plan runs even\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20when the estimate says it does not fit (warned).\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20See DISK BUDGET.\n\
\x20\x20\x20\x20--dry-run walk+sniff+infer, print the plan, index nothing\n\
\x20\x20\x20\x20--label <FILE> label each document at ingest through POST /_decide\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20(#1062): the question-set file's noul/choice questions\n\
@@ -659,6 +678,23 @@ pub fn help_text_with(feedback: bool) -> String {
\x20\x20\x20\x20Credentials come from the environment only: AWS_ACCESS_KEY_ID,\n\
\x20\x20\x20\x20AWS_SECRET_ACCESS_KEY, optionally AWS_SESSION_TOKEN. Never from a flag.\n\
\n\
+ DISK BUDGET:\n\
+ \x20\x20\x20\x20Before hashing, autoindex counts files by extension (code, prose, pdf,\n\
+ \x20\x20\x20\x20data, binary, other) and by top-level folder, and estimates the index\n\
+ \x20\x20\x20\x20from measured per-kind ratios. The budget is the free space on the\n\
+ \x20\x20\x20\x20node's data volume under the flood-stage watermark (the same\n\
+ \x20\x20\x20\x20GET /_cat/allocation read that refuses a volume already at the\n\
+ \x20\x20\x20\x20watermark), optionally lowered with --max-index-size. A node that does\n\
+ \x20\x20\x20\x20not report disk.flood_stage is not guessed at. When the default plan\n\
+ \x20\x20\x20\x20fits, the run continues. When it does not, stdout carries a JSON plan\n\
+ \x20\x20\x20\x20list and the run exits 4 unless --yes takes the recommended plan that\n\
+ \x20\x20\x20\x20fits, or --plan names one (a named plan is the override: it runs\n\
+ \x20\x20\x20\x20over the estimate with a warning; the estimate errs high).\n\
+ \x20\x20\x20\x20A terminal with the question visible is asked. Kinds without a\n\
+ \x20\x20\x20\x20calibration are reported as unmeasured and do not refuse the run.\n\
+ \x20\x20\x20\x20Skipped files are listed after the ok: line (up to 20, then a count).\n\
+ \x20\x20\x20\x20A folder already indexed under another prefix is named before the walk.\n\
+ \n\
EXIT CODES: 0 complete (also: gate answered with --approve cancel);\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x203 completed-with-junk (junk recorded, never fatal) — SUCCESS:\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20the index is searchable, and the summary on stdout opens\n\
@@ -678,9 +714,11 @@ pub fn help_text_with(feedback: bool) -> String {
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20`skipped_appeared` names them. Re-run with --fresh to\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20rebuild the plan in place and include them. Read `reason`\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20to tell this apart from completed-with-junk;\n\
- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x204 NEEDS A DECISION — the estimate exceeded --max-minutes, or a\n\
- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20--watch poll cost more than --max-monthly-ops; nothing was\n\
- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20indexed and a JSON decision request is on stdout;\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x204 NEEDS A DECISION — the estimate exceeded --max-minutes, the\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20estimated index does not fit the disk budget, or a --watch\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20poll cost more than --max-monthly-ops; nothing was indexed\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20and a JSON decision request is on stdout. A plan that does\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20not fit is not started, including with --yes;\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x202 usage; 1 endpoint/journal failure, a refused corpus removal, or a\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20refused unsafe state transition\n",
feedback_block = xerj_common::feedback::block(feedback),
@@ -825,6 +863,9 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
let mut max_minutes_explicit = false;
let mut approve: Option<Approval> = None;
let mut approve_explicit = false;
+ let mut max_index_bytes: Option<u64> = None;
+ let mut max_index_explicit = false;
+ let mut budget_plan: Option<String> = None;
let mut json = false;
let mut quiet = false;
let mut progress: Option<ProgressMode> = None;
@@ -1043,6 +1084,31 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
approve = Some(parsed);
approve_explicit = true;
}
+ "--max-index-size" => {
+ max_index_explicit = true;
+ let raw = it
+ .next()
+ .ok_or("--max-index-size needs a size in bytes or with a KB/MB/GB suffix")?;
+ max_index_bytes = Some(crate::budget::parse_byte_size(&raw)?);
+ }
+ "--plan" => {
+ let raw = it.next().ok_or_else(|| {
+ format!(
+ "--plan needs one of: {}",
+ crate::budget::PLAN_NAMES.join(", ")
+ )
+ })?;
+ if !crate::budget::PLAN_NAMES.contains(&raw.as_str()) {
+ return Err(format!(
+ "--plan {raw} is not a plan. choices: {}",
+ crate::budget::PLAN_NAMES.join(", ")
+ ));
+ }
+ if budget_plan.is_some() {
+ return Err("--plan was given twice".into());
+ }
+ budget_plan = Some(raw);
+ }
"--yes" | "-y" => {
if let Some(previous) = approve {
if previous != Approval::Proceed {
@@ -1228,6 +1294,10 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
if prefix.is_empty() {
return Err("--prefix must contain at least one [a-z0-9] character".into());
}
+ // #2057: reject a prefix that cannot name an index before the walk.
+ if let Some(err) = crate::dataset::index_prefix_error(&prefix) {
+ return Err(err);
+ }
// Contradictory progress requests are refused, never silently resolved in
// one side's favour: accepting a flag we will not honour is the defect
@@ -1524,10 +1594,10 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
.into(),
);
}
- if max_minutes_explicit || approve_explicit {
+ if max_minutes_explicit || approve_explicit || max_index_explicit || budget_plan.is_some() {
return Err(
- "--max-minutes/--approve/--yes are the folder-indexing gate; a watch has its own \
- cost gate (--max-monthly-ops, --allow-cost)"
+ "--max-minutes/--approve/--yes/--max-index-size/--plan are the folder-indexing \
+ gate; a watch has its own cost gate (--max-monthly-ops, --allow-cost)"
.into(),
);
}
@@ -1600,9 +1670,15 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
}
match (sub.as_deref(), folder) {
- (Some("map"), _) | (Some("status"), _) if max_minutes_explicit || approve_explicit => {
+ (Some("map"), _) | (Some("status"), _)
+ if max_minutes_explicit
+ || approve_explicit
+ || max_index_explicit
+ || budget_plan.is_some() =>
+ {
Err(format!(
- "--max-minutes/--approve/--yes apply only to indexing, not `autoindex {}`",
+ "--max-minutes/--approve/--yes/--max-index-size/--plan apply only to indexing, \
+ not `autoindex {}`",
sub.as_deref().unwrap_or_default()
))
}
@@ -1738,6 +1814,8 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
dry_run,
max_minutes,
approve,
+ max_index_bytes,
+ budget_plan,
json,
quiet,
progress,
@@ -2646,6 +2724,9 @@ mod tests {
"--max-minutes",
"--approve <ID>",
"--yes, -y",
+ "--max-index-size",
+ "--plan <NAME>",
+ "DISK BUDGET:",
"0 disables the gate",
"ESTIMATE + DECISION GATE:",
"exits 4",
@@ -2663,6 +2744,20 @@ mod tests {
}
}
+ /// #2057: a digit-leading prefix is a usage error, before any walk. The
+ /// budget flags parse here too, and an unknown plan is refused.
+ #[test]
+ fn a_digit_leading_prefix_and_an_unknown_plan_are_usage_errors() {
+ let prefix_err = err(&["data", "--prefix", "2042"]);
+ assert!(prefix_err.contains("p2042"), "{prefix_err}");
+ assert!(prefix_err.contains("lowercase letter"), "{prefix_err}");
+ let cfg = index(&["data", "--plan", "code-and-docs", "--max-index-size", "2MB"]);
+ assert_eq!(cfg.budget_plan.as_deref(), Some("code-and-docs"));
+ assert_eq!(cfg.max_index_bytes, Some(2 * 1024 * 1024));
+ let plan_err = err(&["data", "--plan", "everything"]);
+ assert!(plan_err.contains("not a plan"), "{plan_err}");
+ }
+
/// A `\` at the end of a help line used to eat the next line's indent, so
/// `--limit`'s wrap started at column 0. The wrap stays under the
/// description.
diff --git a/engine/crates/xerj-autoindex/src/dataset.rs b/engine/crates/xerj-autoindex/src/dataset.rs
index 61303d74a..427eee7e1 100644
--- a/engine/crates/xerj-autoindex/src/dataset.rs
+++ b/engine/crates/xerj-autoindex/src/dataset.rs
@@ -330,6 +330,30 @@ pub fn sanitize_prefix(s: &str) -> String {
out
}
+/// Why `{prefix}-docs` cannot be an index name, if it cannot.
+///
+/// #2057: the server used to be the first thing that noticed a digit-leading
+/// prefix, and it noticed after the hash. `None` means the prefix can name
+/// an index.
+pub fn index_prefix_error(prefix: &str) -> Option<String> {
+ let index = format!("{prefix}-docs");
+ match xerj_common::IndexName::validate(&index) {
+ Ok(()) => None,
+ Err(err) => {
+ let hint = if prefix.starts_with(|c: char| c.is_ascii_digit()) {
+ format!(" A prefix that starts with a letter works, for example p{prefix}.")
+ } else {
+ String::new()
+ };
+ Some(format!(
+ "--prefix {prefix} is not a valid index prefix: an index is named \
+ {prefix}-<dataset>, and `{index}` is rejected ({err}). An index name must \
+ start with a lowercase letter or '.'.{hint} Nothing was walked."
+ ))
+ }
+ }
+}
+
fn numericish_segment(s: &str) -> bool {
let t = s.trim();
!t.is_empty()
@@ -497,6 +521,15 @@ pub fn drop_repeated_prefix(prefix: &str, clusters: &mut [Cluster]) {
mod tests {
use super::*;
+ #[test]
+ fn a_digit_leading_prefix_is_rejected_before_any_walk() {
+ let err = index_prefix_error("2042").expect("2042-docs is not an index name");
+ assert!(err.contains("lowercase letter"), "{err}");
+ assert!(err.contains("p2042"), "{err}");
+ assert!(err.contains("Nothing was walked"), "{err}");
+ assert!(index_prefix_error("proj").is_none());
+ }
+
#[test]
fn sanitize_prefix_keeps_a_literal_double_hyphen() {
// issue #1889: `owner--repo` corpora must index under the name they verify under
diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rs
index 337acff22..529b8d48c 100644
--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs
+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs
@@ -376,6 +376,8 @@ pub(super) fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
// behaviour is covered in `gate_tests` and `cli::tests`.
max_minutes: 0,
approve: None,
+ max_index_bytes: None,
+ budget_plan: None,
dry_run: false,
json: false,
quiet: true,
diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
index ac61fdf98..cf9cf7431 100644
--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
@@ -814,6 +814,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
// behaviour is covered in `gate_tests` and `cli::tests`.
max_minutes: 0,
approve: None,
+ max_index_bytes: None,
+ budget_plan: None,
dry_run: false,
json: false,
quiet: true,
diff --git a/engine/crates/xerj-autoindex/src/flood.rs b/engine/crates/xerj-autoindex/src/flood.rs
index 8d84c708b..3ae8ececa 100644
--- a/engine/crates/xerj-autoindex/src/flood.rs
+++ b/engine/crates/xerj-autoindex/src/flood.rs
@@ -20,15 +20,73 @@ use crate::esclient::Es;
const ALLOCATION_PATH: &str = "/_cat/allocation?format=json";
-/// `None` when the node cannot be asked or does not report a flood-stage
-/// threshold. A stats blip must not abort a run whose disk is fine, and the
-/// write path still refuses if the block is actually engaged.
-pub(crate) fn refusal(es: &Es) -> Option<String> {
- let (status, body) = es.request_json("GET", ALLOCATION_PATH, None).ok()?;
+/// One row of `GET /_cat/allocation?format=json`.
+///
+/// The budget gate reads free space and the flood threshold from this same
+/// response. It does not issue a second probe, and it does not fill in a
+/// threshold the node left out.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) struct Volume {
+ pub total: Option<u64>,
+ pub avail: Option<u64>,
+ pub used_percent: Option<u64>,
+ pub flood_stage: Option<u8>,
+ pub threshold_enabled: bool,
+ pub flood_blocked: bool,
+}
+
+/// The one allocation read: the parsed volume, and the refusal when the
+/// node says writes will fail.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Probe {
+ pub volume: Option<Volume>,
+ pub refusal: Option<String>,
+}
+
+/// `None` fields when the node cannot be asked. A stats blip must not abort
+/// a run whose disk is fine, and the write path still refuses if the block
+/// is actually engaged.
+pub(crate) fn probe(es: &Es) -> Probe {
+ let Ok((status, body)) = es.request_json("GET", ALLOCATION_PATH, None) else {
+ return Probe {
+ volume: None,
+ refusal: None,
+ };
+ };
if !(200..300).contains(&status) {
- return None;
+ return Probe {
+ volume: None,
+ refusal: None,
+ };
}
- refusal_from_allocation(&body)
+ Probe {
+ volume: volume_from_allocation(&body),
+ refusal: refusal_from_allocation(&body),
+ }
+}
+
+/// Parsed disk columns from one allocation body. `None` when the body is
+/// not an allocation row (the HTTP doubles answer `{"acknowledged": true}`).
+pub(crate) fn volume_from_allocation(body: &Value) -> Option<Volume> {
+ let row = body.as_array()?.first()?;
+ let flood_stage = row
+ .get("disk.flood_stage")
+ .and_then(parse_u64)
+ .and_then(|n| u8::try_from(n.min(100)).ok());
+ Some(Volume {
+ total: row.get("disk.total").and_then(parse_bytes),
+ avail: row.get("disk.avail").and_then(parse_bytes),
+ used_percent: row.get("disk.percent").and_then(parse_u64),
+ flood_stage,
+ threshold_enabled: row
+ .get("disk.threshold_enabled")
+ .and_then(parse_bool)
+ .unwrap_or(true),
+ flood_blocked: row
+ .get("disk.flood_blocked")
+ .and_then(parse_bool)
+ .unwrap_or(false),
+ })
}
/// Decide from one `/_cat/allocation?format=json` body.
@@ -214,6 +272,24 @@ mod tests {
assert!(refusal_from_allocation(&json!([])).is_none());
}
+ #[test]
+ fn the_same_row_carries_free_and_total_bytes_for_the_budget() {
+ let body = json!([{
+ "disk.percent": "40",
+ "disk.avail": "1000b",
+ "disk.total": "10000b",
+ "disk.flood_stage": "95",
+ "disk.threshold_enabled": "true",
+ "disk.flood_blocked": "false",
+ }]);
+ let volume = volume_from_allocation(&body).expect("an allocation row");
+ assert_eq!(volume.avail, Some(1000));
+ assert_eq!(volume.total, Some(10000));
+ assert_eq!(volume.used_percent, Some(40));
+ assert_eq!(volume.flood_stage, Some(95));
+ assert!(volume_from_allocation(&json!({"acknowledged": true})).is_none());
+ }
+
/// The probe runs before the walk: a flooded node refuses with the
/// override text and never opens the folder, and a healthy node is not
/// refused (the walk then reports that the root is a file).
diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
index 5dffa0c1a..3027e6e8e 100644
--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs
@@ -970,6 +970,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {
// timing-derived stop. See `gate_tests` and `cli::tests`.
max_minutes: 0,
approve: None,
+ max_index_bytes: None,
+ budget_plan: None,
dry_run: false,
json: false,
quiet: true,
@@ -6410,6 +6412,93 @@ mod delete_gate {
assert!(landed, "the replayed document landed");
}
+ /// A cap between the measured `code-and-docs` estimate and the default
+ /// estimate stops before any data bulk (exit 4). `--plan code-and-docs`
+ /// is the plan that fits, and the csv it drops is not indexed. The sizes
+ /// come from `budget::MEASURED`. An empty table fails here: skipping would
+ /// hide a gate that can no longer tell the plans apart.
+ #[test]
+ fn a_budget_too_small_for_the_default_plan_accepts_a_smaller_one() {
+ let _guard = HTTP_E2E_LOCK.lock().unwrap_or_else(|p| p.into_inner());
+ let corpus = tempfile::tempdir().unwrap();
+ fs::write(
+ corpus.path().join("main.rs"),
+ "fn main() { println!(\"budget\"); }\n",
+ )
+ .unwrap();
+ let mut csv = String::from("id,value\n");
+ for i in 0..2000 {
+ csv.push_str(&format!("{i},row-{i}-budget-padding\n"));
+ }
+ fs::write(corpus.path().join("big.csv"), csv).unwrap();
+
+ let (files, _) = walk::walk_reporting_opts(
+ corpus.path(),
+ false,
+ false,
+ crate::ignore_rules::IgnoreOptions::default(),
+ )
+ .unwrap();
+ let priced = budget::assess(&budget::Request {
+ root: corpus.path().to_str().unwrap_or("/corpus"),
+ files: &files,
+ volume: None,
+ max_index_bytes: Some(u64::MAX),
+ no_semantic: true,
+ plan: None,
+ approve: None,
+ duplicate_prefixes: &[],
+ });
+ let plan_bytes = |name: &str| -> u64 {
+ let plan = priced.body["plans"]
+ .as_array()
+ .and_then(|plans| plans.iter().find(|plan| plan["name"] == name))
+ .unwrap_or_else(|| panic!("missing plan {name} in {}", priced.body));
+ plan["index_bytes"].as_u64().unwrap_or_else(|| {
+ panic!(
+ "plan {name} has no measured index size. Fill budget::MEASURED from a real \
+ xerj run; an unmeasured kind must not be given a guessed ratio. {}",
+ priced.body
+ )
+ })
+ };
+ let default_bytes = plan_bytes("default");
+ let reduced_bytes = plan_bytes("code-and-docs");
+ assert!(
+ reduced_bytes < default_bytes,
+ "code-and-docs ({reduced_bytes}) has to be smaller than default ({default_bytes}) \
+ so the cap can sit between them"
+ );
+
+ let endpoint = HttpEndpoint::start();
+ let refused_state = tempfile::tempdir().unwrap();
+ let mut refused = cfg(corpus.path(), refused_state.path(), &endpoint.url, false);
+ refused.prefix = "budget-http".into();
+ refused.max_index_bytes = Some(reduced_bytes);
+ assert_eq!(run_index(refused).unwrap(), gate::EXIT_NEEDS_DECISION);
+ assert_eq!(
+ endpoint.data_bulk_requests(),
+ 0,
+ "a plan that does not fit must not bulk-index"
+ );
+
+ let accepted_state = tempfile::tempdir().unwrap();
+ let mut accepted = cfg(corpus.path(), accepted_state.path(), &endpoint.url, false);
+ accepted.prefix = "budget-http".into();
+ accepted.max_index_bytes = Some(reduced_bytes);
+ accepted.budget_plan = Some("code-and-docs".into());
+ assert_eq!(run_index(accepted).unwrap(), 0);
+ let indexed = paths(&endpoint.data_docs());
+ assert!(
+ indexed.iter().any(|path| path.ends_with("main.rs")),
+ "the code file is the part the plan keeps: {indexed:?}"
+ );
+ assert!(
+ indexed.iter().all(|path| !path.ends_with("big.csv")),
+ "the csv is the part the plan drops: {indexed:?}"
+ );
+ }
+
fn base() -> CommittedManifest {
CommittedManifest {
generation: 0,
diff --git a/engine/crates/xerj-autoindex/src/label_http_tests.rs b/engine/crates/xerj-autoindex/src/label_http_tests.rs
index 426ad13a9..941150495 100644
--- a/engine/crates/xerj-autoindex/src/label_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/label_http_tests.rs
@@ -384,6 +384,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, label: &Path) -> IndexCfg {
no_graph: true,
max_minutes: 0,
approve: None,
+ max_index_bytes: None,
+ budget_plan: None,
dry_run: false,
json: false,
quiet: true,
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index 98fbbf36d..92e96e5b0 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -3,6 +3,7 @@
//! it does NOT link xerj-engine, works against any endpoint, and cannot
//! destabilize the server.
+pub mod budget;
pub mod catalog;
pub mod cli;
pub mod coerce;
@@ -3520,6 +3521,8 @@ mod phase_a_grouping_tests {
// behaviour is covered in `gate_tests` and `cli::tests`.
max_minutes: 0,
approve: None,
+ max_index_bytes: None,
+ budget_plan: None,
dry_run: true,
json: false,
quiet: true,
@@ -3997,7 +4000,7 @@ fn report_pdf_spool_fallbacks(
}
for (path, fallback) in reasons.iter().take(3) {
pr.note(&format!(
- " PDF reuse fallback for {path}: {}",
+ " PDF reuse fallback for {path}: {}. The PDF is still indexed; phase B parses it again.",
fallback.message
));
}
@@ -5370,6 +5373,14 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -
let junk: Vec<&JunkFile> = committed.plan.junk_files.iter().collect();
unextracted_archive_lines(&junk)
};
+ // Taken while `committed` still borrows the journal. `journal.finish`
+ // needs that borrow back, and the printed summary (human and --json)
+ // lists the same skips the plan recorded (#2056).
+ let skip_lines = skipped_lines_from_plan(&committed.plan.junk_files);
+ let skip_doc = {
+ let refs: Vec<&JunkFile> = committed.plan.junk_files.iter().collect();
+ skipped_json(&refs)
+ };
let sync::SourceExecutionPolicy::DurableSnapshot { reference, .. } = &execution.source_policy
else {
anyhow::bail!("generated run does not reference a durable snapshot");
@@ -5419,6 +5430,9 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -
if es.bulk_requests_split() > 0 {
summary["bulk_splits"] = json!(es.bulk_requests_split());
}
+ if let Some(skipped) = skip_doc {
+ summary["skipped"] = skipped;
+ }
journal.finish(&summary)?;
if cfg.json {
println!("{summary}");
@@ -5434,6 +5448,9 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -
0,
)
);
+ for line in &skip_lines {
+ println!("{line}");
+ }
println!(
"generation {} committed — {}, {} live",
generation,
@@ -6041,6 +6058,65 @@ const UNEXTRACTED_PREFIX: &str = "unextracted ";
/// "successful" empty index with the one sentence that explains it filed where
/// they would never look (review finding on PR #949). These few lines are the
/// run saying it.
+const SKIP_LIST_CAP: usize = 20;
+
+/// #2056: names of skipped files, capped, with the reason. The headline stays
+/// one `ok:` line; this is the list under it.
+fn skipped_lines(junk: &[&JunkFile]) -> Vec<String> {
+ if junk.is_empty() {
+ return Vec::new();
+ }
+ let mut rows: Vec<&&JunkFile> = junk.iter().collect();
+ rows.sort_by(|a, b| a.rel.cmp(&b.rel).then(a.reason.cmp(&b.reason)));
+ let mut lines = vec![format!(
+ "skipped: {}",
+ estimate::count_phrase(rows.len() as u64, "file", "files")
+ )];
+ for jf in rows.iter().take(SKIP_LIST_CAP) {
+ lines.push(format!(
+ " skipped: {} ({}: {})",
+ jf.rel, jf.status, jf.reason
+ ));
+ }
+ if rows.len() > SKIP_LIST_CAP {
+ lines.push(format!(
+ " … and {} more; `xerj autoindex map` lists every one",
+ rows.len() - SKIP_LIST_CAP
+ ));
+ }
+ lines
+}
+
+fn skipped_lines_from_plan(junk: &[JunkFile]) -> Vec<String> {
+ let refs: Vec<&JunkFile> = junk.iter().collect();
+ skipped_lines(&refs)
+}
+
+fn skipped_json(junk: &[&JunkFile]) -> Option<Value> {
+ if junk.is_empty() {
+ return None;
+ }
+ let mut rows: Vec<&&JunkFile> = junk.iter().collect();
+ rows.sort_by(|a, b| a.rel.cmp(&b.rel).then(a.reason.cmp(&b.reason)));
+ let total = rows.len();
+ let files: Vec<Value> = rows
+ .iter()
+ .take(SKIP_LIST_CAP)
+ .map(|jf| {
+ json!({
+ "path": jf.rel,
+ "status": jf.status,
+ "reason": jf.reason,
+ })
+ })
+ .collect();
+ let mut doc = json!({ "count": total, "files": files });
+ if total > SKIP_LIST_CAP {
+ doc["not_listed"] = json!(total - SKIP_LIST_CAP);
+ }
+ Some(doc)
+}
+
fn unextracted_archive_lines(junk: &[&JunkFile]) -> Vec<String> {
const SHOWN: usize = 10;
let mut hits: Vec<&&JunkFile> = junk
@@ -6332,6 +6408,14 @@ fn legacy_rebuild_argv(cfg: &IndexCfg, state_dir: &Path, prefix: &str) -> Vec<St
if cfg.no_semantic {
argv.push("--no-semantic".to_owned());
}
+ if let Some(bytes) = cfg.max_index_bytes {
+ argv.push("--max-index-size".to_owned());
+ argv.push(bytes.to_string());
+ }
+ if let Some(plan) = &cfg.budget_plan {
+ argv.push("--plan".to_owned());
+ argv.push(plan.clone());
+ }
if cfg.follow_symlinks {
argv.push("--follow-symlinks".to_owned());
}
@@ -6365,12 +6449,11 @@ fn legacy_rebuild_argv(cfg: &IndexCfg, state_dir: &Path, prefix: &str) -> Vec<St
/// check on a 401 or a 500 would let the overwrite this guard exists to stop
/// proceed. The caller re-checks the prefix exactly; a legacy analyzed
/// `prefix` field can over-return sibling corpora.
-fn catalog_runs_for_prefix(es: &Es, prefix: &str) -> Result<Vec<Value>> {
+fn catalog_runs(es: &Es) -> Result<Vec<Value>> {
let body = json!({
"size": 500,
"query": {"bool": {"filter": [
- {"term": {"doc_kind": "run"}},
- {"term": {"prefix": prefix}}
+ {"term": {"doc_kind": "run"}}
]}}
});
let Some(page) = es.search_present(catalog::CATALOG_INDEX, &body)? else {
@@ -6465,6 +6548,12 @@ fn run_index_report_inner(
// throttle. Its shrink/recover announcements go to stderr, which the
// progress surface owns — so they are emitted exactly when that surface is
// enabled, not merely when `--quiet` is absent.
+ // #2057: a digit-leading prefix is rejected by the server only when the
+ // index is created, which used to be after the hash. The name is wrong
+ // before any walk, and it does not need the node.
+ if let Some(err) = dataset::index_prefix_error(&cfg.prefix) {
+ anyhow::bail!("{err}");
+ }
let es = Es::with_bulk_timeout(&cfg.url, cfg.api_key.clone(), cfg.bulk_timeout_secs)?
.with_bulk_concurrency(cfg.workers, pr.enabled());
// #1102: ping's raw context is honest but teaches a first-time user
@@ -6482,13 +6571,18 @@ fn run_index_report_inner(
// `limits.disk_flood_stage_percent = 99` workaround and a disabled
// watermark. Nothing has been walked or written yet, so bailing here
// leaves no journal and no remote mutation.
+ // One allocation read (#2077). The refusal bails before the walk. The
+ // parsed volume is what the budget gate, after the walk and still before
+ // the hash, compares the estimated index against. A second GET is not made.
// A dry run writes nothing, so a full volume is not a reason to refuse
// it: the plan (and its size) is exactly what a user short on disk needs.
+ let disk = flood::probe(&es);
if !cfg.dry_run {
- if let Some(message) = flood::refusal(&es) {
+ if let Some(message) = disk.refusal {
anyhow::bail!("{message}");
}
}
+ let disk_volume = disk.volume;
// `--label <question-set>` (#1062): every record's payload goes through
// the node's /_decide at ingest and the answers are stamped on as
@@ -6553,12 +6647,10 @@ fn run_index_report_inner(
// only remote calls on that path are the readiness GET and the read-only
// flood-stage probe (#2077), so this guard must not search the catalog
// there. A different folder has no such journal and still hits the check.
+ let mut duplicate_prefixes: Vec<String> = Vec::new();
if !state::journal_continues_folder(&state_dir, &root_str, &cfg.url, &cfg.prefix) {
- let foreign_roots = catalog::foreign_prefix_roots(
- &catalog_runs_for_prefix(&es, &cfg.prefix)?,
- &cfg.prefix,
- &root_str,
- );
+ let runs = catalog_runs(&es)?;
+ let foreign_roots = catalog::foreign_prefix_roots(&runs, &cfg.prefix, &root_str);
if !foreign_roots.is_empty() {
let message = catalog::prefix_reuse_message(
&cfg.prefix,
@@ -6577,6 +6669,16 @@ fn run_index_report_inner(
anyhow::bail!("{message}");
}
}
+ // #2078: the same folder under a second prefix writes a second copy
+ // and used to say nothing. This is a warning, not a refusal — the
+ // prefixes do not share catalog ids.
+ duplicate_prefixes = catalog::other_prefixes_for_root(&runs, &root_str, &cfg.prefix);
+ if !duplicate_prefixes.is_empty() {
+ pr.warn(&catalog::same_root_other_prefix_line(
+ &cfg.prefix,
+ &duplicate_prefixes,
+ ));
+ }
}
// Acquire state authority before discovery as well as hashing. A waiter
// must never classify a path snapshot taken while another owner was
@@ -6849,7 +6951,7 @@ fn run_index_report_inner(
cfg.exclude_rels.join(", ")
));
}
- let (discovered_files, data_files, data_bytes) = if cfg.skip_corpus_data {
+ let (mut discovered_files, data_files, data_bytes) = if cfg.skip_corpus_data {
without_corpus_data_files(discovered_files)
} else {
(discovered_files, 0, 0)
@@ -6865,7 +6967,7 @@ fn run_index_report_inner(
is data, not reference code — pass --include-data to index them"
));
}
- let discovered_bytes: u64 = discovered_files.iter().map(|f| f.size).sum();
+ let mut discovered_bytes: u64 = discovered_files.iter().map(|f| f.size).sum();
pr.note(&format!(
"autoindex: {} files ({} MB) under {}",
discovered_files.len(),
@@ -6885,6 +6987,82 @@ fn run_index_report_inner(
pr.finish(true, 0, "no-files", &[]);
return Ok((0, None));
}
+ // Count, estimate, and — when the default plan does not fit the disk
+ // budget — stop before hashing. The volume is the allocation row read
+ // above; this does not probe the disk again.
+ {
+ let assessment = budget::assess(&budget::Request {
+ root: &root_str,
+ files: &discovered_files,
+ volume: disk_volume,
+ max_index_bytes: cfg.max_index_bytes,
+ no_semantic: cfg.no_semantic,
+ plan: cfg.budget_plan.as_deref(),
+ approve: cfg.approve,
+ duplicate_prefixes: &duplicate_prefixes,
+ });
+ for line in &assessment.lines {
+ if assessment.needs_choice {
+ pr.note(line);
+ }
+ }
+ let choice = if assessment.needs_choice {
+ let blocked = gate::detect_prompt_block(pr.enabled());
+ if let Some(why) = blocked {
+ pr.note(&format!(
+ "budget: {}. Re-run with --plan <name> or --yes",
+ why.explain()
+ ));
+ budget::Choice::Absent
+ } else {
+ budget::read_choice(
+ &mut std::io::stdin().lock(),
+ &assessment.plan_names,
+ |line| {
+ pr.note(line);
+ },
+ )
+ }
+ } else {
+ budget::Choice::Absent
+ };
+ match budget::resolve(&assessment, choice) {
+ budget::Outcome::Stop { code, body } => {
+ if !body.is_null() {
+ println!("{}", serde_json::to_string(&body)?);
+ }
+ if code == 0 {
+ pr.note("gate: --approve cancel — nothing was indexed");
+ pr.finish(true, 0, "cancelled", &[]);
+ } else {
+ pr.finish(false, code, "needs-decision", &[]);
+ }
+ return Ok((code, None));
+ }
+ budget::Outcome::Continue {
+ keep,
+ no_semantic,
+ note,
+ } => {
+ if no_semantic {
+ cfg.no_semantic = true;
+ }
+ if let Some(keep) = keep {
+ let keep: std::collections::HashSet<String> = keep.into_iter().collect();
+ let before = discovered_files.len();
+ discovered_files.retain(|f| keep.contains(&f.rel));
+ discovered_bytes = discovered_files.iter().map(|f| f.size).sum();
+ pr.note(&format!(
+ "autoindex: budget plan kept {} of {before} walked files",
+ discovered_files.len()
+ ));
+ }
+ if !note.is_empty() {
+ pr.note(&format!("autoindex: {note}"));
+ }
+ }
+ }
+ }
// Full hashing on every run is deliberate: size/mtime/inode fingerprints
// cannot prove byte identity across all supported local and network
// filesystems. A metadata-only shortcut could leave stale live documents
@@ -9824,6 +10002,15 @@ fn run_index_report_inner(
if skipped_appeared > 0 {
run_doc["skipped_appeared"] = json!(skipped_appeared);
}
+ // #2056: the exit-3 headline stays one line. The paths themselves are a
+ // separate list, capped, with the reason. Present only when something
+ // was skipped, so a clean run document is unchanged.
+ if let Some(skipped) = skipped_json(&all_junk) {
+ run_doc["skipped"] = skipped;
+ }
+ // Owned lines: the resume-plan update below mutates `plan.junk_files`,
+ // which `all_junk` borrows. The summary prints these after that update.
+ let skip_lines = skipped_lines(&all_junk);
push_doc(&format!("run:{run_id}"), &run_doc, &mut cat_buf);
if !cat_buf.is_empty() {
@@ -9921,6 +10108,9 @@ fn run_index_report_inner(
skipped_appeared as u64,
)
);
+ for line in &skip_lines {
+ println!("{line}");
+ }
println!(
"{}",
estimate::done_summary_line(
diff --git a/engine/crates/xerj-console-api/src/knowledge.rs b/engine/crates/xerj-console-api/src/knowledge.rs
index 6133ec207..78e24004d 100644
--- a/engine/crates/xerj-console-api/src/knowledge.rs
+++ b/engine/crates/xerj-console-api/src/knowledge.rs
@@ -495,7 +495,7 @@ fn capabilities(
caps.push(json!({
"id": "map",
"title": "Print the data map",
- "blurb": "The same field/coverage table this page shows, as markdown — for you or for an agent (MCP: xerj_map; `xerj mcp` exposes 18 tools).",
+ "blurb": "The same field/coverage table this page shows, as markdown — for you or for an agent (MCP: xerj_map; `xerj mcp` exposes 19 tools).",
"command": "xerj autoindex map",
"kind": "cli",
}));
diff --git a/engine/crates/xerj-mcp/Cargo.toml b/engine/crates/xerj-mcp/Cargo.toml
index d6dbbf102..e32089d8a 100644
--- a/engine/crates/xerj-mcp/Cargo.toml
+++ b/engine/crates/xerj-mcp/Cargo.toml
@@ -49,6 +49,10 @@ unicode-segmentation = "1"
# itself (`xerj_hub::ask`), the one implementation the CLI and the gRPC
# service share — the tools are adapters, not a second copy of the flow.
xerj-hub = { path = "../xerj-hub" }
+# `xerj_index_budget` walks a folder and prices it before hashing. The walk,
+# the flood-stage allocation parse, and the plan list live in xerj-autoindex;
+# this crate only exposes them. No cycle: autoindex does not depend on mcp.
+xerj-autoindex = { path = "../xerj-autoindex" }
[dev-dependencies]
# The loopback admin-key fallback tests sandbox cwd + HOME so discovery sees
diff --git a/engine/crates/xerj-mcp/src/lib.rs b/engine/crates/xerj-mcp/src/lib.rs
index 988998d20..950894b96 100644
--- a/engine/crates/xerj-mcp/src/lib.rs
+++ b/engine/crates/xerj-mcp/src/lib.rs
@@ -9,11 +9,12 @@
//!
//! It speaks the MCP **stdio transport**: newline-delimited
//! JSON-RPC 2.0 messages on stdin/stdout. It exposes XERJ to any MCP-capable
-//! agent host (Claude Desktop, IDE agents, custom orchestrators) as eighteen
-//! tools: fifteen without a hub key, and three Corpus Hub tools
+//! agent host (Claude Desktop, IDE agents, custom orchestrators) as nineteen
+//! tools: sixteen without a hub key, and three Corpus Hub tools
//! (`xerj_hub_*`) that run the `xerj ask` flow itself. Thirteen of those
-//! fifteen construct exactly the request the running engine already accepts
-//! and forward it to a configurable base URL. `xerj_corpus_feedback` posts an
+//! sixteen construct exactly the request the running engine already accepts
+//! and forward it to a configurable base URL. `xerj_index_budget` counts a
+//! folder and returns the budget plans; it does not index. `xerj_corpus_feedback` posts an
//! agent's review of a hub corpus to hub.xerj.org. `xerj_field_report` does not:
//! it files a field report, not to the node: a GitHub PR first (through
//! `xerj feedback --submit --github-only`), and a POST to the web endpoint only
@@ -1127,6 +1128,47 @@ pub fn tool_specs_with(include_hub: bool) -> Value {
}
}
},
+ {
+ "name": "xerj_index_budget",
+ "description": concat!(
+ "Count a folder and estimate the index BEFORE xerj autoindex hashes or ",
+ "embeds anything. Returns file and byte counts by kind (code, prose, pdf, ",
+ "data, binary, other — by extension, before content sniff) and by top-level ",
+ "folder, the free-space budget from the node's flood-stage watermark ",
+ "(the same GET /_cat/allocation row autoindex uses; a node that omits ",
+ "disk.flood_stage is not guessed at), and one or more plans with what ",
+ "each drops. This tool does not index. When decision_required is true, ",
+ "show the plans to the user and re-run `xerj autoindex` with --plan <name> ",
+ "or --yes for the recommended plan that fits. Do not start a run that ",
+ "does not fit. Default embeddings in the estimate are the server's ",
+ "lexical feature-hash vectors unless that node was started with ",
+ "--embed-mode neural. Kinds without a measured calibration are ",
+ "unmeasured and do not refuse the run."
+ ),
+ "inputSchema": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "type": "string",
+ "description": "Folder to count. Nothing inside it is hashed."
+ },
+ "prefix": {
+ "type": "string",
+ "description": "Index prefix to check for an existing copy of this folder (default ax)."
+ },
+ "max_index_bytes": {
+ "type": "integer",
+ "minimum": 1,
+ "description": "Optional cap on the estimated index, in bytes. The disk budget is the free space under the flood-stage watermark; this lowers it."
+ },
+ "no_semantic": {
+ "type": "boolean",
+ "description": "Price the plans without semantic_text vectors, as --no-semantic would."
+ }
+ },
+ "required": ["path"]
+ }
+ },
{
"name": "xerj_code_search",
"description": concat!(
@@ -1769,6 +1811,7 @@ async fn call_tool(ctx: &Ctx, msg: &Value) -> Value {
// code_search — it renders the per-index field map itself, so it
// returns a tool result directly instead of a BuiltRequest.
"xerj_map" => return map::run(ctx, &args).await,
+ "xerj_index_budget" => return run_index_budget(ctx, &args).await,
"xerj_code_search" => return run_code_search(ctx, &args).await,
// Corpus Hub v2 (plan §7): the `xerj ask` flow itself, not a proxy.
hub_tool if hub_tool.starts_with("xerj_hub_") => {
@@ -2507,6 +2550,68 @@ impl xerj_common::xccode::XcHttp for McpXc {
}
}
+/// `xerj_index_budget` — the autoindex preflight, without indexing.
+///
+/// The walk and the plan math are `xerj_autoindex::budget`. Disk numbers come
+/// from the same allocation probe autoindex uses. A node that is down still
+/// returns the file counts; the disk field is then null.
+async fn run_index_budget(ctx: &Ctx, args: &Value) -> Value {
+ let path = match args.get("path").and_then(Value::as_str) {
+ Some(p) if !p.is_empty() => p.to_string(),
+ _ => return tool_text("xerj_index_budget: `path` is required", true),
+ };
+ let prefix = args
+ .get("prefix")
+ .and_then(Value::as_str)
+ .filter(|s| !s.is_empty())
+ .unwrap_or("ax")
+ .to_string();
+ let no_semantic = match args.get("no_semantic") {
+ None | Some(Value::Null) => false,
+ Some(Value::Bool(v)) => *v,
+ Some(_) => return tool_text("xerj_index_budget: `no_semantic` must be a boolean", true),
+ };
+ let max_index_bytes = match args.get("max_index_bytes") {
+ None | Some(Value::Null) => None,
+ Some(Value::Number(n)) => match n.as_u64() {
+ Some(n) if n > 0 => Some(n),
+ _ => {
+ return tool_text(
+ "xerj_index_budget: `max_index_bytes` must be a positive integer",
+ true,
+ )
+ }
+ },
+ Some(_) => {
+ return tool_text(
+ "xerj_index_budget: `max_index_bytes` must be a positive integer",
+ true,
+ )
+ }
+ };
+ let url = ctx.base_url.clone();
+ let auth = ctx.auth.clone();
+ let joined = tokio::task::spawn_blocking(move || {
+ xerj_autoindex::budget::assess_path(
+ std::path::Path::new(&path),
+ &url,
+ auth.as_deref(),
+ &prefix,
+ max_index_bytes,
+ no_semantic,
+ )
+ })
+ .await;
+ match joined {
+ Ok(Ok(body)) => tool_text(
+ serde_json::to_string_pretty(&body).unwrap_or_else(|_| body.to_string()),
+ false,
+ ),
+ Ok(Err(err)) => tool_text(format!("xerj_index_budget: {err:#}"), true),
+ Err(err) => tool_text(format!("xerj_index_budget: {err}"), true),
+ }
+}
+
/// `xerj_code_search` — the SAME pipeline `xerj code` runs, from the SAME
/// renderer; only the surface differs. Warnings ride the TOP of the tool text
/// (MCP has no stderr channel, so warnings are content), and the exit-code
@@ -4280,7 +4385,7 @@ mod tests {
.iter()
.map(|t| t["name"].as_str().unwrap())
.collect();
- assert_eq!(names.len(), 18);
+ assert_eq!(names.len(), 19);
for n in [
"xerj_search",
"xerj_semantic_search",
@@ -4294,6 +4399,7 @@ mod tests {
"xerj_brain_overview",
// #1055: the per-index field map from the autoindex catalog.
"xerj_map",
+ "xerj_index_budget",
// #977: reference-coding retrieval, served by the same pipeline
// as `xerj code`.
"xerj_code_search",
@@ -4370,23 +4476,23 @@ mod tests {
.collect()
};
let without = names(false);
- assert_eq!(without.len(), 15, "{without:?}");
+ assert_eq!(without.len(), 16, "{without:?}");
assert!(
without.iter().all(|n| !n.starts_with("xerj_hub_")),
"{without:?}"
);
let with = names(true);
- assert_eq!(with.len(), 18, "{with:?}");
+ assert_eq!(with.len(), 19, "{with:?}");
for n in ["xerj_hub_find", "xerj_hub_def", "xerj_hub_ask"] {
assert!(with.iter().any(|name| name == n), "missing {n}");
}
let help = tools_paragraph(false);
- assert!(help.contains("proxies 15 tools"), "{help}");
+ assert!(help.contains("proxies 16 tools"), "{help}");
assert!(!help.contains("xerj_hub_"), "{help}");
assert!(help.contains("~/xerj-data"), "{help}");
let help = tools_paragraph(true);
- assert!(help.contains("proxies 18 tools"), "{help}");
+ assert!(help.contains("proxies 19 tools"), "{help}");
assert!(help.contains("xerj_hub_ask"), "{help}");
assert!(help.contains("serves 3 Corpus Hub tools"), "{help}");
}
diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rs
index d13d51bf6..27fea7b1f 100644
--- a/engine/crates/xerj-server/src/brain.rs
+++ b/engine/crates/xerj-server/src/brain.rs
@@ -642,6 +642,8 @@ fn index_cfg(cfg: &BrainCfg, brain: &str, api_key: Option<String>) -> IndexCfg {
// grows its own answer flag. The estimate itself is still printed.
max_minutes: 0,
approve: None,
+ max_index_bytes: None,
+ budget_plan: None,
json: false,
quiet: false,
// `xerj brain` is a foreground command a human watches; it gets the
diff --git a/landing/docs/agents/schemas/mcp-tools.json b/landing/docs/agents/schemas/mcp-tools.json
index 30fc8e3c8..7dc3a0398 100644
--- a/landing/docs/agents/schemas/mcp-tools.json
+++ b/landing/docs/agents/schemas/mcp-tools.json
@@ -537,6 +537,35 @@
}
}
},
+ {
+ "name": "xerj_index_budget",
+ "description": "Count a folder and estimate the index BEFORE xerj autoindex hashes or embeds anything. Returns file and byte counts by kind (code, prose, pdf, data, binary, other — by extension, before content sniff) and by top-level folder, the free-space budget from the node's flood-stage watermark (the same GET /_cat/allocation row autoindex uses; a node that omits disk.flood_stage is not guessed at), and one or more plans with what each drops. This tool does not index. When decision_required is true, show the plans to the user and re-run `xerj autoindex` with --plan <name> or --yes for the recommended plan that fits. Do not start a run that does not fit. Default embeddings in the estimate are the server's lexical feature-hash vectors unless that node was started with --embed-mode neural. Kinds without a measured calibration are unmeasured and do not refuse the run.",
+ "inputSchema": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "type": "string",
+ "description": "Folder to count. Nothing inside it is hashed."
+ },
+ "prefix": {
+ "type": "string",
+ "description": "Index prefix to check for an existing copy of this folder (default ax)."
+ },
+ "max_index_bytes": {
+ "type": "integer",
+ "minimum": 1,
+ "description": "Optional cap on the estimated index, in bytes. The disk budget is the free space under the flood-stage watermark; this lowers it."
+ },
+ "no_semantic": {
+ "type": "boolean",
+ "description": "Price the plans without semantic_text vectors, as --no-semantic would."
+ }
+ },
+ "required": [
+ "path"
+ ]
+ }
+ },
{
"name": "xerj_code_search",
"description": "Reference-coding retrieval over a corpus of peer projects: how did real codebases solve this problem? Returns the matching definition (not a byte window), the file:line to cite, and per-hit licence warnings — AGPL/SSPL/Elastic/BUSL/GPL/LGPL/MPL sources are approach-only. Refuses indices older than 30 days (override with stale_ok). A no-match is NOT an error: it says the corpus is wrong for the task. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule). Corpora: `xerj corpus list`.",