AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(autoindex): budget the index before hashing (#2102)
xerj-orgc0091f310502
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9a3d0cdb6..0078c9c56 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -63,6 +63,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0disk. A node that does not report the threshold is unchanged, and`--dry-run` (which writes nothing) is never refused for disk.+- `xerj autoindex` counts the walked folder and estimates the index before+ hashing. The free-space budget reuses that same allocation row (free bytes+ under the reported flood-stage watermark; the threshold is not guessed),+ and `--max-index-size` can lower it. When the default plan does not fit,+ stdout is a JSON plan list and the run exits 4 until `--yes` takes the+ recommended plan that fits or `--plan` names one; a named plan runs even over+ the estimate, with a warning, since the estimate errs high. A kind with no measured ratio stays unmeasured and+ does not refuse the run. A digit-leading `--prefix` is rejected before the+ walk ([#2057](https://github.com/xerj-org/xerj/issues/2057)). Skipped files+ are named after the `ok:` line, and a PDF reuse-fallback line says the PDF+ is still indexed ([#2056](https://github.com/xerj-org/xerj/issues/2056)).+ A folder already indexed under another prefix is named before the walk+ ([#2078](https://github.com/xerj-org/xerj/issues/2078)). MCP:+ `xerj_index_budget` returns the same estimate and does not index.+- Playground user dashboards resolve their inherited template for chart dataand panel editing. Cloning a user dashboard retains the underlying template;missing or cyclic template references keep the blank fallback. Customdiff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 2b7e228c2..86acce186 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -7694,6 +7694,7 @@ dependencies = ["tokio","unicode-segmentation","xerj-api",+ "xerj-autoindex","xerj-common","xerj-engine","xerj-hub",diff --git a/engine/crates/xerj-autoindex/src/budget.rs b/engine/crates/xerj-autoindex/src/budget.rsnew file mode 100644index 000000000..2425673a5--- /dev/null+++ b/engine/crates/xerj-autoindex/src/budget.rs@@ -0,0 +1,1553 @@+//! Disk budget for `xerj autoindex`, decided before hashing.+//!+//! The walk has already counted files. Hashing and embedding have not+//! started. This module estimates how large the index will be, compares that+//! with the room left under the node's flood-stage watermark, and — when the+//! default plan does not fit — names a smaller plan and waits for an explicit+//! yes. It never invents a ratio: a kind with no measured calibration is+//! reported as unmeasured, and an unmeasured estimate does not refuse a run.+//!+//! Free space and the watermark come from [`crate::flood`], the one+//! `GET /_cat/allocation?format=json` already made before the walk. A second+//! probe is not issued. A node that omits `disk.flood_stage` does not get a+//! guessed 95% margin.++use std::collections::BTreeMap;+use std::io::BufRead;+use std::path::Path;++use anyhow::Result;+use serde_json::{json, Value};++use crate::estimate::{count_phrase, human_bytes};+use crate::extract::code::is_code_ext;+use crate::flood::{self, Volume};+use crate::gate::{self, Approval};+use crate::order;+use crate::walk::{self, FileEntry};++/// Plans a caller can name with `--plan`. The order is the tie-break: an+/// earlier plan wins when two that fit keep the same number of source bytes,+/// so `default` wins whenever it fits.+pub const PLAN_NAMES: &[&str] = &[+ "default",+ "skip-vendored",+ "skip-data",+ "code-and-docs",+ "no-vectors",+ "cap-1mb",+ "code-and-docs-no-vectors",+];++const CAP_1MB: u64 = 1 << 20;++/// What the calibration measured, and where. An empty table is not a zero+/// estimate. The numbers in [`MEASURED`] are dataset-index store sizes from+/// one release binary on one host; they are not a promise about this machine.+const CALIBRATION_NOTE: &str = "per-kind ratios are dataset-index store sizes measured 2026-10-21 with xerj v1.0.0-rc.97 on one 4-core Linux x86_64 host. Code and prose were measured on real repository sources and docs; data, pdf and unindexed kinds on synthetic files. Embeddings were the default lexical feature-hash (the neural embedder was not enabled). Catalog store size is not included. time_s is a rough projection from that host, not this machine. A kind with no row is unmeasured and does not refuse the run";++/// One measured index of one kind.+#[derive(Debug, Clone, Copy)]+struct Point {+ files: u64,+ source_bytes: u64,+ index_bytes: u64,+ wall_ms: u64,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]+enum Model {+ Code,+ Prose,+ Pdf,+ Data,+ /// Binary and scientific containers the extractor does not read. Their+ /// index cost is a catalog row, not a copy of the file.+ Unindexed,+ Other,+}++#[derive(Debug, Clone, Copy)]+struct Calib {+ model: Model,+ semantic: bool,+ small: Point,+ large: Point,+}++/// Release-binary calibration, 2026-10-21.+///+/// Binary: `xerj v1.0.0-rc.97`, started `--insecure` (lexical feature-hash+/// embeddings; `--embed-mode neural` was not passed). Host: Linux 6.12.94++/// x86_64, 4 cores, 16387280 KiB RAM.+///+/// Code and prose points are real repository content, not synthetic files:+/// copies of this repository's `.rs` sources (5 files / 291,007 B and+/// 97 files / 5,096,601 B) and `.md` documents (24 files / 520,410 B and+/// 169 files / 2,969,566 B). Synthetic distinct-token files under-predicted+/// those by about 3.5x, so they were replaced. Data, PDF and unindexed points+/// are still the synthetic runs described below.+///+/// `index_bytes` is the sum of `store.size` for that prefix's indices from+/// `GET /_cat/indices?bytes=b&h=index,store.size` right after the run, before+/// segment merges settle: that peak is what the disk has to hold. A force-merge+/// later shrank the real-content stores by roughly 20-40%, so this errs high.+/// `autoindex-catalog` is omitted: its store size moved by tens of KB in both+/// directions between runs. For code and prose the per-file intercept comes+/// out negative, so the fitter uses the steeper of the two measured ratios.+/// `wall_ms` for code and prose is the wall time of the whole `xerj autoindex`+/// command, so it still contains the fixed per-run cost.+///+/// Synthetic points (data, PDF, unindexed): each is one `xerj autoindex` of+/// files with distinct tokens. `.npy` files were recorded as junk+/// (`files_junk` equal to the file count) and their dataset-index store delta+/// was 0. The two PDF points are about 1 KB and 10 KB; their per-byte slope is+/// negative, so the fitter uses the steeper of the two measured ratios.+static MEASURED: &[Calib] = &[+ Calib {+ model: Model::Code,+ semantic: true,+ small: Point {+ files: 5,+ source_bytes: 291_007,+ index_bytes: 1_337_452,+ wall_ms: 2_279,+ },+ large: Point {+ files: 97,+ source_bytes: 5_096_601,+ index_bytes: 15_252_875,+ wall_ms: 5_886,+ },+ },+ Calib {+ model: Model::Code,+ semantic: false,+ small: Point {+ files: 5,+ source_bytes: 291_007,+ index_bytes: 574_967,+ wall_ms: 1_740,+ },+ large: Point {+ files: 97,+ source_bytes: 5_096_601,+ index_bytes: 7_367_909,+ wall_ms: 5_139,+ },+ },+ Calib {+ model: Model::Prose,+ semantic: true,+ small: Point {+ files: 24,+ source_bytes: 520_410,+ index_bytes: 4_656_359,+ wall_ms: 1_749,+ },+ large: Point {+ files: 169,+ source_bytes: 2_969_566,+ index_bytes: 15_859_387,+ wall_ms: 5_308,+ },+ },+ Calib {+ model: Model::Prose,+ semantic: false,+ small: Point {+ files: 24,+ source_bytes: 520_410,+ index_bytes: 748_083,+ wall_ms: 3_796,+ },+ large: Point {+ files: 169,+ source_bytes: 2_969_566,+ index_bytes: 2_805_069,+ wall_ms: 3_646,+ },+ },+ Calib {+ model: Model::Data,+ semantic: true,+ small: Point {+ files: 1,+ source_bytes: 20_039,+ index_bytes: 63_682,+ wall_ms: 600,+ },+ large: Point {+ files: 4,+ source_bytes: 320_156,+ index_bytes: 681_267,+ wall_ms: 900,+ },+ },+ Calib {+ model: Model::Data,+ semantic: false,+ small: Point {+ files: 1,+ source_bytes: 20_039,+ index_bytes: 63_682,+ wall_ms: 600,+ },+ large: Point {+ files: 4,+ source_bytes: 320_156,+ index_bytes: 669_741,+ wall_ms: 900,+ },+ },+ Calib {+ model: Model::Other,+ semantic: true,+ small: Point {+ files: 1,+ source_bytes: 4_000,+ index_bytes: 19_611,+ wall_ms: 600,+ },+ large: Point {+ files: 8,+ source_bytes: 160_000,+ index_bytes: 169_502,+ wall_ms: 600,+ },+ },+ Calib {+ model: Model::Other,+ semantic: false,+ small: Point {+ files: 1,+ source_bytes: 4_000,+ index_bytes: 19_611,+ wall_ms: 600,+ },+ large: Point {+ files: 8,+ source_bytes: 160_000,+ index_bytes: 169_134,+ wall_ms: 600,+ },+ },+ Calib {+ model: Model::Pdf,+ semantic: true,+ small: Point {+ files: 1,+ source_bytes: 1_082,+ index_bytes: 14_542,+ wall_ms: 0,+ },+ large: Point {+ files: 4,+ source_bytes: 10_336,+ index_bytes: 29_218,+ wall_ms: 100,+ },+ },+ Calib {+ model: Model::Pdf,+ semantic: false,+ small: Point {+ files: 1,+ source_bytes: 1_082,+ index_bytes: 14_542,+ wall_ms: 0,+ },+ large: Point {+ files: 4,+ source_bytes: 10_336,+ index_bytes: 29_180,+ wall_ms: 100,+ },+ },+ Calib {+ model: Model::Unindexed,+ semantic: true,+ small: Point {+ files: 1,+ source_bytes: 4_000,+ index_bytes: 0,+ wall_ms: 0,+ },+ large: Point {+ files: 8,+ source_bytes: 320_000,+ index_bytes: 0,+ wall_ms: 0,+ },+ },+ Calib {+ model: Model::Unindexed,+ semantic: false,+ small: Point {+ files: 1,+ source_bytes: 4_000,+ index_bytes: 0,+ wall_ms: 0,+ },+ large: Point {+ files: 8,+ source_bytes: 320_000,+ index_bytes: 0,+ wall_ms: 0,+ },+ },+];++#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]+enum Kind {+ Code,+ Prose,+ Pdf,+ Data,+ Binary,+ Other,+}++impl Kind {+ fn name(self) -> &'static str {+ match self {+ Kind::Code => "code",+ Kind::Prose => "prose",+ Kind::Pdf => "pdf",+ Kind::Data => "data",+ Kind::Binary => "binary",+ Kind::Other => "other",+ }+ }+}++struct Fact {+ rel: String,+ size: u64,+ kind: Kind,+ model: Model,+ vendored: bool,+}++/// What the caller already decided, plus the walked files and the volume from+/// the flood-stage probe.+pub(crate) struct Request<'a> {+ pub root: &'a str,+ pub files: &'a [FileEntry],+ pub volume: Option<Volume>,+ pub max_index_bytes: Option<u64>,+ pub no_semantic: bool,+ pub plan: Option<&'a str>,+ pub approve: Option<Approval>,+ pub duplicate_prefixes: &'a [String],+}++/// A line the operator typed at the budget prompt.+#[derive(Debug, PartialEq, Eq)]+pub enum Choice {+ /// Nobody was asked, or the reply was empty.+ Absent,+ Plan(String),+ Cancel,+}++enum Preselect {+ None,+ Plan(String),+ Cancel,+ Refuse(&'static str),+}++struct PlanEst {+ name: &'static str,+ files: u64,+ source_bytes: u64,+ index_bytes: Option<u64>,+ time_s: Option<f64>,+ drops: Vec<String>,+ no_semantic: bool,+ keep: Option<Vec<String>>,+ fits: Option<bool>,+}++pub struct Assessment {+ pub body: Value,+ /// Lines to show before a prompt, or with a refusal.+ pub lines: Vec<String>,+ /// One line when the run continues. Empty when there is nothing new to say.+ pub note: String,+ pub needs_choice: bool,+ pub plan_names: Vec<String>,+ preselect: Preselect,+ plans: Vec<PlanEst>,+}++pub enum Outcome {+ Continue {+ /// `None` keeps every walked file.+ keep: Option<Vec<String>>,+ no_semantic: bool,+ note: String,+ },+ /// `body` is `Null` for a cancel, which prints no JSON.+ Stop { code: i32, body: Value },+}++/// Room the index may occupy.+#[derive(Debug, Clone, PartialEq, Eq)]+pub(crate) enum DiskBudget {+ /// Bytes until `disk.avail` would cross the flood-stage reserve.+ UntilFlood {+ bytes: u64,+ flood_percent: u8,+ avail: u64,+ total: u64,+ },+ /// Watermark off (`disk.flood_stage` 0, or the threshold disabled).+ WatermarkOff { bytes: u64, avail: u64 },+}++/// Index budget from the allocation row `#2077` already parsed.+///+/// `None` when the node did not report a flood threshold, or reported one+/// but not the total the reserve is a fraction of. The missing number is+/// not filled in from the default 95.+pub(crate) fn disk_budget(volume: &Volume) -> Option<DiskBudget> {+ let avail = volume.avail?;+ let flood = volume.flood_stage?;+ if !volume.threshold_enabled || flood == 0 {+ return Some(DiskBudget::WatermarkOff {+ bytes: avail,+ avail,+ });+ }+ let total = volume.total?;+ let headroom = u64::from(100u8.saturating_sub(flood));+ let reserve = ((total as u128) * u128::from(headroom) / 100) as u64;+ Some(DiskBudget::UntilFlood {+ bytes: avail.saturating_sub(reserve),+ flood_percent: flood,+ avail,+ total,+ })+}++/// Parse `--max-index-size`. A bare integer is bytes. `KB`/`MB`/`GB` (and the+/// one-letter forms) are powers of 1024. Zero is rejected.+pub fn parse_byte_size(raw: &str) -> Result<u64, String> {+ let raw = raw.trim();+ if raw.is_empty() {+ return Err("--max-index-size needs a size, for example 2GB".into());+ }+ let split = raw.find(|c: char| !c.is_ascii_digit());+ let (number, suffix) = match split {+ Some(i) => (&raw[..i], raw[i..].trim()),+ None => (raw, ""),+ };+ if number.is_empty() {+ return Err(format!(+ "--max-index-size needs a size in bytes or with a KB/MB/GB suffix, got {raw}"+ ));+ }+ let n: u64 = number.parse().map_err(|_| {+ format!("--max-index-size needs a size in bytes or with a KB/MB/GB suffix, got {raw}")+ })?;+ let mult: u64 = match suffix.to_ascii_lowercase().as_str() {+ "" | "b" => 1,+ "k" | "kb" | "kib" => 1024,+ "m" | "mb" | "mib" => 1024 * 1024,+ "g" | "gb" | "gib" => 1024 * 1024 * 1024,+ _ => {+ return Err(format!(+ "--max-index-size suffix must be B, KB, MB, or GB, got {suffix}"+ ))+ }+ };+ let bytes = n+ .checked_mul(mult)+ .ok_or_else(|| format!("--max-index-size {raw} overflows"))?;+ if bytes == 0 {+ return Err("--max-index-size must be greater than 0".into());+ }+ Ok(bytes)+}++pub(crate) fn assess(req: &Request<'_>) -> Assessment {+ let facts: Vec<Fact> = req.files.iter().map(fact_of).collect();+ let semantic_default = !req.no_semantic;+ let (budget_bytes, budget_basis, disk_json) = budget_view(req.volume, req.max_index_bytes);+ let plans = PLAN_NAMES+ .iter()+ .filter_map(|name| estimate_plan(name, &facts, semantic_default, budget_bytes))+ .collect::<Vec<_>>();+ let plan_names: Vec<String> = plans.iter().map(|p| p.name.to_string()).collect();+ let recommended = recommended_plan(&plans).map(|p| p.name);+ let default_fits = plans+ .iter()+ .find(|p| p.name == "default")+ .and_then(|p| p.fits);++ let mut preselect = Preselect::None;+ let mut needs_choice = false;+ if req.approve == Some(Approval::Cancel) {+ preselect = Preselect::Cancel;+ } else if let Some(name) = req.plan {+ match plans.iter().find(|p| p.name == name) {+ // A plan named on the command line is the override when the+ // estimate is wrong: it runs even if the estimate says it does+ // not fit, with a warning. The flood-stage guard still applies.+ Some(_) => preselect = Preselect::Plan(name.to_string()),+ // `no-vectors` is omitted when the run is already `--no-semantic`:+ // it is the same plan as `default`.+ None if PLAN_NAMES.contains(&name) => {+ preselect = Preselect::Plan("default".to_string())+ }+ None => preselect = Preselect::Refuse("unknown"),+ }+ } else if default_fits == Some(false) {+ let accepted = matches!(req.approve, Some(Approval::Proceed | Approval::Fast));+ if accepted {+ match recommended {+ Some(name) => preselect = Preselect::Plan(name.to_string()),+ None => preselect = Preselect::Refuse("none"),+ }+ } else {+ needs_choice = true;+ }+ }++ let note = if needs_choice || matches!(preselect, Preselect::Refuse(_) | Preselect::Cancel) {+ String::new()+ } else if let Preselect::Plan(name) = &preselect {+ plans+ .iter()+ .find(|p| p.name == name)+ .map(|p| continue_note(p, budget_bytes, &budget_basis))+ .unwrap_or_default()+ } else if default_fits == Some(true) {+ plans+ .iter()+ .find(|p| p.name == "default")+ .map(|p| {+ format!(+ "{} under the {} budget ({budget_basis})",+ plan_size_phrase(p),+ budget_bytes+ .map(human_bytes)+ .unwrap_or_else(|| "unset".into())+ )+ })+ .unwrap_or_default()+ } else {+ String::new()+ };++ let lines = summary_lines(&facts, &plans, budget_bytes, &budget_basis, recommended);+ let body = json!({+ "xerj": "autoindex-budget",+ "root": req.root,+ "classification": "by extension, before content sniff",+ "files": facts.len(),+ "bytes": facts.iter().map(|f| f.size).sum::<u64>(),+ "by_kind": counts_by(&facts, |f| f.kind.name()),+ "by_folder": counts_by(&facts, |f| top_folder(&f.rel)),+ "disk": disk_json,+ "budget_bytes": budget_bytes,+ "budget_basis": budget_basis,+ "max_index_bytes": req.max_index_bytes,+ "decision_required": needs_choice || matches!(preselect, Preselect::Refuse(_)),+ "plans": plans.iter().map(plan_json).collect::<Vec<_>>(),+ "recommended": recommended,+ "duplicate_prefixes": req.duplicate_prefixes,+ "accept": "re-run with --plan <name>, or --yes to take the recommended plan that fits. A named plan runs even when the estimate says it does not fit (the estimate errs high). --approve cancel indexes nothing",+ "calibration": CALIBRATION_NOTE,+ });++ Assessment {+ body,+ lines,+ note,+ needs_choice,+ plan_names,+ preselect,+ plans,+ }+}++pub fn resolve(assessment: &Assessment, choice: Choice) -> Outcome {+ if matches!(assessment.preselect, Preselect::Cancel) || matches!(choice, Choice::Cancel) {+ return Outcome::Stop {+ code: 0,+ body: Value::Null,+ };+ }+ if let Preselect::Refuse(why) = assessment.preselect {+ let mut body = assessment.body.clone();+ body["exit_code"] = json!(gate::EXIT_NEEDS_DECISION);+ body["refused"] = json!(match why {+ "none" => "no plan fits the budget",+ _ => "unknown plan",+ });+ return Outcome::Stop {+ code: gate::EXIT_NEEDS_DECISION,+ body,+ };+ }+ let name = match &assessment.preselect {+ Preselect::Plan(name) => Some(name.as_str()),+ Preselect::None if assessment.needs_choice => match choice {+ Choice::Plan(ref name) => Some(name.as_str()),+ Choice::Absent => {+ let mut body = assessment.body.clone();+ body["exit_code"] = json!(gate::EXIT_NEEDS_DECISION);+ return Outcome::Stop {+ code: gate::EXIT_NEEDS_DECISION,+ body,+ };+ }+ Choice::Cancel => {+ return Outcome::Stop {+ code: 0,+ body: Value::Null,+ };+ }+ },+ _ => None,+ };+ let Some(name) = name else {+ return Outcome::Continue {+ keep: None,+ no_semantic: false,+ note: assessment.note.clone(),+ };+ };+ match assessment.plans.iter().find(|p| p.name == name) {+ Some(plan) if plan.keep.as_ref().is_some_and(|kept| kept.is_empty()) => {+ let mut body = assessment.body.clone();+ body["exit_code"] = json!(gate::EXIT_NEEDS_DECISION);+ body["refused"] = json!(format!(+ "--plan {} keeps no files, so nothing was hashed. Pick another plan, or cancel",+ plan.name+ ));+ Outcome::Stop {+ code: gate::EXIT_NEEDS_DECISION,+ body,+ }+ }+ Some(plan) => Outcome::Continue {+ keep: plan.keep.clone(),+ no_semantic: plan.no_semantic,+ note: if assessment.note.is_empty() {+ continue_note(plan, budget_bytes_of(&assessment.body), "")+ } else {+ assessment.note.clone()+ },+ },+ None => {+ let mut body = assessment.body.clone();+ body["exit_code"] = json!(gate::EXIT_NEEDS_DECISION);+ body["refused"] = json!(format!(+ "unknown plan {name}; choices: {}",+ assessment.plan_names.join(", ")+ ));+ Outcome::Stop {+ code: gate::EXIT_NEEDS_DECISION,+ body,+ }+ }+ }+}++fn budget_bytes_of(body: &Value) -> Option<u64> {+ body.get("budget_bytes").and_then(Value::as_u64)+}++pub fn read_choice<R: BufRead>(+ input: &mut R,+ names: &[String],+ mut echo: impl FnMut(&str),+) -> Choice {+ for _ in 0..3 {+ echo(" plan name, or cancel: ");+ let mut line = String::new();+ match input.read_line(&mut line) {+ Ok(0) | Err(_) => return Choice::Absent,+ Ok(_) => {}+ }+ let typed = line.trim();+ if typed.eq_ignore_ascii_case("cancel") || typed.eq_ignore_ascii_case("c") {+ return Choice::Cancel;+ }+ if let Some(name) = names.iter().find(|n| n.eq_ignore_ascii_case(typed)) {+ return Choice::Plan(name.clone());+ }+ echo(&format!(+ " {typed:?} is not a plan. choices: {}",+ names.join(", ")+ ));+ }+ Choice::Absent+}++/// Walk `root`, read the node's allocation once, and return the budget JSON.+/// Does not index. A node that is down still returns the file counts; `disk`+/// is null and the basis says the probe did not answer.+pub fn assess_path(+ root: &Path,+ url: &str,+ auth: Option<&str>,+ prefix: &str,+ max_index_bytes: Option<u64>,+ no_semantic: bool,+) -> Result<Value> {+ let (files, _) = walk::walk_reporting_opts(+ root,+ false,+ false,+ crate::ignore_rules::IgnoreOptions::default(),+ )?;+ let root_str = root+ .canonicalize()+ .unwrap_or_else(|_| root.to_path_buf())+ .to_string_lossy()+ .to_string();+ let (volume, duplicates) =+ match crate::esclient::Es::with_bulk_timeout(url, auth.map(str::to_string), 30) {+ Ok(es) => {+ let volume = flood::probe(&es).volume;+ let duplicates = other_prefixes(&es, &root_str, prefix).unwrap_or_default();+ (volume, duplicates)+ }+ Err(_) => (None, Vec::new()),+ };+ let assessment = assess(&Request {+ root: &root_str,+ files: &files,+ volume,+ max_index_bytes,+ no_semantic,+ plan: None,+ approve: None,+ duplicate_prefixes: &duplicates,+ });+ Ok(assessment.body)+}++fn other_prefixes(es: &crate::esclient::Es, root: &str, prefix: &str) -> Result<Vec<String>> {+ let body = json!({+ "size": 500,+ "query": {"bool": {"filter": [{"term": {"doc_kind": "run"}}]}}+ });+ let Some(page) = es.search_present(crate::catalog::CATALOG_INDEX, &body)? else {+ return Ok(Vec::new());+ };+ let runs = page+ .pointer("/hits/hits")+ .and_then(Value::as_array)+ .map(|hits| {+ hits.iter()+ .filter_map(|hit| hit.get("_source").cloned())+ .collect::<Vec<_>>()+ })+ .unwrap_or_default();+ Ok(crate::catalog::other_prefixes_for_root(&runs, root, prefix))+}++fn budget_view(+ volume: Option<Volume>,+ max_index_bytes: Option<u64>,+) -> (Option<u64>, String, Value) {+ let disk = volume.as_ref().and_then(disk_budget);+ let (disk_bytes, basis, disk_json) = match (&volume, &disk) {+ (Some(vol), Some(DiskBudget::UntilFlood { bytes, flood_percent, avail, total })) => {+ let basis = format!(+ "free {} minus the {}% flood-stage reserve on {}",+ human_bytes(*avail),+ flood_percent,+ human_bytes(*total)+ );+ (+ Some(*bytes),+ basis,+ json!({+ "total": total,+ "avail": avail,+ "used_percent": vol.used_percent,+ "flood_stage": flood_percent,+ "threshold_enabled": vol.threshold_enabled,+ "reserve_bytes": avail.saturating_sub(*bytes),+ "budget_bytes": bytes,+ }),+ )+ }+ (Some(vol), Some(DiskBudget::WatermarkOff { bytes, avail })) => (+ Some(*bytes),+ "flood-stage watermark is off; budget is the free space".to_string(),+ json!({+ "total": vol.total,+ "avail": avail,+ "used_percent": vol.used_percent,+ "flood_stage": vol.flood_stage,+ "threshold_enabled": vol.threshold_enabled,+ "watermark": "off",+ "budget_bytes": bytes,+ }),+ ),+ (Some(_), None) => (+ None,+ "the node did not report a flood-stage threshold and a total, so no disk margin was applied".to_string(),+ json!(null_disk(volume.as_ref())),+ ),+ (None, _) => (+ None,+ "the node did not report disk allocation, so no disk margin was applied".to_string(),+ Value::Null,+ ),+ };+ let (bytes, basis) = match (disk_bytes, max_index_bytes) {+ (Some(disk), Some(cap)) => (+ Some(disk.min(cap)),+ format!(+ "{basis}; capped at {} by --max-index-size",+ human_bytes(cap)+ ),+ ),+ (None, Some(cap)) => (+ Some(cap),+ format!("--max-index-size {} ({basis})", human_bytes(cap)),+ ),+ (disk, None) => (disk, basis),+ };+ (bytes, basis, disk_json)+}++fn null_disk(volume: Option<&Volume>) -> Value {+ match volume {+ Some(vol) => json!({+ "total": vol.total,+ "avail": vol.avail,+ "used_percent": vol.used_percent,+ "flood_stage": vol.flood_stage,+ "threshold_enabled": vol.threshold_enabled,+ }),+ None => Value::Null,+ }+}++fn estimate_plan(+ name: &str,+ facts: &[Fact],+ semantic_default: bool,+ budget_bytes: Option<u64>,+) -> Option<PlanEst> {+ let force_no_semantic = matches!(name, "no-vectors" | "code-and-docs-no-vectors");+ let no_semantic = force_no_semantic || !semantic_default;+ if force_no_semantic && !semantic_default {+ // Already indexing without vectors; this plan is the same as another.+ return None;+ }+ let mut kept = Vec::new();+ let mut dropped_kinds: BTreeMap<&'static str, u64> = BTreeMap::new();+ let mut dropped_vendored = 0u64;+ let mut dropped_cap = 0u64;+ for fact in facts {+ if keep(name, fact) {+ kept.push(fact);+ } else if fact.vendored && name == "skip-vendored" {+ dropped_vendored += 1;+ } else if fact.size > CAP_1MB && name == "cap-1mb" {+ dropped_cap += 1;+ } else {+ *dropped_kinds.entry(fact.kind.name()).or_default() += 1;+ }+ }+ let mut drops = Vec::new();+ if dropped_vendored > 0 {+ drops.push(format!(+ "{} vendored",+ count_phrase(dropped_vendored, "path", "paths")+ ));+ }+ for (kind, n) in &dropped_kinds {+ drops.push(format!("{n} {kind}"));+ }+ if dropped_cap > 0 {+ drops.push(format!(+ "{} over 1 MB",+ count_phrase(dropped_cap, "file", "files")+ ));+ }+ if force_no_semantic {+ drops.push("vectors (semantic_text)".to_string());+ }+ let (index_bytes, time_s) = project_kept(&kept, !no_semantic);+ let fits = match (index_bytes, budget_bytes) {+ (Some(index), Some(budget)) => Some(index <= budget),+ _ => None,+ };+ let source_bytes = kept.iter().map(|f| f.size).sum();+ let keep_rels = if kept.len() == facts.len() && !force_no_semantic {+ None+ } else {+ Some(kept.iter().map(|f| f.rel.clone()).collect())+ };+ Some(PlanEst {+ name: PLAN_NAMES+ .iter()+ .copied()+ .find(|n| *n == name)+ .unwrap_or("default"),+ files: kept.len() as u64,+ source_bytes,+ index_bytes,+ time_s,+ drops,+ no_semantic,+ keep: keep_rels,+ fits,+ })+}++fn keep(name: &str, fact: &Fact) -> bool {+ match name {+ "skip-vendored" => !fact.vendored,+ "skip-data" => !matches!(fact.kind, Kind::Data | Kind::Binary),+ "code-and-docs" | "code-and-docs-no-vectors" => {+ matches!(fact.kind, Kind::Code | Kind::Prose | Kind::Pdf)+ }+ "cap-1mb" => fact.size <= CAP_1MB,+ _ => true,+ }+}++fn project_kept(kept: &[&Fact], semantic: bool) -> (Option<u64>, Option<f64>) {+ if MEASURED.is_empty() {+ return (None, None);+ }+ let mut index = 0u64;+ let mut wall_ms = 0u64;+ let mut timed = false;+ let mut measured_all = true;+ let mut groups: BTreeMap<Model, (u64, u64)> = BTreeMap::new();+ for fact in kept {+ let entry = groups.entry(fact.model).or_default();+ entry.0 += 1;+ entry.1 += fact.size;+ }+ for (model, (files, source)) in groups {+ let Some(cal) = MEASURED+ .iter()+ .find(|c| c.model == model && c.semantic == semantic)+ else {+ measured_all = false;+ continue;+ };+ index = index.saturating_add(project_points(&cal.small, &cal.large, files, source));+ let sample = if cal.large.source_bytes >= cal.small.source_bytes {+ &cal.large+ } else {+ &cal.small+ };+ if sample.source_bytes > 0 && sample.wall_ms > 0 {+ wall_ms = wall_ms.saturating_add(+ ((source as u128) * u128::from(sample.wall_ms) / u128::from(sample.source_bytes))+ as u64,+ );+ timed = true;+ }+ }+ let index = measured_all.then_some(index);+ let time_s = (timed && measured_all).then_some((wall_ms as f64) / 1000.0);+ (index, time_s)+}++/// `index_bytes ≈ per_file * files + source_bytes * num / den`, fitted to the+/// two measured points. A negative per-file intercept is clamped to 0 and the+/// slope becomes the steeper of the two measured index/source ratios, so the+/// projection stays at or above both measurements.+fn project_points(a: &Point, b: &Point, files: u64, source: u64) -> u64 {+ let fit = fit_points(a, b);+ fit.per_file+ .saturating_mul(files)+ .saturating_add(mul_div(source, fit.num, fit.den))+}++struct Fit {+ per_file: u64,+ num: u64,+ den: u64,+}++fn fit_points(a: &Point, b: &Point) -> Fit {+ let f1 = a.files as i128;+ let s1 = a.source_bytes as i128;+ let i1 = a.index_bytes as i128;+ let f2 = b.files as i128;+ let s2 = b.source_bytes as i128;+ let i2 = b.index_bytes as i128;+ let pden = f1 * s2 - f2 * s1;+ let pnum = i1 * s2 - i2 * s1;+ if pden == 0 {+ return steeper_ratio(a, b);+ }+ // p and pden have opposite signs when the intercept is negative.+ if pnum.signum() * pden.signum() < 0 {+ return steeper_ratio(a, b);+ }+ let per_file = (pnum / pden) as u64;+ let rnum = i1 * pden - pnum * f1;+ let rden = s1 * pden;+ if rden == 0 || rnum.signum() * rden.signum() < 0 {+ return steeper_ratio(a, b);+ }+ let (num, den) = reduce(rnum.unsigned_abs(), rden.unsigned_abs());+ Fit { per_file, num, den }+}++fn steeper_ratio(a: &Point, b: &Point) -> Fit {+ let pick = |p: &Point| -> Option<(u128, u128)> {+ if p.source_bytes == 0 {+ None+ } else {+ Some((u128::from(p.index_bytes), u128::from(p.source_bytes)))+ }+ };+ let best = match (pick(a), pick(b)) {+ (Some((i1, s1)), Some((i2, s2))) => {+ if i1 * s2 >= i2 * s1 {+ Some((i1, s1))+ } else {+ Some((i2, s2))+ }+ }+ (Some(r), None) | (None, Some(r)) => Some(r),+ (None, None) => None,+ };+ match best {+ Some((num, den)) => {+ let (num, den) = reduce(num, den);+ Fit {+ per_file: 0,+ num,+ den,+ }+ }+ None => {+ let files = a.files.max(b.files).max(1);+ let index = a.index_bytes.max(b.index_bytes);+ Fit {+ per_file: index / files,+ num: 0,+ den: 1,+ }+ }+ }+}++fn reduce(num: u128, den: u128) -> (u64, u64) {+ if den == 0 {+ return (0, 1);+ }+ let g = gcd(num, den);+ ((num / g) as u64, (den / g) as u64)+}++fn gcd(mut a: u128, mut b: u128) -> u128 {+ while b != 0 {+ let r = a % b;+ a = b;+ b = r;+ }+ a+}++fn mul_div(value: u64, num: u64, den: u64) -> u64 {+ if den == 0 {+ return 0;+ }+ ((u128::from(value) * u128::from(num)) / u128::from(den)) as u64+}++fn recommended_plan(plans: &[PlanEst]) -> Option<&PlanEst> {+ let mut best: Option<&PlanEst> = None;+ for plan in plans {+ if plan.fits != Some(true) {+ continue;+ }+ best = Some(match best {+ None => plan,+ Some(cur) if plan.source_bytes > cur.source_bytes => plan,+ Some(cur) => cur,+ });+ }+ best+}++fn plan_json(plan: &PlanEst) -> Value {+ json!({+ "name": plan.name,+ "fits": plan.fits,+ "index_bytes": plan.index_bytes,+ "source_bytes": plan.source_bytes,+ "files": plan.files,+ "drops": plan.drops,+ "no_semantic": plan.no_semantic,+ "time_s": plan.time_s,+ "time_basis": plan.time_s.map(|_| "rough projection from the calibration host, not this machine"),+ })+}++fn plan_size_phrase(plan: &PlanEst) -> String {+ match plan.index_bytes {+ Some(bytes) => format!(+ "estimated index {} for {} ({} source)",+ human_bytes(bytes),+ count_phrase(plan.files, "file", "files"),+ human_bytes(plan.source_bytes)+ ),+ None => format!(+ "index size unmeasured for {} ({} source)",+ count_phrase(plan.files, "file", "files"),+ human_bytes(plan.source_bytes)+ ),+ }+}++fn continue_note(plan: &PlanEst, budget: Option<u64>, basis: &str) -> String {+ let budget = budget+ .map(human_bytes)+ .unwrap_or_else(|| "an unknown".into());+ let drops = if plan.drops.is_empty() {+ String::new()+ } else {+ format!(". drops: {}", plan.drops.join(", "))+ };+ let basis = if basis.is_empty() {+ String::new()+ } else {+ format!(" ({basis})")+ };+ let over = if plan.fits == Some(false) {+ ". warning: the estimate is over the budget; running because the plan was named. \+ The node's flood-stage watermark still stops writes"+ } else {+ ""+ };+ format!(+ "--plan {} keeps {}{drops}. {} Budget {budget}{basis}{over}",+ plan.name,+ count_phrase(plan.files, "file", "files"),+ plan_size_phrase(plan)+ )+}++fn summary_lines(+ facts: &[Fact],+ plans: &[PlanEst],+ budget: Option<u64>,+ basis: &str,+ recommended: Option<&str>,+) -> Vec<String> {+ let bytes = facts.iter().map(|f| f.size).sum::<u64>();+ let mut lines = vec![format!(+ "budget: {} ({}), {}",+ count_phrase(facts.len() as u64, "file", "files"),+ human_bytes(bytes),+ kind_phrase(facts)+ )];+ if let Some(default) = plans.iter().find(|p| p.name == "default") {+ let fit = match default.fits {+ Some(true) => "fits",+ Some(false) => "does not fit",+ None => "is not compared (unmeasured, or no budget)",+ };+ lines.push(format!(+ "budget: default plan {fit}: {}. budget {} ({basis})",+ plan_size_phrase(default),+ budget.map(human_bytes).unwrap_or_else(|| "unset".into())+ ));+ }+ for plan in plans+ .iter()+ .filter(|p| p.name != "default" && p.fits != Some(false))+ {+ let mark = if Some(plan.name) == recommended {+ "recommended"+ } else {+ "option"+ };+ let drop = if plan.drops.is_empty() {+ String::new()+ } else {+ format!(" — drops {}", plan.drops.join(", "))+ };+ lines.push(format!(+ " {mark} {}: {}{drop}",+ plan.name,+ plan_size_phrase(plan)+ ));+ }+ lines+}++fn kind_phrase(facts: &[Fact]) -> String {+ let mut parts = Vec::new();+ for kind in [+ Kind::Code,+ Kind::Prose,+ Kind::Pdf,+ Kind::Data,+ Kind::Binary,+ Kind::Other,+ ] {+ let n = facts.iter().filter(|f| f.kind == kind).count();+ if n > 0 {+ parts.push(format!("{n} {}", kind.name()));+ }+ }+ if parts.is_empty() {+ "empty".to_string()+ } else {+ parts.join(", ")+ }+}++fn counts_by(facts: &[Fact], key: impl Fn(&Fact) -> &str) -> Value {+ let mut map: BTreeMap<&str, (u64, u64)> = BTreeMap::new();+ for fact in facts {+ let entry = map.entry(key(fact)).or_default();+ entry.0 += 1;+ entry.1 += fact.size;+ }+ let mut out = serde_json::Map::new();+ for (k, (files, bytes)) in map {+ out.insert(k.to_string(), json!({"files": files, "bytes": bytes}));+ }+ Value::Object(out)+}++fn top_folder(rel: &str) -> &str {+ match rel.split_once('/') {+ Some((dir, rest)) if !dir.is_empty() && !rest.is_empty() => dir,+ _ => "(root)",+ }+}++fn fact_of(file: &FileEntry) -> Fact {+ let (kind, model) = classify(&file.rel);+ Fact {+ rel: file.rel.clone(),+ size: file.size,+ kind,+ model,+ vendored: order::vendored_reason(&file.rel).is_some(),+ }+}++/// Extension only. Sniff has not run; the census says so.+fn classify(rel: &str) -> (Kind, Model) {+ let ext = extension(rel);+ if is_code_ext(&ext) {+ return (Kind::Code, Model::Code);+ }+ if ext == "pdf" {+ return (Kind::Pdf, Model::Pdf);+ }+ if matches!(+ ext.as_str(),+ "csv" | "tsv" | "jsonl" | "ndjson" | "sqlite" | "db" | "xlsx" | "xls" | "sql"+ ) {+ return (Kind::Data, Model::Data);+ }+ if matches!(+ ext.as_str(),+ "hdf"+ | "h5"+ | "hdf5"+ | "parquet"+ | "npy"+ | "npz"+ | "nc"+ | "cdf"+ | "feather"+ | "arrow"+ | "mat"+ | "fits"+ | "bvh"+ ) {+ return (Kind::Data, Model::Unindexed);+ }+ if matches!(+ ext.as_str(),+ "md" | "markdown"+ | "rst"+ | "txt"+ | "adoc"+ | "asciidoc"+ | "org"+ | "html"+ | "htm"+ | "tex"+ | "rtf"+ | "docx"+ | "doc"+ | "pptx"+ | "epub"+ ) {+ return (Kind::Prose, Model::Prose);+ }+ if matches!(+ ext.as_str(),+ "png"+ | "jpg"+ | "jpeg"+ | "gif"+ | "webp"+ | "bmp"+ | "ico"+ | "zip"+ | "gz"+ | "tgz"+ | "bz2"+ | "xz"+ | "7z"+ | "rar"+ | "tar"+ | "exe"+ | "dll"+ | "so"+ | "dylib"+ | "bin"+ | "wasm"+ | "class"+ | "jar"+ | "mp3"+ | "mp4"+ | "mov"+ | "avi"+ | "pyc"+ | "woff"+ | "woff2"+ | "ttf"+ | "otf"+ | "dmg"+ | "iso"+ | "img"+ ) {+ return (Kind::Binary, Model::Unindexed);+ }+ (Kind::Other, Model::Other)+}++fn extension(rel: &str) -> String {+ let name = rel.rsplit('/').next().unwrap_or(rel);+ match name.rsplit_once('.') {+ Some((stem, ext)) if !stem.is_empty() && !ext.is_empty() => ext.to_ascii_lowercase(),+ _ => String::new(),+ }+}++/// Test seam: project `files`/`source` through two measured points.+#[cfg(test)]+fn project_for_test(a: Point, b: Point, files: u64, source: u64) -> u64 {+ project_points(&a, &b, files, source)+}++#[cfg(test)]+fn entry(rel: &str, size: u64) -> FileEntry {+ FileEntry {+ path: Path::new(rel).to_path_buf(),+ rel: rel.to_string(),+ rel_id: rel.to_string(),+ is_symlink: false,+ size,+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn point(files: u64, source: u64, index: u64) -> Point {+ Point {+ files,+ source_bytes: source,+ index_bytes: index,+ wall_ms: 0,+ }+ }++ #[test]+ fn two_points_reproduce_within_one_byte() {+ let a = point(1, 100, 150);+ let b = point(2, 400, 500);+ assert_eq!(project_for_test(a, b, 1, 100), 150);+ assert_eq!(project_for_test(a, b, 2, 400), 500);+ // Same slope, a third size: 50 per file + 1 per source byte.+ assert_eq!(project_for_test(a, b, 4, 1000), 4 * 50 + 1000);+ }++ #[test]+ fn a_negative_per_file_intercept_uses_the_steeper_measured_ratio() {+ // Unconstrained per-file term is negative (see fit_points).+ let a = point(5, 100, 10);+ let b = point(1, 1000, 900);+ let fit = fit_points(&a, &b);+ assert_eq!(fit.per_file, 0);+ // 900/1000 is steeper than 10/100.+ assert_eq!(fit.num as u128 * 1000, fit.den as u128 * 900);+ let projected = project_for_test(a, b, 1, 1000);+ assert!(projected >= 900, "{projected}");+ assert!(project_for_test(a, b, 5, 100) >= 10);+ }++ #[test]+ fn disk_budget_uses_the_reported_flood_reserve_and_does_not_guess() {+ let volume = Volume {+ total: Some(10_000),+ avail: Some(4_000),+ used_percent: Some(60),+ flood_stage: Some(95),+ threshold_enabled: true,+ flood_blocked: false,+ };+ // Reserve is 5% of 10000 = 500. Budget is 4000 - 500.+ assert_eq!(+ disk_budget(&volume),+ Some(DiskBudget::UntilFlood {+ bytes: 3_500,+ flood_percent: 95,+ avail: 4_000,+ total: 10_000,+ })+ );+ let mut off = volume;+ off.flood_stage = Some(0);+ assert_eq!(+ disk_budget(&off),+ Some(DiskBudget::WatermarkOff {+ bytes: 4_000,+ avail: 4_000,+ })+ );+ let mut missing = volume;+ missing.flood_stage = None;+ assert!(disk_budget(&missing).is_none());+ let mut no_total = volume;+ no_total.total = None;+ assert!(disk_budget(&no_total).is_none());+ }++ fn sample_files() -> Vec<FileEntry> {+ vec![+ entry("src/main.rs", 100),+ entry("docs/readme.md", 50),+ entry("data/big.csv", 10_000),+ entry("vendor/lib/x.rs", 80),+ entry("pic.png", 500),+ ]+ }++ #[test]+ fn the_measured_points_are_not_projected_below_themselves() {+ for cal in MEASURED {+ for point in [&cal.small, &cal.large] {+ let got = project_points(&cal.small, &cal.large, point.files, point.source_bytes);+ assert!(+ got + 8 >= point.index_bytes,+ "{:?} semantic={} files={} source={} measured={} projected={got}",+ cal.model,+ cal.semantic,+ point.files,+ point.source_bytes,+ point.index_bytes+ );+ }+ }+ }++ #[test]+ fn a_one_byte_cap_does_not_fit_and_does_not_start() {+ let files = sample_files();+ let assessment = assess(&Request {+ root: "/data",+ files: &files,+ volume: None,+ max_index_bytes: Some(1),+ no_semantic: true,+ plan: None,+ approve: None,+ duplicate_prefixes: &[],+ });+ assert!(assessment.needs_choice, "{}", assessment.body);+ assert_eq!(+ assessment.body["plans"][0]["fits"],+ serde_json::Value::Bool(false)+ );+ assert!(matches!(+ resolve(&assessment, Choice::Absent),+ Outcome::Stop {+ code: gate::EXIT_NEEDS_DECISION,+ ..+ }+ ));+ }++ #[test]+ fn a_named_plan_over_the_estimate_runs_with_a_warning() {+ let files = sample_files();+ let assessment = assess(&Request {+ root: "/data",+ files: &files,+ volume: None,+ max_index_bytes: Some(1),+ no_semantic: true,+ plan: Some("default"),+ approve: None,+ duplicate_prefixes: &[],+ });+ assert!(!assessment.needs_choice, "{}", assessment.body);+ match resolve(&assessment, Choice::Absent) {+ Outcome::Continue { keep, note, .. } => {+ assert!(keep.is_none(), "default keeps every file");+ assert!(note.contains("over the budget"), "{note}");+ }+ Outcome::Stop { code, body } => panic!("stopped with {code}: {body}"),+ }+ }++ #[test]+ fn kinds_follow_the_extension_before_sniff() {+ assert_eq!(classify("a.rs").0, Kind::Code);+ assert_eq!(classify("a.md").0, Kind::Prose);+ assert_eq!(classify("a.pdf").0, Kind::Pdf);+ assert_eq!(classify("a.csv").0, Kind::Data);+ assert_eq!(classify("a.h5").0, Kind::Data);+ assert_eq!(classify("a.h5").1, Model::Unindexed);+ assert_eq!(classify("a.png").0, Kind::Binary);+ assert_eq!(classify("Makefile").0, Kind::Other);+ assert!(order::vendored_reason("vendor/lib.rs").is_some());+ }++ #[test]+ fn byte_sizes_reject_zero_and_accept_a_suffix() {+ assert_eq!(parse_byte_size("1024").unwrap(), 1024);+ assert_eq!(parse_byte_size("2MB").unwrap(), 2 * 1024 * 1024);+ assert_eq!(parse_byte_size("1GB").unwrap(), 1024 * 1024 * 1024);+ assert!(parse_byte_size("0").is_err());+ assert!(parse_byte_size("0MB").is_err());+ assert!(parse_byte_size("soon").is_err());+ }++ #[test]+ fn a_choice_of_cancel_or_a_plan_name_is_read_once() {+ let names = vec!["code-and-docs".to_string()];+ let mut input = std::io::Cursor::new("code-and-docs\n");+ assert_eq!(+ read_choice(&mut input, &names, |_| {}),+ Choice::Plan("code-and-docs".into())+ );+ let mut input = std::io::Cursor::new("cancel\n");+ assert_eq!(read_choice(&mut input, &names, |_| {}), Choice::Cancel);+ }+}diff --git a/engine/crates/xerj-autoindex/src/catalog.rs b/engine/crates/xerj-autoindex/src/catalog.rsindex 119a859b7..e748a331a 100644--- a/engine/crates/xerj-autoindex/src/catalog.rs+++ b/engine/crates/xerj-autoindex/src/catalog.rs@@ -2,6 +2,8 @@//! wildcard so data-wide searches never hit metadata. Its dataset docs ARE//! the agent-facing data map; `xerj autoindex map` renders them.+use std::collections::BTreeMap;+use crate::correlate::KeyCorr;use crate::state::PlanDataset;use serde_json::{json, Value};@@ -597,6 +599,54 @@ pub fn foreign_prefix_roots(runs: &[Value], prefix: &str, root: &str) -> Vec<Strforeign.into_iter().collect()}+/// Prefixes other than `prefix` whose latest run recorded this same `root`.+///+/// #2078: indexing one folder twice, under `ax` and under a named prefix,+/// does not collide on catalog ids, so it is a warning rather than the+/// refusal [`foreign_prefix_roots`] produces. `root` is not a mapped catalog+/// field; callers fetch `doc_kind: run` and filter here.+pub fn other_prefixes_for_root(runs: &[Value], root: &str, prefix: &str) -> Vec<String> {+ let mut latest: BTreeMap<&str, &str> = BTreeMap::new();+ for run in runs {+ if run.get("doc_kind").and_then(Value::as_str) != Some("run") {+ continue;+ }+ let Some(recorded) = run.get("prefix").and_then(Value::as_str) else {+ continue;+ };+ if recorded == prefix || recorded.is_empty() {+ continue;+ }+ if run.get("root").and_then(Value::as_str) != Some(root) {+ continue;+ }+ let started = run.get("started").and_then(Value::as_str).unwrap_or("");+ match latest.get(recorded) {+ Some(prev) if *prev >= started => {}+ _ => {+ latest.insert(recorded, started);+ }+ }+ }+ latest.into_keys().map(str::to_string).collect()+}++/// One line. `warn` strips newlines, so this stays a single sentence.+pub fn same_root_other_prefix_line(prefix: &str, others: &[String]) -> String {+ let listed = others+ .iter()+ .map(|other| format!("{other} (indices {other}-*)"))+ .collect::<Vec<_>>()+ .join(", ");+ let first = others.first().map(String::as_str).unwrap_or(prefix);+ format!(+ "autoindex: this folder is already indexed under {listed}. This run uses {prefix} and \+ will index it again. Re-run with --prefix {first} to update that copy instead. Both \+ copies are kept: a {first}-* index can also hold other folders, so nothing here \+ suggests deleting it."+ )+}+/// What a reused prefix will do, for both the refusal and the/// `--replace-prefix` warning. `replacing` is true only when the operator/// passed the flag and the run is going to continue.@@ -1483,6 +1533,29 @@ mod prefix_reuse_tests {);}+ #[test]+ fn the_same_folder_under_another_prefix_is_named() {+ let runs = vec![+ json!({"doc_kind": "run", "prefix": "ax", "root": "/data/proj", "started": "2026-01-01T00:00:00Z"}),+ json!({"doc_kind": "run", "prefix": "ax", "root": "/data/old", "started": "2026-02-01T00:00:00Z"}),+ json!({"doc_kind": "run", "prefix": "proj", "root": "/data/proj", "started": "2026-03-01T00:00:00Z"}),+ ];+ assert_eq!(+ other_prefixes_for_root(&runs, "/data/proj", "proj"),+ vec!["ax".to_string()]+ );+ let line = same_root_other_prefix_line("proj", &["ax".into()]);+ assert!(line.contains("under ax"), "{line}");+ // `ax` also holds /data/old here, so a delete hint would destroy a+ // folder this run never touched (#2114).+ assert!(!line.contains("DELETE"), "{line}");+ assert!(line.contains("--prefix ax"), "{line}");+ assert_eq!(+ other_prefixes_for_root(&runs, "/data/proj", "ax"),+ vec!["proj".to_string()]+ );+ }+#[test]fn replace_prefix_says_the_run_continues() {let message = prefix_reuse_message("ref", "/data/new", &["/data/old".into()], true);diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex d177e5739..e95c4934e 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -111,6 +111,13 @@ pub struct IndexCfg {/// An answer to a previous decision request. `None` means "nobody has/// answered yet", which is what arms the gate.pub approve: Option<Approval>,+ /// `--max-index-size`, in bytes. `None` means the only budget is the free+ /// space under the node's flood-stage watermark, when the node reports one.+ pub max_index_bytes: Option<u64>,+ /// `--plan`. `None` means the default plan, which is indexed when it fits+ /// and refused when it does not. A named plan runs even when the estimate+ /// says it does not fit (with a warning): that is the override.+ pub budget_plan: Option<String>,pub json: bool,pub quiet: bool,/// Progress surface. Orthogonal to `json`: `--json` shapes *stdout* (the@@ -380,7 +387,19 @@ pub fn help_text_with(feedback: bool) -> String {\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20`fast` also applies --no-semantic --no-graph. `narrower`\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20is NOT accepted here — it means re-running against a\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20subdirectory, which this flag cannot do.\n\- \x20\x20\x20\x20--yes, -y alias for --approve proceed\n\+ \x20\x20\x20\x20--yes, -y alias for --approve proceed. When the default index\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20plan does not fit the disk budget, --yes takes the\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20recommended plan that does fit. It does not index a\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20plan that is over budget.\n\+ \x20\x20\x20\x20--max-index-size <N> cap the estimated index (bytes, or a KB/MB/GB suffix).\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20The disk budget is the free space under the node's\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20flood-stage watermark; this flag lowers that cap.\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x200 is rejected\n\+ \x20\x20\x20\x20--plan <NAME> index with one plan: default, skip-vendored, skip-data,\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20code-and-docs, no-vectors, cap-1mb,\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20code-and-docs-no-vectors. A named plan runs even\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20when the estimate says it does not fit (warned).\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20See DISK BUDGET.\n\\x20\x20\x20\x20--dry-run walk+sniff+infer, print the plan, index nothing\n\\x20\x20\x20\x20--label <FILE> label each document at ingest through POST /_decide\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20(#1062): the question-set file's noul/choice questions\n\@@ -659,6 +678,23 @@ pub fn help_text_with(feedback: bool) -> String {\x20\x20\x20\x20Credentials come from the environment only: AWS_ACCESS_KEY_ID,\n\\x20\x20\x20\x20AWS_SECRET_ACCESS_KEY, optionally AWS_SESSION_TOKEN. Never from a flag.\n\\n\+ DISK BUDGET:\n\+ \x20\x20\x20\x20Before hashing, autoindex counts files by extension (code, prose, pdf,\n\+ \x20\x20\x20\x20data, binary, other) and by top-level folder, and estimates the index\n\+ \x20\x20\x20\x20from measured per-kind ratios. The budget is the free space on the\n\+ \x20\x20\x20\x20node's data volume under the flood-stage watermark (the same\n\+ \x20\x20\x20\x20GET /_cat/allocation read that refuses a volume already at the\n\+ \x20\x20\x20\x20watermark), optionally lowered with --max-index-size. A node that does\n\+ \x20\x20\x20\x20not report disk.flood_stage is not guessed at. When the default plan\n\+ \x20\x20\x20\x20fits, the run continues. When it does not, stdout carries a JSON plan\n\+ \x20\x20\x20\x20list and the run exits 4 unless --yes takes the recommended plan that\n\+ \x20\x20\x20\x20fits, or --plan names one (a named plan is the override: it runs\n\+ \x20\x20\x20\x20over the estimate with a warning; the estimate errs high).\n\+ \x20\x20\x20\x20A terminal with the question visible is asked. Kinds without a\n\+ \x20\x20\x20\x20calibration are reported as unmeasured and do not refuse the run.\n\+ \x20\x20\x20\x20Skipped files are listed after the ok: line (up to 20, then a count).\n\+ \x20\x20\x20\x20A folder already indexed under another prefix is named before the walk.\n\+ \n\EXIT CODES: 0 complete (also: gate answered with --approve cancel);\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x203 completed-with-junk (junk recorded, never fatal) — SUCCESS:\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20the index is searchable, and the summary on stdout opens\n\@@ -678,9 +714,11 @@ pub fn help_text_with(feedback: bool) -> String {\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20`skipped_appeared` names them. Re-run with --fresh to\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20rebuild the plan in place and include them. Read `reason`\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20to tell this apart from completed-with-junk;\n\- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x204 NEEDS A DECISION — the estimate exceeded --max-minutes, or a\n\- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20--watch poll cost more than --max-monthly-ops; nothing was\n\- \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20indexed and a JSON decision request is on stdout;\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x204 NEEDS A DECISION — the estimate exceeded --max-minutes, the\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20estimated index does not fit the disk budget, or a --watch\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20poll cost more than --max-monthly-ops; nothing was indexed\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20and a JSON decision request is on stdout. A plan that does\n\+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20not fit is not started, including with --yes;\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x202 usage; 1 endpoint/journal failure, a refused corpus removal, or a\n\\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20refused unsafe state transition\n",feedback_block = xerj_common::feedback::block(feedback),@@ -825,6 +863,9 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {let mut max_minutes_explicit = false;let mut approve: Option<Approval> = None;let mut approve_explicit = false;+ let mut max_index_bytes: Option<u64> = None;+ let mut max_index_explicit = false;+ let mut budget_plan: Option<String> = None;let mut json = false;let mut quiet = false;let mut progress: Option<ProgressMode> = None;@@ -1043,6 +1084,31 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {approve = Some(parsed);approve_explicit = true;}+ "--max-index-size" => {+ max_index_explicit = true;+ let raw = it+ .next()+ .ok_or("--max-index-size needs a size in bytes or with a KB/MB/GB suffix")?;+ max_index_bytes = Some(crate::budget::parse_byte_size(&raw)?);+ }+ "--plan" => {+ let raw = it.next().ok_or_else(|| {+ format!(+ "--plan needs one of: {}",+ crate::budget::PLAN_NAMES.join(", ")+ )+ })?;+ if !crate::budget::PLAN_NAMES.contains(&raw.as_str()) {+ return Err(format!(+ "--plan {raw} is not a plan. choices: {}",+ crate::budget::PLAN_NAMES.join(", ")+ ));+ }+ if budget_plan.is_some() {+ return Err("--plan was given twice".into());+ }+ budget_plan = Some(raw);+ }"--yes" | "-y" => {if let Some(previous) = approve {if previous != Approval::Proceed {@@ -1228,6 +1294,10 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {if prefix.is_empty() {return Err("--prefix must contain at least one [a-z0-9] character".into());}+ // #2057: reject a prefix that cannot name an index before the walk.+ if let Some(err) = crate::dataset::index_prefix_error(&prefix) {+ return Err(err);+ }// Contradictory progress requests are refused, never silently resolved in// one side's favour: accepting a flag we will not honour is the defect@@ -1524,10 +1594,10 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {.into(),);}- if max_minutes_explicit || approve_explicit {+ if max_minutes_explicit || approve_explicit || max_index_explicit || budget_plan.is_some() {return Err(- "--max-minutes/--approve/--yes are the folder-indexing gate; a watch has its own \- cost gate (--max-monthly-ops, --allow-cost)"+ "--max-minutes/--approve/--yes/--max-index-size/--plan are the folder-indexing \+ gate; a watch has its own cost gate (--max-monthly-ops, --allow-cost)".into(),);}@@ -1600,9 +1670,15 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {}match (sub.as_deref(), folder) {- (Some("map"), _) | (Some("status"), _) if max_minutes_explicit || approve_explicit => {+ (Some("map"), _) | (Some("status"), _)+ if max_minutes_explicit+ || approve_explicit+ || max_index_explicit+ || budget_plan.is_some() =>+ {Err(format!(- "--max-minutes/--approve/--yes apply only to indexing, not `autoindex {}`",+ "--max-minutes/--approve/--yes/--max-index-size/--plan apply only to indexing, \+ not `autoindex {}`",sub.as_deref().unwrap_or_default()))}@@ -1738,6 +1814,8 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {dry_run,max_minutes,approve,+ max_index_bytes,+ budget_plan,json,quiet,progress,@@ -2646,6 +2724,9 @@ mod tests {"--max-minutes","--approve <ID>","--yes, -y",+ "--max-index-size",+ "--plan <NAME>",+ "DISK BUDGET:","0 disables the gate","ESTIMATE + DECISION GATE:","exits 4",@@ -2663,6 +2744,20 @@ mod tests {}}+ /// #2057: a digit-leading prefix is a usage error, before any walk. The+ /// budget flags parse here too, and an unknown plan is refused.+ #[test]+ fn a_digit_leading_prefix_and_an_unknown_plan_are_usage_errors() {+ let prefix_err = err(&["data", "--prefix", "2042"]);+ assert!(prefix_err.contains("p2042"), "{prefix_err}");+ assert!(prefix_err.contains("lowercase letter"), "{prefix_err}");+ let cfg = index(&["data", "--plan", "code-and-docs", "--max-index-size", "2MB"]);+ assert_eq!(cfg.budget_plan.as_deref(), Some("code-and-docs"));+ assert_eq!(cfg.max_index_bytes, Some(2 * 1024 * 1024));+ let plan_err = err(&["data", "--plan", "everything"]);+ assert!(plan_err.contains("not a plan"), "{plan_err}");+ }+/// A `\` at the end of a help line used to eat the next line's indent, so/// `--limit`'s wrap started at column 0. The wrap stays under the/// description.diff --git a/engine/crates/xerj-autoindex/src/dataset.rs b/engine/crates/xerj-autoindex/src/dataset.rsindex 61303d74a..427eee7e1 100644--- a/engine/crates/xerj-autoindex/src/dataset.rs+++ b/engine/crates/xerj-autoindex/src/dataset.rs@@ -330,6 +330,30 @@ pub fn sanitize_prefix(s: &str) -> String {out}+/// Why `{prefix}-docs` cannot be an index name, if it cannot.+///+/// #2057: the server used to be the first thing that noticed a digit-leading+/// prefix, and it noticed after the hash. `None` means the prefix can name+/// an index.+pub fn index_prefix_error(prefix: &str) -> Option<String> {+ let index = format!("{prefix}-docs");+ match xerj_common::IndexName::validate(&index) {+ Ok(()) => None,+ Err(err) => {+ let hint = if prefix.starts_with(|c: char| c.is_ascii_digit()) {+ format!(" A prefix that starts with a letter works, for example p{prefix}.")+ } else {+ String::new()+ };+ Some(format!(+ "--prefix {prefix} is not a valid index prefix: an index is named \+ {prefix}-<dataset>, and `{index}` is rejected ({err}). An index name must \+ start with a lowercase letter or '.'.{hint} Nothing was walked."+ ))+ }+ }+}+fn numericish_segment(s: &str) -> bool {let t = s.trim();!t.is_empty()@@ -497,6 +521,15 @@ pub fn drop_repeated_prefix(prefix: &str, clusters: &mut [Cluster]) {mod tests {use super::*;+ #[test]+ fn a_digit_leading_prefix_is_rejected_before_any_walk() {+ let err = index_prefix_error("2042").expect("2042-docs is not an index name");+ assert!(err.contains("lowercase letter"), "{err}");+ assert!(err.contains("p2042"), "{err}");+ assert!(err.contains("Nothing was walked"), "{err}");+ assert!(index_prefix_error("proj").is_none());+ }+#[test]fn sanitize_prefix_keeps_a_literal_double_hyphen() {// issue #1889: `owner--repo` corpora must index under the name they verify underdiff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rsindex 337acff22..529b8d48c 100644--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs@@ -376,6 +376,8 @@ pub(super) fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {// behaviour is covered in `gate_tests` and `cli::tests`.max_minutes: 0,approve: None,+ max_index_bytes: None,+ budget_plan: None,dry_run: false,json: false,quiet: true,diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rsindex ac61fdf98..cf9cf7431 100644--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs@@ -814,6 +814,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {// behaviour is covered in `gate_tests` and `cli::tests`.max_minutes: 0,approve: None,+ max_index_bytes: None,+ budget_plan: None,dry_run: false,json: false,quiet: true,diff --git a/engine/crates/xerj-autoindex/src/flood.rs b/engine/crates/xerj-autoindex/src/flood.rsindex 8d84c708b..3ae8ececa 100644--- a/engine/crates/xerj-autoindex/src/flood.rs+++ b/engine/crates/xerj-autoindex/src/flood.rs@@ -20,15 +20,73 @@ use crate::esclient::Es;const ALLOCATION_PATH: &str = "/_cat/allocation?format=json";-/// `None` when the node cannot be asked or does not report a flood-stage-/// threshold. A stats blip must not abort a run whose disk is fine, and the-/// write path still refuses if the block is actually engaged.-pub(crate) fn refusal(es: &Es) -> Option<String> {- let (status, body) = es.request_json("GET", ALLOCATION_PATH, None).ok()?;+/// One row of `GET /_cat/allocation?format=json`.+///+/// The budget gate reads free space and the flood threshold from this same+/// response. It does not issue a second probe, and it does not fill in a+/// threshold the node left out.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub(crate) struct Volume {+ pub total: Option<u64>,+ pub avail: Option<u64>,+ pub used_percent: Option<u64>,+ pub flood_stage: Option<u8>,+ pub threshold_enabled: bool,+ pub flood_blocked: bool,+}++/// The one allocation read: the parsed volume, and the refusal when the+/// node says writes will fail.+#[derive(Debug, Clone, PartialEq, Eq)]+pub(crate) struct Probe {+ pub volume: Option<Volume>,+ pub refusal: Option<String>,+}++/// `None` fields when the node cannot be asked. A stats blip must not abort+/// a run whose disk is fine, and the write path still refuses if the block+/// is actually engaged.+pub(crate) fn probe(es: &Es) -> Probe {+ let Ok((status, body)) = es.request_json("GET", ALLOCATION_PATH, None) else {+ return Probe {+ volume: None,+ refusal: None,+ };+ };if !(200..300).contains(&status) {- return None;+ return Probe {+ volume: None,+ refusal: None,+ };}- refusal_from_allocation(&body)+ Probe {+ volume: volume_from_allocation(&body),+ refusal: refusal_from_allocation(&body),+ }+}++/// Parsed disk columns from one allocation body. `None` when the body is+/// not an allocation row (the HTTP doubles answer `{"acknowledged": true}`).+pub(crate) fn volume_from_allocation(body: &Value) -> Option<Volume> {+ let row = body.as_array()?.first()?;+ let flood_stage = row+ .get("disk.flood_stage")+ .and_then(parse_u64)+ .and_then(|n| u8::try_from(n.min(100)).ok());+ Some(Volume {+ total: row.get("disk.total").and_then(parse_bytes),+ avail: row.get("disk.avail").and_then(parse_bytes),+ used_percent: row.get("disk.percent").and_then(parse_u64),+ flood_stage,+ threshold_enabled: row+ .get("disk.threshold_enabled")+ .and_then(parse_bool)+ .unwrap_or(true),+ flood_blocked: row+ .get("disk.flood_blocked")+ .and_then(parse_bool)+ .unwrap_or(false),+ })}/// Decide from one `/_cat/allocation?format=json` body.@@ -214,6 +272,24 @@ mod tests {assert!(refusal_from_allocation(&json!([])).is_none());}+ #[test]+ fn the_same_row_carries_free_and_total_bytes_for_the_budget() {+ let body = json!([{+ "disk.percent": "40",+ "disk.avail": "1000b",+ "disk.total": "10000b",+ "disk.flood_stage": "95",+ "disk.threshold_enabled": "true",+ "disk.flood_blocked": "false",+ }]);+ let volume = volume_from_allocation(&body).expect("an allocation row");+ assert_eq!(volume.avail, Some(1000));+ assert_eq!(volume.total, Some(10000));+ assert_eq!(volume.used_percent, Some(40));+ assert_eq!(volume.flood_stage, Some(95));+ assert!(volume_from_allocation(&json!({"acknowledged": true})).is_none());+ }+/// The probe runs before the walk: a flooded node refuses with the/// override text and never opens the folder, and a healthy node is not/// refused (the walk then reports that the root is a file).diff --git a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rsindex 5dffa0c1a..3027e6e8e 100644--- a/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/incremental_reconcile_http_tests.rs@@ -970,6 +970,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, semantic: bool) -> IndexCfg {// timing-derived stop. See `gate_tests` and `cli::tests`.max_minutes: 0,approve: None,+ max_index_bytes: None,+ budget_plan: None,dry_run: false,json: false,quiet: true,@@ -6410,6 +6412,93 @@ mod delete_gate {assert!(landed, "the replayed document landed");}+ /// A cap between the measured `code-and-docs` estimate and the default+ /// estimate stops before any data bulk (exit 4). `--plan code-and-docs`+ /// is the plan that fits, and the csv it drops is not indexed. The sizes+ /// come from `budget::MEASURED`. An empty table fails here: skipping would+ /// hide a gate that can no longer tell the plans apart.+ #[test]+ fn a_budget_too_small_for_the_default_plan_accepts_a_smaller_one() {+ let _guard = HTTP_E2E_LOCK.lock().unwrap_or_else(|p| p.into_inner());+ let corpus = tempfile::tempdir().unwrap();+ fs::write(+ corpus.path().join("main.rs"),+ "fn main() { println!(\"budget\"); }\n",+ )+ .unwrap();+ let mut csv = String::from("id,value\n");+ for i in 0..2000 {+ csv.push_str(&format!("{i},row-{i}-budget-padding\n"));+ }+ fs::write(corpus.path().join("big.csv"), csv).unwrap();++ let (files, _) = walk::walk_reporting_opts(+ corpus.path(),+ false,+ false,+ crate::ignore_rules::IgnoreOptions::default(),+ )+ .unwrap();+ let priced = budget::assess(&budget::Request {+ root: corpus.path().to_str().unwrap_or("/corpus"),+ files: &files,+ volume: None,+ max_index_bytes: Some(u64::MAX),+ no_semantic: true,+ plan: None,+ approve: None,+ duplicate_prefixes: &[],+ });+ let plan_bytes = |name: &str| -> u64 {+ let plan = priced.body["plans"]+ .as_array()+ .and_then(|plans| plans.iter().find(|plan| plan["name"] == name))+ .unwrap_or_else(|| panic!("missing plan {name} in {}", priced.body));+ plan["index_bytes"].as_u64().unwrap_or_else(|| {+ panic!(+ "plan {name} has no measured index size. Fill budget::MEASURED from a real \+ xerj run; an unmeasured kind must not be given a guessed ratio. {}",+ priced.body+ )+ })+ };+ let default_bytes = plan_bytes("default");+ let reduced_bytes = plan_bytes("code-and-docs");+ assert!(+ reduced_bytes < default_bytes,+ "code-and-docs ({reduced_bytes}) has to be smaller than default ({default_bytes}) \+ so the cap can sit between them"+ );++ let endpoint = HttpEndpoint::start();+ let refused_state = tempfile::tempdir().unwrap();+ let mut refused = cfg(corpus.path(), refused_state.path(), &endpoint.url, false);+ refused.prefix = "budget-http".into();+ refused.max_index_bytes = Some(reduced_bytes);+ assert_eq!(run_index(refused).unwrap(), gate::EXIT_NEEDS_DECISION);+ assert_eq!(+ endpoint.data_bulk_requests(),+ 0,+ "a plan that does not fit must not bulk-index"+ );++ let accepted_state = tempfile::tempdir().unwrap();+ let mut accepted = cfg(corpus.path(), accepted_state.path(), &endpoint.url, false);+ accepted.prefix = "budget-http".into();+ accepted.max_index_bytes = Some(reduced_bytes);+ accepted.budget_plan = Some("code-and-docs".into());+ assert_eq!(run_index(accepted).unwrap(), 0);+ let indexed = paths(&endpoint.data_docs());+ assert!(+ indexed.iter().any(|path| path.ends_with("main.rs")),+ "the code file is the part the plan keeps: {indexed:?}"+ );+ assert!(+ indexed.iter().all(|path| !path.ends_with("big.csv")),+ "the csv is the part the plan drops: {indexed:?}"+ );+ }+fn base() -> CommittedManifest {CommittedManifest {generation: 0,diff --git a/engine/crates/xerj-autoindex/src/label_http_tests.rs b/engine/crates/xerj-autoindex/src/label_http_tests.rsindex 426ad13a9..941150495 100644--- a/engine/crates/xerj-autoindex/src/label_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/label_http_tests.rs@@ -384,6 +384,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str, label: &Path) -> IndexCfg {no_graph: true,max_minutes: 0,approve: None,+ max_index_bytes: None,+ budget_plan: None,dry_run: false,json: false,quiet: true,diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 98fbbf36d..92e96e5b0 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -3,6 +3,7 @@//! it does NOT link xerj-engine, works against any endpoint, and cannot//! destabilize the server.+pub mod budget;pub mod catalog;pub mod cli;pub mod coerce;@@ -3520,6 +3521,8 @@ mod phase_a_grouping_tests {// behaviour is covered in `gate_tests` and `cli::tests`.max_minutes: 0,approve: None,+ max_index_bytes: None,+ budget_plan: None,dry_run: true,json: false,quiet: true,@@ -3997,7 +4000,7 @@ fn report_pdf_spool_fallbacks(}for (path, fallback) in reasons.iter().take(3) {pr.note(&format!(- " PDF reuse fallback for {path}: {}",+ " PDF reuse fallback for {path}: {}. The PDF is still indexed; phase B parses it again.",fallback.message));}@@ -5370,6 +5373,14 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -let junk: Vec<&JunkFile> = committed.plan.junk_files.iter().collect();unextracted_archive_lines(&junk)};+ // Taken while `committed` still borrows the journal. `journal.finish`+ // needs that borrow back, and the printed summary (human and --json)+ // lists the same skips the plan recorded (#2056).+ let skip_lines = skipped_lines_from_plan(&committed.plan.junk_files);+ let skip_doc = {+ let refs: Vec<&JunkFile> = committed.plan.junk_files.iter().collect();+ skipped_json(&refs)+ };let sync::SourceExecutionPolicy::DurableSnapshot { reference, .. } = &execution.source_policyelse {anyhow::bail!("generated run does not reference a durable snapshot");@@ -5419,6 +5430,9 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -if es.bulk_requests_split() > 0 {summary["bulk_splits"] = json!(es.bulk_requests_split());}+ if let Some(skipped) = skip_doc {+ summary["skipped"] = skipped;+ }journal.finish(&summary)?;if cfg.json {println!("{summary}");@@ -5434,6 +5448,9 @@ fn finish_generated_run(es: &Es, journal: &mut state::Journal, cfg: &IndexCfg) -0,));+ for line in &skip_lines {+ println!("{line}");+ }println!("generation {} committed — {}, {} live",generation,@@ -6041,6 +6058,65 @@ const UNEXTRACTED_PREFIX: &str = "unextracted ";/// "successful" empty index with the one sentence that explains it filed where/// they would never look (review finding on PR #949). These few lines are the/// run saying it.+const SKIP_LIST_CAP: usize = 20;++/// #2056: names of skipped files, capped, with the reason. The headline stays+/// one `ok:` line; this is the list under it.+fn skipped_lines(junk: &[&JunkFile]) -> Vec<String> {+ if junk.is_empty() {+ return Vec::new();+ }+ let mut rows: Vec<&&JunkFile> = junk.iter().collect();+ rows.sort_by(|a, b| a.rel.cmp(&b.rel).then(a.reason.cmp(&b.reason)));+ let mut lines = vec![format!(+ "skipped: {}",+ estimate::count_phrase(rows.len() as u64, "file", "files")+ )];+ for jf in rows.iter().take(SKIP_LIST_CAP) {+ lines.push(format!(+ " skipped: {} ({}: {})",+ jf.rel, jf.status, jf.reason+ ));+ }+ if rows.len() > SKIP_LIST_CAP {+ lines.push(format!(+ " … and {} more; `xerj autoindex map` lists every one",+ rows.len() - SKIP_LIST_CAP+ ));+ }+ lines+}++fn skipped_lines_from_plan(junk: &[JunkFile]) -> Vec<String> {+ let refs: Vec<&JunkFile> = junk.iter().collect();+ skipped_lines(&refs)+}++fn skipped_json(junk: &[&JunkFile]) -> Option<Value> {+ if junk.is_empty() {+ return None;+ }+ let mut rows: Vec<&&JunkFile> = junk.iter().collect();+ rows.sort_by(|a, b| a.rel.cmp(&b.rel).then(a.reason.cmp(&b.reason)));+ let total = rows.len();+ let files: Vec<Value> = rows+ .iter()+ .take(SKIP_LIST_CAP)+ .map(|jf| {+ json!({+ "path": jf.rel,+ "status": jf.status,+ "reason": jf.reason,+ })+ })+ .collect();+ let mut doc = json!({ "count": total, "files": files });+ if total > SKIP_LIST_CAP {+ doc["not_listed"] = json!(total - SKIP_LIST_CAP);+ }+ Some(doc)+}+fn unextracted_archive_lines(junk: &[&JunkFile]) -> Vec<String> {const SHOWN: usize = 10;let mut hits: Vec<&&JunkFile> = junk@@ -6332,6 +6408,14 @@ fn legacy_rebuild_argv(cfg: &IndexCfg, state_dir: &Path, prefix: &str) -> Vec<Stif cfg.no_semantic {argv.push("--no-semantic".to_owned());}+ if let Some(bytes) = cfg.max_index_bytes {+ argv.push("--max-index-size".to_owned());+ argv.push(bytes.to_string());+ }+ if let Some(plan) = &cfg.budget_plan {+ argv.push("--plan".to_owned());+ argv.push(plan.clone());+ }if cfg.follow_symlinks {argv.push("--follow-symlinks".to_owned());}@@ -6365,12 +6449,11 @@ fn legacy_rebuild_argv(cfg: &IndexCfg, state_dir: &Path, prefix: &str) -> Vec<St/// check on a 401 or a 500 would let the overwrite this guard exists to stop/// proceed. The caller re-checks the prefix exactly; a legacy analyzed/// `prefix` field can over-return sibling corpora.-fn catalog_runs_for_prefix(es: &Es, prefix: &str) -> Result<Vec<Value>> {+fn catalog_runs(es: &Es) -> Result<Vec<Value>> {let body = json!({"size": 500,"query": {"bool": {"filter": [- {"term": {"doc_kind": "run"}},- {"term": {"prefix": prefix}}+ {"term": {"doc_kind": "run"}}]}}});let Some(page) = es.search_present(catalog::CATALOG_INDEX, &body)? else {@@ -6465,6 +6548,12 @@ fn run_index_report_inner(// throttle. Its shrink/recover announcements go to stderr, which the// progress surface owns — so they are emitted exactly when that surface is// enabled, not merely when `--quiet` is absent.+ // #2057: a digit-leading prefix is rejected by the server only when the+ // index is created, which used to be after the hash. The name is wrong+ // before any walk, and it does not need the node.+ if let Some(err) = dataset::index_prefix_error(&cfg.prefix) {+ anyhow::bail!("{err}");+ }let es = Es::with_bulk_timeout(&cfg.url, cfg.api_key.clone(), cfg.bulk_timeout_secs)?.with_bulk_concurrency(cfg.workers, pr.enabled());// #1102: ping's raw context is honest but teaches a first-time user@@ -6482,13 +6571,18 @@ fn run_index_report_inner(// `limits.disk_flood_stage_percent = 99` workaround and a disabled// watermark. Nothing has been walked or written yet, so bailing here// leaves no journal and no remote mutation.+ // One allocation read (#2077). The refusal bails before the walk. The+ // parsed volume is what the budget gate, after the walk and still before+ // the hash, compares the estimated index against. A second GET is not made.// A dry run writes nothing, so a full volume is not a reason to refuse// it: the plan (and its size) is exactly what a user short on disk needs.+ let disk = flood::probe(&es);if !cfg.dry_run {- if let Some(message) = flood::refusal(&es) {+ if let Some(message) = disk.refusal {anyhow::bail!("{message}");}}+ let disk_volume = disk.volume;// `--label <question-set>` (#1062): every record's payload goes through// the node's /_decide at ingest and the answers are stamped on as@@ -6553,12 +6647,10 @@ fn run_index_report_inner(// only remote calls on that path are the readiness GET and the read-only// flood-stage probe (#2077), so this guard must not search the catalog// there. A different folder has no such journal and still hits the check.+ let mut duplicate_prefixes: Vec<String> = Vec::new();if !state::journal_continues_folder(&state_dir, &root_str, &cfg.url, &cfg.prefix) {- let foreign_roots = catalog::foreign_prefix_roots(- &catalog_runs_for_prefix(&es, &cfg.prefix)?,- &cfg.prefix,- &root_str,- );+ let runs = catalog_runs(&es)?;+ let foreign_roots = catalog::foreign_prefix_roots(&runs, &cfg.prefix, &root_str);if !foreign_roots.is_empty() {let message = catalog::prefix_reuse_message(&cfg.prefix,@@ -6577,6 +6669,16 @@ fn run_index_report_inner(anyhow::bail!("{message}");}}+ // #2078: the same folder under a second prefix writes a second copy+ // and used to say nothing. This is a warning, not a refusal — the+ // prefixes do not share catalog ids.+ duplicate_prefixes = catalog::other_prefixes_for_root(&runs, &root_str, &cfg.prefix);+ if !duplicate_prefixes.is_empty() {+ pr.warn(&catalog::same_root_other_prefix_line(+ &cfg.prefix,+ &duplicate_prefixes,+ ));+ }}// Acquire state authority before discovery as well as hashing. A waiter// must never classify a path snapshot taken while another owner was@@ -6849,7 +6951,7 @@ fn run_index_report_inner(cfg.exclude_rels.join(", ")));}- let (discovered_files, data_files, data_bytes) = if cfg.skip_corpus_data {+ let (mut discovered_files, data_files, data_bytes) = if cfg.skip_corpus_data {without_corpus_data_files(discovered_files)} else {(discovered_files, 0, 0)@@ -6865,7 +6967,7 @@ fn run_index_report_inner(is data, not reference code — pass --include-data to index them"));}- let discovered_bytes: u64 = discovered_files.iter().map(|f| f.size).sum();+ let mut discovered_bytes: u64 = discovered_files.iter().map(|f| f.size).sum();pr.note(&format!("autoindex: {} files ({} MB) under {}",discovered_files.len(),@@ -6885,6 +6987,82 @@ fn run_index_report_inner(pr.finish(true, 0, "no-files", &[]);return Ok((0, None));}+ // Count, estimate, and — when the default plan does not fit the disk+ // budget — stop before hashing. The volume is the allocation row read+ // above; this does not probe the disk again.+ {+ let assessment = budget::assess(&budget::Request {+ root: &root_str,+ files: &discovered_files,+ volume: disk_volume,+ max_index_bytes: cfg.max_index_bytes,+ no_semantic: cfg.no_semantic,+ plan: cfg.budget_plan.as_deref(),+ approve: cfg.approve,+ duplicate_prefixes: &duplicate_prefixes,+ });+ for line in &assessment.lines {+ if assessment.needs_choice {+ pr.note(line);+ }+ }+ let choice = if assessment.needs_choice {+ let blocked = gate::detect_prompt_block(pr.enabled());+ if let Some(why) = blocked {+ pr.note(&format!(+ "budget: {}. Re-run with --plan <name> or --yes",+ why.explain()+ ));+ budget::Choice::Absent+ } else {+ budget::read_choice(+ &mut std::io::stdin().lock(),+ &assessment.plan_names,+ |line| {+ pr.note(line);+ },+ )+ }+ } else {+ budget::Choice::Absent+ };+ match budget::resolve(&assessment, choice) {+ budget::Outcome::Stop { code, body } => {+ if !body.is_null() {+ println!("{}", serde_json::to_string(&body)?);+ }+ if code == 0 {+ pr.note("gate: --approve cancel — nothing was indexed");+ pr.finish(true, 0, "cancelled", &[]);+ } else {+ pr.finish(false, code, "needs-decision", &[]);+ }+ return Ok((code, None));+ }+ budget::Outcome::Continue {+ keep,+ no_semantic,+ note,+ } => {+ if no_semantic {+ cfg.no_semantic = true;+ }+ if let Some(keep) = keep {+ let keep: std::collections::HashSet<String> = keep.into_iter().collect();+ let before = discovered_files.len();+ discovered_files.retain(|f| keep.contains(&f.rel));+ discovered_bytes = discovered_files.iter().map(|f| f.size).sum();+ pr.note(&format!(+ "autoindex: budget plan kept {} of {before} walked files",+ discovered_files.len()+ ));+ }+ if !note.is_empty() {+ pr.note(&format!("autoindex: {note}"));+ }+ }+ }+ }// Full hashing on every run is deliberate: size/mtime/inode fingerprints// cannot prove byte identity across all supported local and network// filesystems. A metadata-only shortcut could leave stale live documents@@ -9824,6 +10002,15 @@ fn run_index_report_inner(if skipped_appeared > 0 {run_doc["skipped_appeared"] = json!(skipped_appeared);}+ // #2056: the exit-3 headline stays one line. The paths themselves are a+ // separate list, capped, with the reason. Present only when something+ // was skipped, so a clean run document is unchanged.+ if let Some(skipped) = skipped_json(&all_junk) {+ run_doc["skipped"] = skipped;+ }+ // Owned lines: the resume-plan update below mutates `plan.junk_files`,+ // which `all_junk` borrows. The summary prints these after that update.+ let skip_lines = skipped_lines(&all_junk);push_doc(&format!("run:{run_id}"), &run_doc, &mut cat_buf);if !cat_buf.is_empty() {@@ -9921,6 +10108,9 @@ fn run_index_report_inner(skipped_appeared as u64,));+ for line in &skip_lines {+ println!("{line}");+ }println!("{}",estimate::done_summary_line(diff --git a/engine/crates/xerj-console-api/src/knowledge.rs b/engine/crates/xerj-console-api/src/knowledge.rsindex 6133ec207..78e24004d 100644--- a/engine/crates/xerj-console-api/src/knowledge.rs+++ b/engine/crates/xerj-console-api/src/knowledge.rs@@ -495,7 +495,7 @@ fn capabilities(caps.push(json!({"id": "map","title": "Print the data map",- "blurb": "The same field/coverage table this page shows, as markdown — for you or for an agent (MCP: xerj_map; `xerj mcp` exposes 18 tools).",+ "blurb": "The same field/coverage table this page shows, as markdown — for you or for an agent (MCP: xerj_map; `xerj mcp` exposes 19 tools).","command": "xerj autoindex map","kind": "cli",}));diff --git a/engine/crates/xerj-mcp/Cargo.toml b/engine/crates/xerj-mcp/Cargo.tomlindex d6dbbf102..e32089d8a 100644--- a/engine/crates/xerj-mcp/Cargo.toml+++ b/engine/crates/xerj-mcp/Cargo.toml@@ -49,6 +49,10 @@ unicode-segmentation = "1"# itself (`xerj_hub::ask`), the one implementation the CLI and the gRPC# service share — the tools are adapters, not a second copy of the flow.xerj-hub = { path = "../xerj-hub" }+# `xerj_index_budget` walks a folder and prices it before hashing. The walk,+# the flood-stage allocation parse, and the plan list live in xerj-autoindex;+# this crate only exposes them. No cycle: autoindex does not depend on mcp.+xerj-autoindex = { path = "../xerj-autoindex" }[dev-dependencies]# The loopback admin-key fallback tests sandbox cwd + HOME so discovery seesdiff --git a/engine/crates/xerj-mcp/src/lib.rs b/engine/crates/xerj-mcp/src/lib.rsindex 988998d20..950894b96 100644--- a/engine/crates/xerj-mcp/src/lib.rs+++ b/engine/crates/xerj-mcp/src/lib.rs@@ -9,11 +9,12 @@//!//! It speaks the MCP **stdio transport**: newline-delimited//! JSON-RPC 2.0 messages on stdin/stdout. It exposes XERJ to any MCP-capable-//! agent host (Claude Desktop, IDE agents, custom orchestrators) as eighteen-//! tools: fifteen without a hub key, and three Corpus Hub tools+//! agent host (Claude Desktop, IDE agents, custom orchestrators) as nineteen+//! tools: sixteen without a hub key, and three Corpus Hub tools//! (`xerj_hub_*`) that run the `xerj ask` flow itself. Thirteen of those-//! fifteen construct exactly the request the running engine already accepts-//! and forward it to a configurable base URL. `xerj_corpus_feedback` posts an+//! sixteen construct exactly the request the running engine already accepts+//! and forward it to a configurable base URL. `xerj_index_budget` counts a+//! folder and returns the budget plans; it does not index. `xerj_corpus_feedback` posts an//! agent's review of a hub corpus to hub.xerj.org. `xerj_field_report` does not://! it files a field report, not to the node: a GitHub PR first (through//! `xerj feedback --submit --github-only`), and a POST to the web endpoint only@@ -1127,6 +1128,47 @@ pub fn tool_specs_with(include_hub: bool) -> Value {}}},+ {+ "name": "xerj_index_budget",+ "description": concat!(+ "Count a folder and estimate the index BEFORE xerj autoindex hashes or ",+ "embeds anything. Returns file and byte counts by kind (code, prose, pdf, ",+ "data, binary, other — by extension, before content sniff) and by top-level ",+ "folder, the free-space budget from the node's flood-stage watermark ",+ "(the same GET /_cat/allocation row autoindex uses; a node that omits ",+ "disk.flood_stage is not guessed at), and one or more plans with what ",+ "each drops. This tool does not index. When decision_required is true, ",+ "show the plans to the user and re-run `xerj autoindex` with --plan <name> ",+ "or --yes for the recommended plan that fits. Do not start a run that ",+ "does not fit. Default embeddings in the estimate are the server's ",+ "lexical feature-hash vectors unless that node was started with ",+ "--embed-mode neural. Kinds without a measured calibration are ",+ "unmeasured and do not refuse the run."+ ),+ "inputSchema": {+ "type": "object",+ "properties": {+ "path": {+ "type": "string",+ "description": "Folder to count. Nothing inside it is hashed."+ },+ "prefix": {+ "type": "string",+ "description": "Index prefix to check for an existing copy of this folder (default ax)."+ },+ "max_index_bytes": {+ "type": "integer",+ "minimum": 1,+ "description": "Optional cap on the estimated index, in bytes. The disk budget is the free space under the flood-stage watermark; this lowers it."+ },+ "no_semantic": {+ "type": "boolean",+ "description": "Price the plans without semantic_text vectors, as --no-semantic would."+ }+ },+ "required": ["path"]+ }+ },{"name": "xerj_code_search","description": concat!(@@ -1769,6 +1811,7 @@ async fn call_tool(ctx: &Ctx, msg: &Value) -> Value {// code_search — it renders the per-index field map itself, so it// returns a tool result directly instead of a BuiltRequest."xerj_map" => return map::run(ctx, &args).await,+ "xerj_index_budget" => return run_index_budget(ctx, &args).await,"xerj_code_search" => return run_code_search(ctx, &args).await,// Corpus Hub v2 (plan §7): the `xerj ask` flow itself, not a proxy.hub_tool if hub_tool.starts_with("xerj_hub_") => {@@ -2507,6 +2550,68 @@ impl xerj_common::xccode::XcHttp for McpXc {}}+/// `xerj_index_budget` — the autoindex preflight, without indexing.+///+/// The walk and the plan math are `xerj_autoindex::budget`. Disk numbers come+/// from the same allocation probe autoindex uses. A node that is down still+/// returns the file counts; the disk field is then null.+async fn run_index_budget(ctx: &Ctx, args: &Value) -> Value {+ let path = match args.get("path").and_then(Value::as_str) {+ Some(p) if !p.is_empty() => p.to_string(),+ _ => return tool_text("xerj_index_budget: `path` is required", true),+ };+ let prefix = args+ .get("prefix")+ .and_then(Value::as_str)+ .filter(|s| !s.is_empty())+ .unwrap_or("ax")+ .to_string();+ let no_semantic = match args.get("no_semantic") {+ None | Some(Value::Null) => false,+ Some(Value::Bool(v)) => *v,+ Some(_) => return tool_text("xerj_index_budget: `no_semantic` must be a boolean", true),+ };+ let max_index_bytes = match args.get("max_index_bytes") {+ None | Some(Value::Null) => None,+ Some(Value::Number(n)) => match n.as_u64() {+ Some(n) if n > 0 => Some(n),+ _ => {+ return tool_text(+ "xerj_index_budget: `max_index_bytes` must be a positive integer",+ true,+ )+ }+ },+ Some(_) => {+ return tool_text(+ "xerj_index_budget: `max_index_bytes` must be a positive integer",+ true,+ )+ }+ };+ let url = ctx.base_url.clone();+ let auth = ctx.auth.clone();+ let joined = tokio::task::spawn_blocking(move || {+ xerj_autoindex::budget::assess_path(+ std::path::Path::new(&path),+ &url,+ auth.as_deref(),+ &prefix,+ max_index_bytes,+ no_semantic,+ )+ })+ .await;+ match joined {+ Ok(Ok(body)) => tool_text(+ serde_json::to_string_pretty(&body).unwrap_or_else(|_| body.to_string()),+ false,+ ),+ Ok(Err(err)) => tool_text(format!("xerj_index_budget: {err:#}"), true),+ Err(err) => tool_text(format!("xerj_index_budget: {err}"), true),+ }+}+/// `xerj_code_search` — the SAME pipeline `xerj code` runs, from the SAME/// renderer; only the surface differs. Warnings ride the TOP of the tool text/// (MCP has no stderr channel, so warnings are content), and the exit-code@@ -4280,7 +4385,7 @@ mod tests {.iter().map(|t| t["name"].as_str().unwrap()).collect();- assert_eq!(names.len(), 18);+ assert_eq!(names.len(), 19);for n in ["xerj_search","xerj_semantic_search",@@ -4294,6 +4399,7 @@ mod tests {"xerj_brain_overview",// #1055: the per-index field map from the autoindex catalog."xerj_map",+ "xerj_index_budget",// #977: reference-coding retrieval, served by the same pipeline// as `xerj code`."xerj_code_search",@@ -4370,23 +4476,23 @@ mod tests {.collect()};let without = names(false);- assert_eq!(without.len(), 15, "{without:?}");+ assert_eq!(without.len(), 16, "{without:?}");assert!(without.iter().all(|n| !n.starts_with("xerj_hub_")),"{without:?}");let with = names(true);- assert_eq!(with.len(), 18, "{with:?}");+ assert_eq!(with.len(), 19, "{with:?}");for n in ["xerj_hub_find", "xerj_hub_def", "xerj_hub_ask"] {assert!(with.iter().any(|name| name == n), "missing {n}");}let help = tools_paragraph(false);- assert!(help.contains("proxies 15 tools"), "{help}");+ assert!(help.contains("proxies 16 tools"), "{help}");assert!(!help.contains("xerj_hub_"), "{help}");assert!(help.contains("~/xerj-data"), "{help}");let help = tools_paragraph(true);- assert!(help.contains("proxies 18 tools"), "{help}");+ assert!(help.contains("proxies 19 tools"), "{help}");assert!(help.contains("xerj_hub_ask"), "{help}");assert!(help.contains("serves 3 Corpus Hub tools"), "{help}");}diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rsindex d13d51bf6..27fea7b1f 100644--- a/engine/crates/xerj-server/src/brain.rs+++ b/engine/crates/xerj-server/src/brain.rs@@ -642,6 +642,8 @@ fn index_cfg(cfg: &BrainCfg, brain: &str, api_key: Option<String>) -> IndexCfg {// grows its own answer flag. The estimate itself is still printed.max_minutes: 0,approve: None,+ max_index_bytes: None,+ budget_plan: None,json: false,quiet: false,// `xerj brain` is a foreground command a human watches; it gets thediff --git a/landing/docs/agents/schemas/mcp-tools.json b/landing/docs/agents/schemas/mcp-tools.jsonindex 30fc8e3c8..7dc3a0398 100644--- a/landing/docs/agents/schemas/mcp-tools.json+++ b/landing/docs/agents/schemas/mcp-tools.json@@ -537,6 +537,35 @@}}},+ {+ "name": "xerj_index_budget",+ "description": "Count a folder and estimate the index BEFORE xerj autoindex hashes or embeds anything. Returns file and byte counts by kind (code, prose, pdf, data, binary, other — by extension, before content sniff) and by top-level folder, the free-space budget from the node's flood-stage watermark (the same GET /_cat/allocation row autoindex uses; a node that omits disk.flood_stage is not guessed at), and one or more plans with what each drops. This tool does not index. When decision_required is true, show the plans to the user and re-run `xerj autoindex` with --plan <name> or --yes for the recommended plan that fits. Do not start a run that does not fit. Default embeddings in the estimate are the server's lexical feature-hash vectors unless that node was started with --embed-mode neural. Kinds without a measured calibration are unmeasured and do not refuse the run.",+ "inputSchema": {+ "type": "object",+ "properties": {+ "path": {+ "type": "string",+ "description": "Folder to count. Nothing inside it is hashed."+ },+ "prefix": {+ "type": "string",+ "description": "Index prefix to check for an existing copy of this folder (default ax)."+ },+ "max_index_bytes": {+ "type": "integer",+ "minimum": 1,+ "description": "Optional cap on the estimated index, in bytes. The disk budget is the free space under the flood-stage watermark; this lowers it."+ },+ "no_semantic": {+ "type": "boolean",+ "description": "Price the plans without semantic_text vectors, as --no-semantic would."+ }+ },+ "required": [+ "path"+ ]+ }+ },{"name": "xerj_code_search","description": "Reference-coding retrieval over a corpus of peer projects: how did real codebases solve this problem? Returns the matching definition (not a byte window), the file:line to cite, and per-hit licence warnings — AGPL/SSPL/Elastic/BUSL/GPL/LGPL/MPL sources are approach-only. Refuses indices older than 30 days (override with stale_ok). A no-match is NOT an error: it says the corpus is wrong for the task. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule). Corpora: `xerj corpus list`.",