AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(autoindex): real progress, honest percent and ETA — no run is silent (#241)
xerj-orge8d9f5fbf17a
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex 748159478..39ed95faf 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -1,6 +1,8 @@//! Hand-rolled arg parser (house style of xerj-server — no clap).+use crate::progress::ProgressMode;use std::path::PathBuf;+use std::time::Duration;/// Largest `--bulk-mb` accepted. Past this a single bulk body stops being a/// unit of work and starts being a memory incident on the server.@@ -37,6 +39,12 @@ pub struct IndexCfg {pub dry_run: bool,pub json: bool,pub quiet: bool,+ /// Progress surface. Orthogonal to `json`: `--json` shapes *stdout* (the+ /// result), `--progress` shapes *stderr* (liveness).+ pub progress: ProgressMode,+ /// Progress cadence. `None` means "the surface's default" — 1 s on a+ /// terminal, 5 s for a pipe.+ pub progress_interval: Option<Duration>,}#[derive(Debug, Clone)]@@ -101,8 +109,17 @@ pub fn print_help() {--no-graph skip relationship detection (wikilinks, local links,\n\section order, directory chains) — no edges are written\n\--dry-run walk+sniff+infer, print the plan, index nothing\n\- --json machine-readable output (map: raw catalog docs)\n\- --quiet errors only\n\+ --json machine-readable RESULT on stdout (map: raw catalog docs).\n\+ Orthogonal to --progress, which owns stderr.\n\+ --progress <MODE> liveness on stderr: auto|plain|json|none (default auto).\n\+ auto = live redrawn line when stderr is a terminal,\n\+ otherwise one parseable line per interval. plain and\n\+ json force that shape everywhere (CI, pipes, agents).\n\+ --progress-interval <SECS>\n\+ progress cadence, 1..=3600 (default 1 on a terminal,\n\+ 5 otherwise). This is the guaranteed upper bound on\n\+ silence between phases.\n\+ --quiet errors only (implies --progress none)\n\--dataset <SLUG> (map) show a single dataset\n\--help, -h this help\n\\n\@@ -126,6 +143,18 @@ pub fn print_help() {short/structured datasets may infer none). Use --dry-run or `autoindex map` to\n\confirm a semantic field before attributing an indexing result to embeddings.\n\\n\+ PROGRESS STREAM:\n\+ stdout is the RESULT, stderr is PROGRESS — pipe them separately.\n\+ Every run that reaches an exit — success OR error — ends with one\n\+ terminal line, in every mode, so an outcome never has to be guessed\n\+ from silence:\n\+ xerj-done ok=true exit=3 reason=completed-with-junk wall=57.6s …\n\+ (A run killed by a signal cannot print one; a missing terminal line\n\+ after the process is gone means it died, not that it finished.)\n\+ --progress plain emits `xerj-progress phase=… pct=… eta_s=…` lines;\n\+ `pct`/`eta_s` are the literal word `unknown` (JSON null) whenever they\n\+ cannot be computed honestly, never a filler number.\n\+ \n\EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\2 usage; 1 endpoint unreachable / journal-config mismatch\n");@@ -159,6 +188,8 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {let mut dry_run = false;let mut json = false;let mut quiet = false;+ let mut progress: Option<ProgressMode> = None;+ let mut progress_interval: Option<Duration> = None;let mut dataset: Option<String> = None;while let Some(arg) = it.next() {@@ -250,6 +281,25 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {"--dry-run" => dry_run = true,"--json" => json = true,"--md" => json = false,+ "--progress" => {+ let raw = it+ .next()+ .ok_or("--progress needs a value: auto, plain, json or none")?;+ progress = Some(ProgressMode::parse(&raw)?);+ }+ "--progress-interval" => {+ let secs: u64 = it+ .next()+ .ok_or("--progress-interval needs a value in seconds")?+ .parse()+ .map_err(|_| {+ "--progress-interval needs an integer in the range 1..=3600".to_string()+ })?;+ if !(1..=3_600).contains(&secs) {+ return Err("--progress-interval must be in the range 1..=3600 seconds".into());+ }+ progress_interval = Some(Duration::from_secs(secs));+ }"--quiet" => quiet = true,"--dataset" => dataset = it.next(),"--help" | "-h" => return Ok(Cmd::Help),@@ -267,7 +317,41 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {return Err("--prefix must contain at least one [a-z0-9] character".into());}+ // Contradictory progress requests are refused, never silently resolved in+ // one side's favour: accepting a flag we will not honour is the defect+ // class tracked in #204, and "I asked for progress and got none" is the+ // exact complaint this option exists to answer.+ let progress_explicit = progress.is_some() || progress_interval.is_some();+ if quiet {+ match progress {+ Some(ProgressMode::None) | None => {}+ Some(mode) => {+ return Err(format!(+ "--quiet and --progress {} contradict each other: --quiet means no progress \+ output. Drop one of the two",+ mode.as_str()+ ))+ }+ }+ }+ let progress = if quiet {+ ProgressMode::None+ } else {+ progress.unwrap_or(ProgressMode::Auto)+ };+ if progress_interval.is_some() && progress == ProgressMode::None {+ return Err(+ "--progress-interval sets the cadence of a progress stream that --progress none / \+ --quiet turns off. Drop one of the two"+ .into(),+ );+ }+match (sub.as_deref(), folder) {+ (Some("map"), _) | (Some("status"), _) if progress_explicit => Err(format!(+ "--progress/--progress-interval apply only to indexing, not `autoindex {}`",+ sub.as_deref().unwrap_or_default()+ )),(Some("map"), _) if bulk_timeout_explicit => {Err("--bulk-timeout-secs applies only to indexing, not `autoindex map`".into())}@@ -312,6 +396,8 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {dry_run,json,quiet,+ progress,+ progress_interval,}))}_ => Ok(Cmd::Help),@@ -320,7 +406,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {#[cfg(test)]mod tests {- use super::{parse, Cmd};+ use super::{parse, Cmd, Duration, ProgressMode};fn index(args: &[&str]) -> super::IndexCfg {match parse(args.iter().map(|s| s.to_string()).collect()).unwrap() {@@ -413,6 +499,73 @@ mod tests {}}+ #[test]+ fn progress_defaults_to_auto_and_quiet_turns_it_off() {+ let cfg = index(&["data"]);+ assert_eq!(cfg.progress, ProgressMode::Auto);+ assert_eq!(cfg.progress_interval, None);+ assert_eq!(index(&["data", "--quiet"]).progress, ProgressMode::None);+ assert_eq!(+ index(&["data", "--progress", "json"]).progress,+ ProgressMode::Json+ );+ // --json shapes stdout, --progress shapes stderr; they never collide.+ let both = index(&["data", "--json", "--progress", "plain"]);+ assert!(both.json);+ assert_eq!(both.progress, ProgressMode::Plain);+ assert_eq!(+ index(&["data", "--progress-interval", "30"]).progress_interval,+ Some(Duration::from_secs(30))+ );+ }++ #[test]+ fn contradictory_and_malformed_progress_requests_are_refused() {+ for args in [+ vec!["data", "--progress"],+ vec!["data", "--progress", "tty"],+ vec!["data", "--progress", "yes"],+ vec!["data", "--progress-interval"],+ vec!["data", "--progress-interval", "0"],+ vec!["data", "--progress-interval", "3601"],+ vec!["data", "--progress-interval", "soon"],+ // Accepting either of these and honouring only one half would be a+ // silent lie about what the run will print.+ vec!["data", "--quiet", "--progress", "plain"],+ vec!["data", "--progress", "none", "--progress-interval", "5"],+ vec!["data", "--quiet", "--progress-interval", "5"],+ ] {+ let rendered = args.join(" ");+ let err = match parse(args.into_iter().map(str::to_string).collect()) {+ Err(err) => err,+ Ok(other) => panic!("`{rendered}` must not be accepted, got {other:?}"),+ };+ assert!(+ err.contains("--progress"),+ "`{rendered}` must explain itself: {err}"+ );+ }+ // The one benign combination stays benign.+ assert_eq!(+ index(&["data", "--quiet", "--progress", "none"]).progress,+ ProgressMode::None+ );+ }++ #[test]+ fn progress_flags_are_rejected_for_non_index_subcommands() {+ for args in [+ vec!["map", "--progress", "json"],+ vec!["--progress", "json", "map"],+ vec!["status", "--progress-interval", "5"],+ ] {+ let err = parse(args.into_iter().map(str::to_string).collect()).unwrap_err();+ assert!(err.contains("apply only to indexing"), "{err}");+ }+ // --quiet keeps its historical no-op acceptance on map/status.+ assert!(parse(["map", "--quiet"].into_iter().map(str::to_string).collect()).is_ok());+ }+#[test]fn bulk_timeout_is_rejected_for_non_index_subcommands_in_any_position() {for args in [diff --git a/engine/crates/xerj-autoindex/src/content.rs b/engine/crates/xerj-autoindex/src/content.rsindex 9793b7c13..f934ca238 100644--- a/engine/crates/xerj-autoindex/src/content.rs+++ b/engine/crates/xerj-autoindex/src/content.rs@@ -35,9 +35,20 @@ pub(crate) static DIGEST_RAN_OFF_THE_SCAN_POOL: std::sync::atomic::AtomicBool =////// This is linear in corpus bytes for adversarial common-prefix inputs rather/// than pairwise O(n² × file_size).-pub fn resolve(files: Vec<FileEntry>) -> Result<Inventory> {+///+/// `hashed` is called with each file's byte count as it completes.+/// Hashing reads the whole corpus, so on a large tree this is minutes of work+/// that reported nothing before #241. The callback runs on the scan pool's+/// workers and must therefore be `Sync`; it does no I/O of its own — it only+/// bumps the progress counters that the ticker thread reads.+pub fn resolve_reporting(+ files: Vec<FileEntry>,+ hashed: &(dyn Fn(u64) + Sync),+) -> Result<Inventory> {// Phase A belongs to the run's scan pool, not to rayon's global pool:// `--workers` has to bound the CPU-bound phase to mean anything (#240 §2).+ // The progress callback therefore fires from scan-pool threads, which is+ // also the pool whose width the ETA is computed against (#241).let digests: Vec<Result<String>> = crate::pool::install(|| {files.par_iter()@@ -56,7 +67,9 @@ pub fn resolve(files: Vec<FileEntry>) -> Result<Inventory> {.store(true, std::sync::atomic::Ordering::Relaxed);}}- full_digest(&file.path, file.size)+ let digest = full_digest(&file.path, file.size);+ hashed(file.size);+ digest}).collect()});@@ -214,11 +227,20 @@ mod tests {use super::*;use std::fs;+ /// These tests are about identity resolution, not progress reporting.+ fn resolve(files: Vec<FileEntry>) -> Result<Inventory> {+ resolve_reporting(files, &|_| {})+ }+/// #240 §2: the digest phase ran on rayon's default global pool — every/// core on the machine — so `--workers` never bounded it. It must run on/// the run's own scan pool, whose width the CLI sets.+ ///+ /// #241 rides on the same closure: moving phase A into a private pool must+ /// not cost the progress stream a single completion, or the percent the+ /// ticker prints would stall below 100 for the rest of the phase.#[test]- fn the_digest_phase_runs_on_the_runs_scan_pool() {+ fn the_digest_phase_runs_on_the_runs_scan_pool_and_still_reports_every_file() {let dir = tempfile::tempdir().unwrap();for i in 0..32 {fs::write(dir.path().join(format!("f{i}.txt")), format!("body {i}\n")).unwrap();@@ -226,7 +248,16 @@ mod tests {OBSERVED_DIGEST_POOL_THREADS.store(0, std::sync::atomic::Ordering::Relaxed);DIGEST_RAN_OFF_THE_SCAN_POOL.store(false, std::sync::atomic::Ordering::Relaxed);- resolve(crate::walk::walk(dir.path(), false).unwrap()).unwrap();+ let files = crate::walk::walk(dir.path(), false).unwrap();+ let expected_files = files.len() as u64;+ let expected_bytes: u64 = files.iter().map(|f| f.size).sum();+ let reported_files = std::sync::atomic::AtomicU64::new(0);+ let reported_bytes = std::sync::atomic::AtomicU64::new(0);+ resolve_reporting(files, &|bytes| {+ reported_files.fetch_add(1, std::sync::atomic::Ordering::Relaxed);+ reported_bytes.fetch_add(bytes, std::sync::atomic::Ordering::Relaxed);+ })+ .unwrap();assert!(!DIGEST_RAN_OFF_THE_SCAN_POOL.load(std::sync::atomic::Ordering::Relaxed),@@ -237,6 +268,16 @@ mod tests {crate::pool::scan_pool().current_num_threads(),"the digest phase must see exactly the scan pool the run configured");+ assert_eq!(+ reported_files.load(std::sync::atomic::Ordering::Relaxed),+ expected_files,+ "every hashed file must still be credited to the progress stream"+ );+ assert_eq!(+ reported_bytes.load(std::sync::atomic::Ordering::Relaxed),+ expected_bytes,+ "the bytes-based percent must see the whole corpus"+ );}#[test]diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rsindex 72c9446ae..8a228f0de 100644--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs@@ -275,6 +275,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {dry_run: false,json: false,quiet: true,+ progress: crate::progress::ProgressMode::None,+ progress_interval: None,}}diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rsindex fe0c6a816..401998ee6 100644--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs@@ -276,6 +276,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {dry_run: false,json: false,quiet: true,+ progress: crate::progress::ProgressMode::None,+ progress_interval: None,}}@@ -1100,3 +1102,112 @@ fn zero_live_documents_after_journaled_records_fails_the_run() {assert!(error.contains("0 documents are live"), "{error}");assert!(error.contains("--fresh"), "{error}");}++/// #241 regression: a run must never be silent.+///+/// Before this fix `autoindex` printed a handful of phase banners, one line+/// per 200 completed files, and then nothing at all through the entire+/// `finalize` block — measured at 47-64% of wall in the issue and at 45% (100+/// of 222 s) on the corpus used to reproduce it here. An agent watching the+/// stream could not tell a running index from a hung one.+///+/// This asserts the three properties that make the stream readable, through+/// the REAL progress surface a production run uses:+/// 1. phase B reports as it goes, with a percent and an ETA field;+/// 2. the previously-silent finalize block reports at all;+/// 3. every run ends with a terminal line that states the outcome in words,+/// because exit 3 is a success an agent otherwise reads as failure.+#[test]+fn a_run_reports_progress_through_every_phase_and_closes_the_stream() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _sink_guard = crate::progress::SINK_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ // Distinct bytes per file: byte-identical files collapse into one+ // canonical file plus aliases, which would make `files=8` a lie.+ for n in 0..8 {+ fs::write(+ corpus.path().join(format!("rows-{n}.csv")),+ format!("id,value\n{n}0,alpha\n{n}1,beta\n{n}2,gamma\n"),+ )+ .unwrap();+ }++ let endpoint = MockEndpoint::start(usize::MAX);+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ config.quiet = false;+ config.progress = crate::progress::ProgressMode::Plain;+ config.progress_interval = Some(std::time::Duration::from_secs(1));++ let buffer = Arc::new(Mutex::new(Vec::new()));+ let (code, _report) = {+ let _sink = crate::progress::install_test_sink(&buffer);+ run_index_report(config).unwrap()+ };+ let stream = String::from_utf8(buffer.lock().unwrap().clone()).unwrap();++ let progress_lines: Vec<&str> = stream+ .lines()+ .filter(|line| line.starts_with("xerj-progress "))+ .collect();+ assert!(+ progress_lines.len() >= 5,+ "a run must narrate its phases, got {} line(s):\n{stream}",+ progress_lines.len()+ );+ assert!(+ progress_lines+ .iter()+ .any(|line| line.contains("phase=index")),+ "phase B must report:\n{stream}"+ );+ assert!(+ progress_lines+ .iter()+ .any(|line| line.contains("phase=finalize")),+ "the finalize block was the longest silence in the tool; it must report now:\n{stream}"+ );+ assert!(+ progress_lines+ .iter()+ .all(|line| line.contains(" pct=") && line.contains(" eta_s=")),+ "every progress line carries a percent and an ETA field (possibly `unknown`):\n{stream}"+ );++ let done = stream+ .lines()+ .find(|line| line.starts_with("xerj-done "))+ .unwrap_or_else(|| panic!("a run must close its own stream:\n{stream}"));+ assert!(+ done.contains("ok=true") && done.contains(&format!("exit={code}")),+ "{done}"+ );+ assert!(done.contains("reason=completed"), "{done}");+ assert!(done.contains("files=8"), "{done}");+}++/// The other half of the contract: `--quiet` still means silence, so nothing+/// here can spam a caller that asked for none.+#[test]+fn quiet_runs_emit_no_progress_stream_at_all() {+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();+ let _sink_guard = crate::progress::SINK_TEST_LOCK.lock().unwrap();+ let corpus = tempfile::tempdir().unwrap();+ let state_dir = tempfile::tempdir().unwrap();+ fs::write(corpus.path().join("rows.csv"), "id,value\n0,alpha\n").unwrap();++ let endpoint = MockEndpoint::start(usize::MAX);+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);+ assert_eq!(config.progress, crate::progress::ProgressMode::None);++ let buffer = Arc::new(Mutex::new(Vec::new()));+ {+ let _sink = crate::progress::install_test_sink(&buffer);+ run_index_report(config).unwrap();+ }+ assert!(+ buffer.lock().unwrap().is_empty(),+ "--quiet asked for nothing: {}",+ String::from_utf8_lossy(&buffer.lock().unwrap())+ );+}diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex e4d255704..1c3316034 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -15,6 +15,7 @@ pub mod extract;pub mod ids;pub mod infer;pub mod pool;+pub mod progress;pub mod resources;pub mod sniff;pub mod state;@@ -34,6 +35,7 @@ use cli::{Cmd, IndexCfg, MapCfg, StatusCfg};// `Box<dyn EdgeDetector>` like the registry entries).use detect::EdgeDetector as _;use esclient::Es;+use progress::Progress;use sniff::{Family, Sniffed};use state::{DuplicateFile, FileAssignment, FileDone, JunkFile, Plan, PlanDataset};@@ -411,6 +413,8 @@ mod phase_a_grouping_tests {dry_run: true,json: false,quiet: true,+ progress: crate::progress::ProgressMode::None,+ progress_interval: None,}}@@ -421,7 +425,15 @@ mod phase_a_grouping_tests {.map(|f| ids::file_key(&f.path, f.size).unwrap()).collect();let digests: Vec<String> = (0..files.len()).map(|i| format!("d{i}")).collect();- let (plan, _) = build_phase_a(root, &files, &keys, &digests, Vec::new(), &cfg_for(root));+ let (plan, _) = build_phase_a(+ root,+ &files,+ &keys,+ &digests,+ Vec::new(),+ &cfg_for(root),+ &Progress::silent(),+ );plan}@@ -593,14 +605,20 @@ fn build_phase_a(digests: &[String],duplicate_files: Vec<DuplicateFile>,cfg: &IndexCfg,+ pr: &Progress,) -> (Plan, Vec<dataset::Cluster>) {use rayon::prelude::*;// Same pool as the digest phase: sniffing and sampling are the other half- // of the CPU-bound phase `--workers` has to bound (#240 §2).+ // of the CPU-bound phase `--workers` has to bound (#240 §2). Progress is+ // reported from inside that pool, so the straggler the ticker names is the+ // file a scan-pool thread is genuinely sitting on (#241).let scans: Vec<FileScan> = crate::pool::install(|| {files.par_iter()- .map(|f| scan_file(&f.path, f.size, cfg.sample, cfg.max_file_gb))+ .map(|f| {+ let _in_flight = pr.file(&f.rel, f.size);+ scan_file(&f.path, f.size, cfg.sample, cfg.max_file_gb)+ }).collect()});@@ -836,7 +854,8 @@ fn select_resume_plan_keys(// replacement transaction on one ax_file key and delete each// other's freshly published documents. Divert this file to a// deterministic path-derived key, the same discriminator scheme- // content::resolve uses for byte-proven digest collisions.+ // content::resolve_reporting uses for byte-proven digest+ // collisions.Some(format!("{content_key}-claimed-{:032x}",xxhash_rust::xxh3::xxh3_128(file.rel_id.as_bytes())@@ -917,55 +936,76 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// sniffing are the CPU-bound phase, and they used to take every core no// matter what the caller asked for (#240 §2).pool::configure(cfg.scan_workers);+ // What phase A is *actually* running on. `pool::configure` is first-call-wins+ // (rayon pools cannot be resized), so in a process that already indexed once+ // — `xerj-server`'s brain endpoint does exactly that — the installed width+ // can differ from what this run's plan asked for. Progress must state the+ // number the policy got, not the one it requested (#240 + #241).+ let scan_threads = pool::scan_pool().current_num_threads();extract::pdf::configure_workers(cfg.pdf_workers);extract::pdf::configure_timeout(cfg.pdf_timeout_secs);let t0 = Instant::now();- if !cfg.quiet {- // What this run decided to take from the machine, before it takes it.- eprintln!(- "autoindex: {} scan threads, {} index workers, {} pdf workers, --bulk-mb {} \- [{}]",- cfg.scan_workers,- cfg.workers,- cfg.pdf_workers,- cfg.bulk_mb,- xerj_common::resource::describe(),- );- for note in &cfg.resource_notes {- eprintln!("autoindex: {note}");- }- eprintln!(- "autoindex: bulk HTTP request timeout: {}s",- cfg.bulk_timeout_secs- );- }+ // The progress surface and its ticker are the FIRST things built: every+ // later phase reports through them, and the ticker guarantees the stream+ // closes with a terminal line even if this function bails (#241). The+ // resource plan is announced through the same surface rather than through a+ // bare `eprintln!`, so `--quiet` / `--progress none` still means silent and+ // `--progress json` still means one machine-readable stream (#240 + #241).+ let surface = progress::detect(cfg.progress);+ let pr = Progress::new(+ surface,+ cfg.progress_interval+ .unwrap_or_else(|| progress::default_interval(surface)),+ );+ let ticker = pr.spawn_ticker();+ // What this run decided to take from the machine, before it takes it.+ pr.note(&format!(+ "autoindex: {} scan threads, {} index workers, {} pdf workers, --bulk-mb {} [{}]",+ scan_threads,+ cfg.workers,+ cfg.pdf_workers,+ cfg.bulk_mb,+ xerj_common::resource::describe(),+ ));+ for note in &cfg.resource_notes {+ pr.note(&format!("autoindex: {note}"));+ }+ pr.note(&format!(+ "autoindex: bulk HTTP request timeout: {}s",+ cfg.bulk_timeout_secs+ ));// The run's bulk load is admitted through one window `--workers` wide, so// a 429 can shrink what the run offers instead of only delaying it// (#240 §8). Enabled here and nowhere else: probes have nothing to- // throttle.+ // throttle. Its shrink/recover announcements go to stderr, which the+ // progress surface owns — so they are emitted exactly when that surface is+ // enabled, not merely when `--quiet` is absent.let es = Es::with_bulk_timeout(&cfg.url, cfg.api_key.clone(), cfg.bulk_timeout_secs)?- .with_bulk_concurrency(cfg.workers, !cfg.quiet);+ .with_bulk_concurrency(cfg.workers, pr.enabled());es.ping()?;+ // Totals are unknown until the walk returns, so this phase honestly+ // reports `pct=unknown` and proves liveness with the clock alone.+ pr.phase("walk", 0, 0);let discovered_files = walk::walk(&cfg.root, cfg.follow_symlinks)?;if discovered_files.is_empty() {println!("no files found under {}", cfg.root.display());+ pr.finish(true, 0, "no-files", &[]);return Ok((0, None));}+ let discovered_bytes: u64 = discovered_files.iter().map(|f| f.size).sum();let root_str = cfg.root.canonicalize().unwrap_or_else(|_| cfg.root.clone()).to_string_lossy().to_string();- if !cfg.quiet {- eprintln!(- "autoindex: {} files ({} MB) under {}",- discovered_files.len(),- discovered_files.iter().map(|f| f.size).sum::<u64>() / (1 << 20),- root_str- );- }+ pr.note(&format!(+ "autoindex: {} files ({} MB) under {}",+ discovered_files.len(),+ discovered_bytes / (1 << 20),+ root_str+ ));let state_dir = cfg.state_dir@@ -981,18 +1021,21 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {)?;let resumed_with_plan = journal.plan.is_some();let run_id = journal.run_id.clone();- if journal.resumed && !cfg.quiet {- eprintln!(+ if journal.resumed {+ pr.note(&format!("resuming from journal {} ({} files already done)",journal.path().display(),journal.done.len()- );+ ));}// Full hashing on every run is deliberate: size/mtime/inode fingerprints// cannot prove byte identity across all supported local and network// filesystems. A metadata-only shortcut could leave stale live documents// forever after a same-size rewrite with restored or stale timestamps.- let mut inventory = content::resolve(discovered_files)?;+ // Hashing reads every byte of the corpus. On a large tree it is minutes of+ // real work, and before #241 it was minutes with no output at all.+ pr.phase("hash", discovered_files.len() as u64, discovered_bytes);+ let mut inventory = content::resolve_reporting(discovered_files, &|bytes| pr.item_done(bytes))?;let journal_path = journal.path().to_path_buf();let mut content_changed = std::collections::HashSet::new();let mut stale_alias_ids = Vec::new();@@ -1121,19 +1164,20 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {let digests = inventory.digests;let duplicate_files = inventory.duplicates;let paths_discovered = files.len() + duplicate_files.len();- if !duplicate_files.is_empty() && !cfg.quiet {- eprintln!(+ let planned_bytes: u64 = files.iter().map(|f| f.size).sum();+ if !duplicate_files.is_empty() {+ pr.note(&format!("autoindex: {} byte-identical duplicate path(s) will reuse canonical content",duplicate_files.len()- );+ ));for duplicate in duplicate_files.iter().take(10) {- eprintln!(+ pr.note(&format!(" duplicate: {} → {}",duplicate.rel, duplicate.duplicate_of- );+ ));}if duplicate_files.len() > 10 {- eprintln!(" … and {} more", duplicate_files.len() - 10);+ pr.note(&format!(" … and {} more", duplicate_files.len() - 10));}}@@ -1142,9 +1186,14 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {let mut plan: Plan = if let Some(p) = journal.plan.clone() {p} else {- if !cfg.quiet {- eprintln!("phase A: sniffing + sampling {} files…", files.len());- }+ pr.phase("scan", files.len() as u64, planned_bytes);+ // Named with the width phase A is really running at: `--workers` only+ // started governing this phase in #240, so "how wide is it right now"+ // is exactly the question the reader has.+ pr.note(&format!(+ "phase A: sniffing + sampling {} files with {scan_threads} threads…",+ files.len()+ ));let (plan, clusters) = build_phase_a(&cfg.root,&files,@@ -1152,14 +1201,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {&digests,duplicate_files.clone(),&cfg,+ &pr,);- if !cfg.quiet {- eprintln!(- "phase A: {} datasets inferred, {} junk/skipped files",- plan.datasets.len(),- plan.junk_files.len()- );- }+ pr.note(&format!(+ "phase A: {} datasets inferred, {} junk/skipped files",+ plan.datasets.len(),+ plan.junk_files.len()+ ));clusters_rt = Some(clusters);plan};@@ -1197,7 +1245,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {if cfg.dry_run {println!("{}", serde_json::to_string_pretty(&plan)?);- eprintln!("(dry run — nothing indexed)");+ pr.note("(dry run — nothing indexed)");+ pr.finish(true, 0, "dry-run", &[("files", files.len() as u64)]);return Ok((0, None));}@@ -1217,6 +1266,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {}// ── create indices with explicit mappings ────────────────────────────+ // Two round trips per dataset; a 135-dataset plan is a real wait.+ pr.phase("prepare", plan.datasets.len() as u64, 0);for d in &plan.datasets {es.ensure_index(&d.index, &build_mapping(&d.specs)).with_context(|| format!("create index {}", d.index))?;@@ -1225,6 +1276,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {&json!({"properties": {"ax_paths": {"type": "keyword"}}}),).with_context(|| format!("upgrade alias-path mapping for {}", d.index))?;+ pr.item_done(0);}es.ensure_index(catalog::CATALOG_INDEX, &catalog::catalog_mapping())?;es.update_mapping(@@ -1365,6 +1417,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {let graph: Option<GraphRt> = if cfg.no_graph {None} else {+ pr.phase("graph", files.len() as u64, 0);let brain = match &cfg.brain {Some(b) => b.clone(),None => derive_brain_name(&cfg.root),@@ -1404,6 +1457,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {corpus_files.push(detect::corpus_file(&f.rel, key, &slug, &fa.family, mtime_ms,));+ pr.item_done(0);}let corpus = detect::CorpusIndex::build(corpus_files);let detectors = detect::default_detectors();@@ -1476,15 +1530,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {anyhow::bail!("write structural graph edges to {edges_index}: {e}");}}- if !cfg.quiet {- eprintln!(- "graph: brain '{brain}' → {edges_index}; {} structural edges, {} prior edges \- invalidated ({} detectors live)",- assembled.edges.len(),- invalidated,- detectors.len()- );- }+ pr.note(&format!(+ "graph: brain '{brain}' → {edges_index}; {} structural edges, {} prior edges \+ invalidated ({} detectors live)",+ assembled.edges.len(),+ invalidated,+ detectors.len()+ ));Some(GraphRt {corpus,detectors,@@ -1502,12 +1554,15 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// start first and can't serialize the end of the run.todo.sort_by_key(|&i| files[i].size);let n_todo = todo.len();- if !cfg.quiet {- eprintln!(- "phase B: indexing {} files with {} workers → {}",- n_todo, cfg.workers, cfg.url- );- }+ // Percent and ETA are bytes-based, never file-count-based. The queue is+ // biggest-first, so a files-done percent races to ~100% and then sits+ // there for minutes on the one big file still in flight (#241 §5/§6).+ let todo_bytes: u64 = todo.iter().map(|&i| files[i].size).sum();+ pr.phase("index", n_todo as u64, todo_bytes);+ pr.note(&format!(+ "phase B: indexing {} files with {} workers → {}",+ n_todo, cfg.workers, cfg.url+ ));let queue = Mutex::new(todo);let mut paths_by_key: HashMap<String, Vec<String>> = files@@ -1539,6 +1594,11 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {None => break,};let f = &files[i];+ // Counts this file done on EVERY exit path below, junk+ // included: progress measures work drained from the queue,+ // and a `continue` that skipped the count would park the+ // bar short of 100% forever.+ let _in_flight = pr.file(&f.rel, f.size);let key = &keys[i];let expected_digest = &digests[i];let fa = plan.files.get(key).unwrap();@@ -1975,10 +2035,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {continue;}}- let dn = files_done.fetch_add(1, Ordering::Relaxed) + 1;- if !cfg.quiet && (dn.is_multiple_of(200) || f.size > 5 * (1 << 20)) {- eprintln!(" [{dn}/{n_todo}] {} ({} records)", f.rel, file_records);- }+ // Workers no longer print. A worker can block for minutes+ // inside one file, so a worker-driven heartbeat cannot+ // bound silence; the ticker thread can, and does.+ files_done.fetch_add(1, Ordering::Relaxed);}});}@@ -1993,9 +2053,11 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// and edges over a half-read corpus would be edges over a lie.if let Some(gr) = &graph {if bulk_errors.lock().unwrap().is_empty() {+ pr.phase("graph-corpus", gr.detectors.len() as u64, 0);let mut drafts = Vec::new();for det in &gr.detectors {det.detect_corpus(&gr.corpus, &mut drafts);+ pr.item_done(0);}if !drafts.is_empty() {let out = detect::assemble(&drafts, &gr.edges_index, gr.created_at_ms);@@ -2047,16 +2109,27 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {}// ── finalize: refresh, verify, correlate, catalog ────────────────────+ //+ // This block was 47-64% of every measured run and emitted NOTHING between+ // the last phase-B line and the final summary (#241 §1). Its work is fully+ // countable before each loop starts, so it is reported like any other+ // phase — split into named sub-phases because the mix of one-shot refreshes+ // and per-dataset round trips has no single honest denominator.+ pr.phase("finalize-refresh", 1 + graph.is_some() as u64, 0);es.refresh(&format!("{}-*", cfg.prefix)).ok();+ pr.item_done(0);// The dot-prefixed edges index is outside the {prefix}-* pattern.if let Some(gr) = &graph {es.refresh(&gr.edges_index).ok();+ pr.item_done(0);}// live per-dataset counts + time ranges (every claim traces to a run)+ pr.phase("finalize-count", plan.datasets.len() as u64, 0);let mut ds_counts: HashMap<String, u64> = HashMap::new();let mut ds_timerange: HashMap<String, (Option<String>, Option<String>)> = HashMap::new();for d in &plan.datasets {+ let _counted = pr.file(&d.index, 0);let cnt = es.count(&d.index).unwrap_or(0);ds_counts.insert(d.slug.clone(), cnt);if let Some(t) = &d.time_field {@@ -2140,18 +2213,33 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {}}key_corrs = correlate::key_overlaps(&cands);+ // One live query per candidate overlap, and the bound is known before+ // the loop starts — `phase=finalize-correlate items=37/135` needs no+ // new bookkeeping at all (#241 §8).+ pr.phase("finalize-correlate", key_corrs.len() as u64, 0);for c in key_corrs.iter_mut() {+ let _query = pr.file(+ &format!("{}.{} ~ {}.{}", c.a_slug, c.a_field, c.b_slug, c.b_field),+ 0,+ );correlate::confirm(&es, c, 20).ok();}// keep only live-confirmed overlaps in the reportkey_corrs.retain(|c| c.confirmed.map(|(n, _)| n > 0).unwrap_or(false));- } else if !cfg.quiet {- eprintln!("(resumed run: key-overlap correlations kept from the original run's catalog)");+ } else {+ pr.note("(resumed run: key-overlap correlations kept from the original run's catalog)");}+ let timed: Vec<&PlanDataset> = plan+ .datasets+ .iter()+ .filter(|d| d.time_field.is_some())+ .collect();+ pr.phase("finalize-histogram", timed.len() as u64, 0);let mut series = Vec::new();- for d in &plan.datasets {+ for d in timed {if let Some(t) = &d.time_field {+ let _query = pr.file(&d.index, 0);if let Ok(Some(s)) = correlate::fetch_histogram(&es, &d.slug, &d.index, t) {series.push(s);}@@ -2163,7 +2251,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {// Alias IDs changed as identity evolved. Remove by logical path first so// catalogs created by any previous identity scheme cannot survive beside// the one current alias document.+ pr.phase(+ "finalize-catalog",+ alias_paths_to_replace.len() as u64 + 1,+ 0,+ );for path in &alias_paths_to_replace {+ let _delete = pr.file(path, 0);es.delete_by_query(catalog::CATALOG_INDEX,&json!({@@ -2405,7 +2499,11 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {"workers": cfg.workers,// The whole resource decision, so a run can be explained after the// fact from its own summary rather than from the machine it ran on.- "scan_workers": cfg.scan_workers,+ // `scan_workers` is the width phase A actually ran at — see the+ // first-call-wins note at the top of this function; a requested width+ // that lost that race would make the summary a record of an intention+ // rather than of a run.+ "scan_workers": scan_threads,"pdf_workers": cfg.pdf_workers,"bulk_mb": cfg.bulk_mb,"cores_available": xerj_common::resource::cores(),@@ -2445,6 +2543,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {}}es.refresh(catalog::CATALOG_INDEX).ok();+ pr.item_done(0);// ── durable junk record, written last (#238) ─────────────────────────//// Only now is the plan allowed to claim what the catalog holds: the@@ -2523,6 +2622,25 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {} else {0};+ // Terminal line, in every mode. Exit 3 means "completed, some files were+ // unparseable" — success — and an agent reading a bare `3` off a silent+ // stream reads failure (#241 §9). Say it in words.+ pr.finish(+ true,+ code,+ if code == 3 {+ "completed-with-junk"+ } else {+ "completed"+ },+ &[+ ("files", files_done.load(Ordering::Relaxed)),+ ("records", total_records),+ ("datasets", plan.datasets.len() as u64),+ ("junk_files", junk_file_count as u64),+ ],+ );+ drop(ticker);Ok((code, Some(run_doc)))}@@ -2820,7 +2938,7 @@ mod duplicate_integration_tests {fs::write(corpus.path().join("a.txt"), a).unwrap();fs::write(corpus.path().join("b.txt"), b).unwrap();let files = walk::walk(corpus.path(), false).unwrap();- let inventory = content::resolve(files.clone()).unwrap();+ let inventory = content::resolve_reporting(files.clone(), &|_| {}).unwrap();let legacy = ids::file_key(&files[0].path, files[0].size).unwrap();assert_eq!(legacy,@@ -2856,7 +2974,8 @@ mod duplicate_integration_tests {// with — so b.txt's content key IS the planned key a.txt claims by rel.fs::write(corpus.path().join("a.txt"), b"rewritten content\n").unwrap();fs::write(corpus.path().join("b.txt"), b"original planned content\n").unwrap();- let inventory = content::resolve(walk::walk(corpus.path(), false).unwrap()).unwrap();+ let inventory =+ content::resolve_reporting(walk::walk(corpus.path(), false).unwrap(), &|_| {}).unwrap();let planned_key = inventory.keys[1].clone();let mut plan = Plan::default();plan.files@@ -2927,7 +3046,7 @@ mod duplicate_integration_tests {fs::write(corpus.path().join("report-copy.txt"), body).unwrap();let discovered = walk::walk(corpus.path(), false).unwrap();- let inventory = content::resolve(discovered).unwrap();+ let inventory = content::resolve_reporting(discovered, &|_| {}).unwrap();assert_eq!(inventory.files.len(), 1);assert_eq!(inventory.duplicates.len(), 1);@@ -2999,7 +3118,8 @@ mod duplicate_integration_tests {}fs::write(&path, csv).unwrap();- let inventory = content::resolve(walk::walk(corpus.path(), false).unwrap()).unwrap();+ let inventory =+ content::resolve_reporting(walk::walk(corpus.path(), false).unwrap(), &|_| {}).unwrap();let expected_size = inventory.files[0].size;let expected_digest = inventory.digests[0].clone();let sniffed = sniff::sniff(&path).unwrap();diff --git a/engine/crates/xerj-autoindex/src/progress.rs b/engine/crates/xerj-autoindex/src/progress.rsnew file mode 100644index 000000000..bbec0c515--- /dev/null+++ b/engine/crates/xerj-autoindex/src/progress.rs@@ -0,0 +1,1208 @@+//! Progress, percent and ETA for `xerj autoindex` (#241).+//!+//! The contract this module implements:+//!+//! * **stdout is the RESULT, stderr is PROGRESS.** `--json` keeps stdout a+//! single parseable document; liveness never contaminates it.+//! * **One ticker thread owns the progress surface.** Workers only bump+//! atomics. A worker can block for many seconds inside one large file, so a+//! worker-driven heartbeat cannot bound silence — a timer can.+//! * **Liveness is time-based, never item-based.** `--progress-interval`+//! seconds is the guaranteed upper bound on silence between the first phase+//! and the terminal line.+//! * **Nothing is displayed that cannot be computed honestly.** A percent is+//! emitted only when a real denominator exists, and an ETA only after the+//! quality gate below is satisfied; otherwise the field is literally+//! `unknown` (`null` in JSON) rather than a comforting number.+//!+//! ETA is derived from **bytes**, not from file count. File count is a badly+//! skewed proxy for work — in the corpus that produced #241 the largest single+//! file held 40.4% of all bytes, so a files-done ETA collapses to ~0 while+//! minutes of work remain.+//!+//! Prior art, consulted and adapted rather than copied (both Apache-2.0/MIT and+//! cited per the repo's reference-coding rule):+//!+//! * `quickwit-common/src/progress.rs:78` `record_progress()`, `:90`+//! `protect_zone()`, `:117` `registered_activity_since_last_call()` — a+//! supervisor polls an atomic, and work that will block for a long time+//! declares a *protected zone* so the supervisor does not judge it hung. We+//! need the same property but cannot ask each call site to declare a zone, so+//! the ticker here runs unconditionally and [`Progress::file`] names what is+//! being waited on — the in-flight table is this design's answer to the+//! question a protected zone answers ("it is not hung, it is doing *this*").+//! * `meilisearch/crates/milli/src/progress.rs:21` `Progress`, `:137` the+//! nested overall percentage, `:331` `ProgressStepView { current_step,+//! finished, total }` — the shape of an honest cross-phase percentage. We+//! keep the per-phase denominator explicit instead of composing one global+//! number, because autoindex's phases have no common unit.++use std::io::{IsTerminal, Write};+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};+use std::sync::{Arc, Condvar, Mutex};+use std::time::{Duration, Instant};++/// Smoothing factor on the per-tick throughput sample.+const EWMA_ALPHA: f64 = 0.2;+/// No ETA until the phase has run at least this long.+const ETA_MIN_PHASE_SECS: f64 = 5.0;+/// No ETA until this fraction of the phase is complete.+const ETA_MIN_FRACTION: f64 = 0.02;+/// Below this fraction a computed ETA is labelled `rough`, not `good`.+const ETA_GOOD_FRACTION: f64 = 0.10;+/// A displayed ETA may not move by more than this fraction per tick.+const ETA_MAX_STEP: f64 = 0.20;+/// Name at most this many still-running files when the tail goes quiet.+const STRAGGLER_MAX: usize = 3;+/// Fallback terminal width when `COLUMNS` is unset.+const DEFAULT_COLUMNS: usize = 100;++/// Live-redraw cadence on a terminal.+pub const TTY_INTERVAL: Duration = Duration::from_secs(1);+/// Line cadence for pipes, agents and CI.+pub const STREAM_INTERVAL: Duration = Duration::from_secs(5);++/// `--progress` as the user wrote it.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum ProgressMode {+ /// Terminal → live line; anything else → `plain`.+ Auto,+ /// One periodic key=value line, always, whatever stderr is.+ Plain,+ /// One periodic JSON object per line.+ Json,+ /// Emit nothing (what `--quiet` selects).+ None,+}++impl ProgressMode {+ /// Parse the flag value. An unrecognised mode is an error, never a silent+ /// fallback to `auto`: accepting a value we will not honour is exactly the+ /// class of defect tracked in #204.+ pub fn parse(raw: &str) -> Result<Self, String> {+ match raw {+ "auto" => Ok(Self::Auto),+ "plain" => Ok(Self::Plain),+ "json" => Ok(Self::Json),+ "none" => Ok(Self::None),+ other => Err(format!(+ "--progress {other}: expected one of auto, plain, json, none"+ )),+ }+ }++ pub fn as_str(self) -> &'static str {+ match self {+ Self::Auto => "auto",+ Self::Plain => "plain",+ Self::Json => "json",+ Self::None => "none",+ }+ }+}++/// What `auto` actually resolved to.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum Surface {+ /// In-place redraw with `\r`, for a human at a terminal.+ Tty,+ /// One `xerj-progress key=value …` line per tick.+ Plain,+ /// One JSON object per tick.+ Json,+ /// Nothing at all.+ Silent,+}++/// Resolve a mode against the environment. Pure so it can be tested without a+/// terminal: `auto` is a terminal surface only when stderr is a tty, `TERM` is+/// not `dumb`, and `CI` is unset.+pub fn resolve(mode: ProgressMode, stderr_tty: bool, term: Option<&str>, ci: bool) -> Surface {+ match mode {+ ProgressMode::None => Surface::Silent,+ ProgressMode::Plain => Surface::Plain,+ ProgressMode::Json => Surface::Json,+ ProgressMode::Auto => {+ if stderr_tty && term != Some("dumb") && !ci {+ Surface::Tty+ } else {+ Surface::Plain+ }+ }+ }+}++/// `resolve` against this process's real environment.+pub fn detect(mode: ProgressMode) -> Surface {+ let term = std::env::var("TERM").ok();+ let ci = std::env::var_os("CI").is_some_and(|value| !value.is_empty());+ resolve(mode, std::io::stderr().is_terminal(), term.as_deref(), ci)+}++/// Default cadence for a surface: a human wants a second, a log wants five.+pub fn default_interval(surface: Surface) -> Duration {+ match surface {+ Surface::Tty => TTY_INTERVAL,+ _ => STREAM_INTERVAL,+ }+}++/// Serialises the tests that redirect the process-wide progress surface.+#[cfg(test)]+pub static SINK_TEST_LOCK: Mutex<()> = Mutex::new(());++#[cfg(test)]+static TEST_SINK: Mutex<Option<Arc<Mutex<Vec<u8>>>>> = Mutex::new(None);++/// Redirect every `Progress::new` in this process into `buffer` until the+/// returned guard drops. Hold `SINK_TEST_LOCK` for the whole window.+#[cfg(test)]+pub fn install_test_sink(buffer: &Arc<Mutex<Vec<u8>>>) -> TestSinkGuard {+ *TEST_SINK.lock().unwrap() = Some(Arc::clone(buffer));+ TestSinkGuard+}++#[cfg(test)]+pub struct TestSinkGuard;++#[cfg(test)]+impl Drop for TestSinkGuard {+ fn drop(&mut self) {+ *TEST_SINK.lock().unwrap() = None;+ }+}++enum Sink {+ Stderr,+ /// Test-only: the identical render path, captured in memory.+ #[cfg(test)]+ Buffer(Arc<Mutex<Vec<u8>>>),+}++impl Sink {+ fn write(&self, bytes: &[u8]) {+ match self {+ Self::Stderr => {+ let stderr = std::io::stderr();+ let mut handle = stderr.lock();+ let _ = handle.write_all(bytes);+ let _ = handle.flush();+ }+ #[cfg(test)]+ Self::Buffer(buffer) => buffer.lock().unwrap().extend_from_slice(bytes),+ }+ }+}++#[derive(Debug, Clone, Copy, PartialEq)]+enum Eta {+ Unknown,+ /// Work is still running but nothing has completed for a long time, so+ /// there is no current throughput measurement to extrapolate from.+ Stalled,+ Rough(f64),+ Good(f64),+}++impl Eta {+ fn seconds(self) -> Option<f64> {+ match self {+ Self::Unknown | Self::Stalled => None,+ Self::Rough(s) | Self::Good(s) => Some(s),+ }+ }++ fn quality(self) -> &'static str {+ match self {+ Self::Unknown => "unknown",+ Self::Stalled => "stalled",+ Self::Rough(_) => "rough",+ Self::Good(_) => "good",+ }+ }+}++/// Whether the percent/ETA denominator is bytes of work or a count of items.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum Basis {+ Bytes,+ Items,+ None,+}++impl Basis {+ fn as_str(self) -> &'static str {+ match self {+ Self::Bytes => "bytes",+ Self::Items => "items",+ Self::None => "none",+ }+ }+}++struct State {+ phase: &'static str,+ items_total: u64,+ bytes_total: u64,+ phase_started: Instant,+ last_sample_at: Instant,+ last_sample_units: u64,+ /// When a unit last actually completed. Progress inside a single large+ /// file is not observable from here, so this is the only honest signal+ /// that the throughput estimate is still current.+ last_advance_at: Instant,+ rate: Option<f64>,+ shown_eta: Option<f64>,+ in_flight: Vec<(u64, String, u64)>,+ next_seq: u64,+ tty_width: usize,+}++struct Snapshot {+ phase: &'static str,+ basis: Basis,+ items_done: u64,+ items_total: u64,+ bytes_done: u64,+ bytes_total: u64,+ percent: Option<f64>,+ rate: Option<f64>,+ eta: Eta,+ elapsed: f64,+ phase_elapsed: f64,+ since_progress: f64,+ waiting_on: Vec<(String, u64)>,+}++/// Shared progress state. Cheap for workers: the hot path is two relaxed+/// atomic adds plus one short mutex for the in-flight table.+pub struct Progress {+ surface: Surface,+ interval: Duration,+ started: Instant,+ items_done: AtomicU64,+ bytes_done: AtomicU64,+ reported: AtomicBool,+ state: Mutex<State>,+ stopped: Mutex<bool>,+ wake: Condvar,+ sink: Sink,+}++impl Progress {+ fn build(surface: Surface, interval: Duration, sink: Sink) -> Arc<Self> {+ let now = Instant::now();+ Arc::new(Self {+ surface,+ interval,+ started: now,+ items_done: AtomicU64::new(0),+ bytes_done: AtomicU64::new(0),+ reported: AtomicBool::new(false),+ state: Mutex::new(State {+ phase: "starting",+ items_total: 0,+ bytes_total: 0,+ phase_started: now,+ last_sample_at: now,+ last_sample_units: 0,+ last_advance_at: now,+ rate: None,+ shown_eta: None,+ in_flight: Vec::new(),+ next_seq: 0,+ tty_width: 0,+ }),+ stopped: Mutex::new(false),+ wake: Condvar::new(),+ sink,+ })+ }++ pub fn new(surface: Surface, interval: Duration) -> Arc<Self> {+ // Test hook, same shape as `REPLACEMENT_FAILPOINT` in lib.rs: an+ // end-to-end run must be observable through the REAL surface it uses+ // in production, not through a parallel one built for tests.+ #[cfg(test)]+ if let Some(buffer) = TEST_SINK.lock().unwrap().as_ref() {+ return Self::build(surface, interval, Sink::Buffer(Arc::clone(buffer)));+ }+ Self::build(surface, interval, Sink::Stderr)+ }++ /// A progress handle that emits nothing — `--quiet` / `--progress=none`,+ /// and the default for library callers that do not want a surface.+ pub fn silent() -> Arc<Self> {+ Self::new(Surface::Silent, STREAM_INTERVAL)+ }++ /// Test constructor: same code path, output captured in memory.+ #[cfg(test)]+ pub fn capture(surface: Surface, interval: Duration) -> (Arc<Self>, Arc<Mutex<Vec<u8>>>) {+ let buffer = Arc::new(Mutex::new(Vec::new()));+ let progress = Self::build(surface, interval, Sink::Buffer(Arc::clone(&buffer)));+ (progress, buffer)+ }++ pub fn surface(&self) -> Surface {+ self.surface+ }++ pub fn interval(&self) -> Duration {+ self.interval+ }++ pub fn enabled(&self) -> bool {+ self.surface != Surface::Silent+ }++ /// Enter a phase. Counters reset, the ETA estimator restarts (throughput+ /// in one phase says nothing about the next), and a line is emitted at+ /// once so a phase transition is never hidden behind the tick cadence.+ ///+ /// `items_total` / `bytes_total` of 0 mean "not knowable here" and produce+ /// `pct=unknown` rather than an invented denominator.+ pub fn phase(&self, name: &'static str, items_total: u64, bytes_total: u64) {+ if !self.enabled() {+ return;+ }+ let now = Instant::now();+ {+ let mut state = self.state.lock().unwrap();+ state.phase = name;+ state.items_total = items_total;+ state.bytes_total = bytes_total;+ state.phase_started = now;+ state.last_sample_at = now;+ state.last_sample_units = 0;+ state.last_advance_at = now;+ state.rate = None;+ state.shown_eta = None;+ state.in_flight.clear();+ }+ self.items_done.store(0, Ordering::Relaxed);+ self.bytes_done.store(0, Ordering::Relaxed);+ self.emit();+ }++ /// One unit of the current phase finished.+ pub fn item_done(&self, bytes: u64) {+ self.items_done.fetch_add(1, Ordering::Relaxed);+ if bytes > 0 {+ self.bytes_done.fetch_add(bytes, Ordering::Relaxed);+ }+ }++ /// Register a file as in flight. The returned guard counts it done exactly+ /// once, on every exit path — including the `continue`s that skip a junk+ /// file — because progress measures work *drained from the queue*, not+ /// work that succeeded. Forgetting one would stall the bar below 100%.+ pub fn file(&self, rel: &str, bytes: u64) -> FileGuard<'_> {+ let seq = if self.enabled() {+ let mut state = self.state.lock().unwrap();+ let seq = state.next_seq;+ state.next_seq += 1;+ state.in_flight.push((seq, rel.to_string(), bytes));+ seq+ } else {+ 0+ };+ FileGuard {+ progress: self,+ seq,+ bytes,+ }+ }++ fn file_finished(&self, seq: u64, bytes: u64) {+ if self.enabled() {+ let mut state = self.state.lock().unwrap();+ state+ .in_flight+ .retain(|(candidate, _, _)| *candidate != seq);+ }+ self.item_done(bytes);+ }++ /// A one-off human line. Routed through the same surface so it cannot+ /// scramble the terminal's in-place line, and so `--progress=json` keeps+ /// stderr machine-readable end to end.+ pub fn note(&self, message: &str) {+ match self.surface {+ Surface::Silent => {}+ Surface::Json => {+ let line = serde_json::json!({"event": "note", "message": message});+ self.sink.write(format!("{line}\n").as_bytes());+ }+ Surface::Plain => self.sink.write(format!("{message}\n").as_bytes()),+ Surface::Tty => {+ let mut state = self.state.lock().unwrap();+ let mut out = clear_tty(state.tty_width);+ state.tty_width = 0;+ drop(state);+ out.push_str(message);+ out.push('\n');+ self.sink.write(out.as_bytes());+ }+ }+ }++ /// Render one progress line now.+ pub fn tick(&self) {+ self.emit();+ }++ /// The terminal line, in every mode. An exit code alone is ambiguous —+ /// autoindex exits 3 on success-with-junk — so the reason is spelled out.+ ///+ /// Every run that *reaches an exit* emits exactly one of these: on success+ /// from the call sites below, and on any error path from [`Ticker::drop`].+ /// A run killed by a signal, or aborted by a panic (the release profile is+ /// `panic = "abort"`, so unwinding never reaches `Drop`), cannot print one+ /// — which is itself the signal that the process died abnormally.+ pub fn finish(&self, ok: bool, exit: i32, reason: &str, extra: &[(&str, u64)]) {+ if self.reported.swap(true, Ordering::SeqCst) || !self.enabled() {+ return;+ }+ let wall = self.started.elapsed().as_secs_f64();+ match self.surface {+ Surface::Json => {+ let mut doc = serde_json::Map::new();+ doc.insert("event".into(), "done".into());+ doc.insert("ok".into(), ok.into());+ doc.insert("exit".into(), exit.into());+ doc.insert("reason".into(), reason.into());+ doc.insert("wall_s".into(), serde_json::json!(round1(wall)));+ for (key, value) in extra {+ doc.insert((*key).to_string(), (*value).into());+ }+ let line = serde_json::Value::Object(doc);+ self.sink.write(format!("{line}\n").as_bytes());+ }+ _ => {+ let mut line = String::new();+ if self.surface == Surface::Tty {+ let mut state = self.state.lock().unwrap();+ line.push_str(&clear_tty(state.tty_width));+ state.tty_width = 0;+ }+ line.push_str(&format!(+ "xerj-done ok={ok} exit={exit} reason={reason} wall={wall:.1}s"+ ));+ for (key, value) in extra {+ line.push_str(&format!(" {key}={value}"));+ }+ line.push('\n');+ self.sink.write(line.as_bytes());+ }+ }+ }++ /// Terminal line for a run that returned an error before `finish` — so an+ /// agent never has to infer an outcome from silence on the failure path+ /// either.+ fn finish_aborted(&self) {+ self.finish(false, 1, "aborted", &[]);+ }++ fn snapshot(&self) -> Snapshot {+ let now = Instant::now();+ let items_done = self.items_done.load(Ordering::Relaxed);+ let bytes_done = self.bytes_done.load(Ordering::Relaxed);+ let mut state = self.state.lock().unwrap();++ let basis = if state.bytes_total > 0 {+ Basis::Bytes+ } else if state.items_total > 0 {+ Basis::Items+ } else {+ Basis::None+ };+ let (done, total) = match basis {+ Basis::Bytes => (bytes_done.min(state.bytes_total), state.bytes_total),+ Basis::Items => (items_done.min(state.items_total), state.items_total),+ Basis::None => (0, 0),+ };++ // Throughput sample → EWMA. Guarded on dt so a burst of emits cannot+ // divide by a near-zero interval and produce an absurd rate.+ //+ // A sample where NOTHING completed is deliberately not fed in. Bytes+ // are credited when a file finishes, so a worker grinding through one+ // 20 MB file looks like zero throughput from out here — folding those+ // zeroes into the average drags the rate toward 0 and the ETA toward+ // infinity, which is precisely the moment a user is staring at the+ // screen. Nothing completed is an absence of measurement, not a+ // measurement of zero.+ let dt = now.duration_since(state.last_sample_at).as_secs_f64();+ let advanced = done > state.last_sample_units;+ if total > 0 && dt >= 0.05 && advanced {+ let instant = (done - state.last_sample_units) as f64 / dt;+ state.rate = Some(match state.rate {+ Some(previous) => EWMA_ALPHA * instant + (1.0 - EWMA_ALPHA) * previous,+ None => instant,+ });+ state.last_sample_at = now;+ state.last_sample_units = done;+ state.last_advance_at = now;+ }++ let phase_elapsed = now.duration_since(state.phase_started).as_secs_f64();+ let since_progress = now.duration_since(state.last_advance_at).as_secs_f64();+ let percent = (total > 0).then(|| (done as f64 / total as f64) * 100.0);+ let fraction = if total > 0 {+ done as f64 / total as f64+ } else {+ 0.0+ };+ let eta = if total == 0 || phase_elapsed < ETA_MIN_PHASE_SECS || fraction < ETA_MIN_FRACTION+ {+ Eta::Unknown+ } else if since_progress > self.stall_threshold().as_secs_f64() {+ // The last measurement is too old to extrapolate from. The named+ // straggler below is the real answer to "what is it doing?" — an+ // extrapolated number here would be pure invention.+ Eta::Stalled+ } else {+ match state.rate {+ Some(rate) if rate > 0.0 => {+ let raw = (total - done) as f64 / rate;+ // Clamp the *displayed* movement. Instantaneous throughput+ // swung 34.8x within one measured run; an unclamped ETA+ // jumps around so much it reads as noise.+ let shown = match state.shown_eta {+ Some(previous) if previous > 0.001 => raw.clamp(+ previous * (1.0 - ETA_MAX_STEP),+ previous * (1.0 + ETA_MAX_STEP),+ ),+ _ => raw,+ };+ state.shown_eta = Some(shown);+ if fraction < ETA_GOOD_FRACTION {+ Eta::Rough(shown)+ } else {+ Eta::Good(shown)+ }+ }+ _ => Eta::Unknown,+ }+ };++ // Name the tail. Once the queue is nearly drained the run looks hung+ // while one worker chews through the biggest file in the corpus.+ let mut waiting_on = Vec::new();+ if !state.in_flight.is_empty() && state.in_flight.len() <= STRAGGLER_MAX {+ waiting_on = state+ .in_flight+ .iter()+ .map(|(_, rel, bytes)| (rel.clone(), *bytes))+ .collect();+ }++ Snapshot {+ phase: state.phase,+ basis,+ items_done,+ items_total: state.items_total,+ bytes_done,+ bytes_total: state.bytes_total,+ percent,+ rate: state.rate,+ eta,+ elapsed: now.duration_since(self.started).as_secs_f64(),+ phase_elapsed,+ since_progress,+ waiting_on,+ }+ }++ /// How long nothing may complete before the ETA is withdrawn. Three ticks,+ /// and never less than 15 s so a slow-but-honest phase is not flagged by a+ /// short `--progress-interval`.+ fn stall_threshold(&self) -> Duration {+ (self.interval * 3).max(Duration::from_secs(15))+ }++ fn emit(&self) {+ if !self.enabled() {+ return;+ }+ let snapshot = self.snapshot();+ match self.surface {+ Surface::Silent => {}+ Surface::Json => self+ .sink+ .write(format!("{}\n", render_json(&snapshot)).as_bytes()),+ Surface::Plain => self+ .sink+ .write(format!("{}\n", render_plain(&snapshot)).as_bytes()),+ Surface::Tty => {+ let body = render_tty(&snapshot);+ let width = columns();+ let body = truncate(&body, width);+ let mut state = self.state.lock().unwrap();+ let pad = state.tty_width.saturating_sub(body.chars().count());+ state.tty_width = body.chars().count();+ drop(state);+ let line = format!("\r{body}{}\r", " ".repeat(pad));+ self.sink.write(line.as_bytes());+ }+ }+ }++ /// Start the ticker. Nothing is spawned for a silent surface.+ pub fn spawn_ticker(self: &Arc<Self>) -> Ticker {+ if !self.enabled() {+ return Ticker {+ progress: Arc::clone(self),+ handle: None,+ };+ }+ let progress = Arc::clone(self);+ let interval = self.interval;+ let handle = std::thread::Builder::new()+ .name("autoindex-progress".into())+ .spawn(move || loop {+ let stopped = progress.stopped.lock().unwrap();+ let (stopped, _timeout) = progress+ .wake+ .wait_timeout(stopped, interval)+ .expect("progress ticker mutex poisoned");+ if *stopped {+ break;+ }+ drop(stopped);+ progress.emit();+ })+ .ok();+ Ticker {+ progress: Arc::clone(self),+ handle,+ }+ }++ fn stop(&self) {+ *self.stopped.lock().unwrap() = true;+ self.wake.notify_all();+ }+}++/// Counts its file done on drop, on every path.+pub struct FileGuard<'a> {+ progress: &'a Progress,+ seq: u64,+ bytes: u64,+}++impl Drop for FileGuard<'_> {+ fn drop(&mut self) {+ self.progress.file_finished(self.seq, self.bytes);+ }+}++/// Owns the ticker thread. Dropping it stops the thread and — if the run never+/// called `finish` — closes the stream with an honest `ok=false` line.+pub struct Ticker {+ progress: Arc<Progress>,+ handle: Option<std::thread::JoinHandle<()>>,+}++impl Drop for Ticker {+ fn drop(&mut self) {+ self.progress.stop();+ if let Some(handle) = self.handle.take() {+ let _ = handle.join();+ }+ self.progress.finish_aborted();+ }+}++fn clear_tty(width: usize) -> String {+ if width == 0 {+ String::new()+ } else {+ format!("\r{}\r", " ".repeat(width))+ }+}++fn columns() -> usize {+ std::env::var("COLUMNS")+ .ok()+ .and_then(|value| value.parse::<usize>().ok())+ .filter(|value| *value >= 20)+ .unwrap_or(DEFAULT_COLUMNS)+}++fn truncate(text: &str, width: usize) -> String {+ if text.chars().count() <= width {+ return text.to_string();+ }+ text.chars().take(width.saturating_sub(1)).collect()+}++fn round1(value: f64) -> f64 {+ (value * 10.0).round() / 10.0+}++fn fmt_bytes(bytes: u64) -> String {+ const GB: u64 = 1 << 30;+ const MB: u64 = 1 << 20;+ const KB: u64 = 1 << 10;+ if bytes >= GB {+ format!("{:.1}GB", bytes as f64 / GB as f64)+ } else if bytes >= MB {+ format!("{:.1}MB", bytes as f64 / MB as f64)+ } else if bytes >= KB {+ format!("{:.1}KB", bytes as f64 / KB as f64)+ } else {+ format!("{bytes}B")+ }+}++fn fmt_secs(seconds: f64) -> String {+ let seconds = seconds.max(0.0);+ if seconds < 60.0 {+ format!("{seconds:.0}s")+ } else if seconds < 3600.0 {+ format!(+ "{}m{:02}s",+ (seconds / 60.0) as u64,+ (seconds % 60.0) as u64+ )+ } else {+ format!(+ "{}h{:02}m",+ (seconds / 3600.0) as u64,+ ((seconds % 3600.0) / 60.0) as u64+ )+ }+}++fn render_tty(snapshot: &Snapshot) -> String {+ let mut line = format!("{} ", snapshot.phase);+ match snapshot.percent {+ Some(percent) => line.push_str(&format!("{percent:5.1}% ")),+ None => line.push_str(" ..% "),+ }+ if snapshot.items_total > 0 {+ line.push_str(&format!(+ "| {}/{} items ",+ snapshot.items_done, snapshot.items_total+ ));+ } else if snapshot.items_done > 0 {+ line.push_str(&format!("| {} items ", snapshot.items_done));+ }+ if snapshot.bytes_total > 0 {+ line.push_str(&format!(+ "| {}/{} ",+ fmt_bytes(snapshot.bytes_done),+ fmt_bytes(snapshot.bytes_total)+ ));+ }+ if let Some(rate) = snapshot.rate {+ match snapshot.basis {+ Basis::Bytes => line.push_str(&format!("| {}/s ", fmt_bytes(rate as u64))),+ Basis::Items => line.push_str(&format!("| {rate:.1} items/s ")),+ Basis::None => {}+ }+ }+ match snapshot.eta.seconds() {+ Some(seconds) => line.push_str(&format!("| eta {}", fmt_secs(seconds))),+ None => line.push_str("| eta unknown"),+ }+ if snapshot.waiting_on.len() == 1 {+ line.push_str(&format!(+ " | waiting on {}",+ straggler(&snapshot.waiting_on[0])+ ));+ } else if snapshot.waiting_on.len() > 1 {+ line.push_str(&format!(+ " | waiting on {} items",+ snapshot.waiting_on.len()+ ));+ }+ line+}++/// `path(22.4MB)`, or just `path` for an item with no meaningful byte size+/// (a catalog delete, a correlation query) — `(0B)` would read as a defect.+fn straggler((name, bytes): &(String, u64)) -> String {+ if *bytes > 0 {+ format!("{name}({})", fmt_bytes(*bytes))+ } else {+ name.clone()+ }+}++fn render_plain(snapshot: &Snapshot) -> String {+ let percent = match snapshot.percent {+ Some(percent) => format!("{percent:.1}"),+ None => "unknown".to_string(),+ };+ let rate = match snapshot.rate {+ Some(rate) => format!("{rate:.1}"),+ None => "unknown".to_string(),+ };+ let eta = match snapshot.eta.seconds() {+ Some(seconds) => format!("{seconds:.1}"),+ None => "unknown".to_string(),+ };+ let mut line = format!(+ "xerj-progress phase={} basis={} pct={} items={}/{} bytes={}/{} rate={} eta_s={} \+ eta_quality={} since_progress_s={:.1} phase_elapsed_s={:.1} elapsed_s={:.1}",+ snapshot.phase,+ snapshot.basis.as_str(),+ percent,+ snapshot.items_done,+ snapshot.items_total,+ snapshot.bytes_done,+ snapshot.bytes_total,+ rate,+ eta,+ snapshot.eta.quality(),+ snapshot.since_progress,+ snapshot.phase_elapsed,+ snapshot.elapsed,+ );+ if !snapshot.waiting_on.is_empty() {+ let names: Vec<String> = snapshot.waiting_on.iter().map(straggler).collect();+ line.push_str(&format!(" waiting_on={}", names.join(",")));+ }+ line+}++fn render_json(snapshot: &Snapshot) -> String {+ let waiting: Vec<serde_json::Value> = snapshot+ .waiting_on+ .iter()+ .map(|(rel, bytes)| serde_json::json!({"path": rel, "bytes": bytes}))+ .collect();+ serde_json::json!({+ "event": "progress",+ "phase": snapshot.phase,+ "basis": snapshot.basis.as_str(),+ "pct": snapshot.percent.map(round1),+ "items_done": snapshot.items_done,+ "items_total": snapshot.items_total,+ "bytes_done": snapshot.bytes_done,+ "bytes_total": snapshot.bytes_total,+ "rate": snapshot.rate.map(round1),+ "eta_s": snapshot.eta.seconds().map(round1),+ "eta_quality": snapshot.eta.quality(),+ "since_progress_s": round1(snapshot.since_progress),+ "phase_elapsed_s": round1(snapshot.phase_elapsed),+ "elapsed_s": round1(snapshot.elapsed),+ "waiting_on": waiting,+ })+ .to_string()+}++#[cfg(test)]+mod tests {+ use super::*;++ fn captured(buffer: &Arc<Mutex<Vec<u8>>>) -> String {+ String::from_utf8(buffer.lock().unwrap().clone()).unwrap()+ }++ #[test]+ fn auto_is_a_terminal_surface_only_on_a_real_terminal() {+ assert_eq!(+ resolve(ProgressMode::Auto, true, Some("xterm"), false),+ Surface::Tty+ );+ // The three ways `auto` must degrade to parseable lines.+ assert_eq!(+ resolve(ProgressMode::Auto, false, Some("xterm"), false),+ Surface::Plain+ );+ assert_eq!(+ resolve(ProgressMode::Auto, true, Some("dumb"), false),+ Surface::Plain+ );+ assert_eq!(+ resolve(ProgressMode::Auto, true, Some("xterm"), true),+ Surface::Plain+ );+ // Explicit modes ignore the environment entirely.+ assert_eq!(+ resolve(ProgressMode::Json, true, Some("xterm"), false),+ Surface::Json+ );+ assert_eq!(+ resolve(ProgressMode::None, true, Some("xterm"), false),+ Surface::Silent+ );+ }++ #[test]+ fn unknown_progress_mode_is_rejected_not_defaulted() {+ assert!(ProgressMode::parse("tty").is_err());+ assert!(ProgressMode::parse("").is_err());+ assert_eq!(ProgressMode::parse("json").unwrap(), ProgressMode::Json);+ }++ /// The defect in #241: a blocked phase emitted nothing. Liveness must come+ /// from the clock, not from item completions — a worker can sit inside one+ /// 22 MB file for many seconds while every counter stands still.+ ///+ /// Deliberately written as "wait until it happens, fail on a deadline"+ /// rather than "sleep N and count": a rate assertion measures the machine's+ /// scheduler, not this code. An earlier `sleep(400ms)` + `ticks >= 4` form+ /// of this test failed once at load average 357 with 2 lines — where the+ /// ticker had in fact fired, just twice instead of ten times. The property+ /// under test is *repeated emission with nothing completing*, so three+ /// lines (one from `phase`, two from independent ticker wakeups) prove it+ /// and no upper bound on the machine's slowness can make that false.+ #[test]+ fn ticker_keeps_emitting_while_no_item_completes() {+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_millis(40));+ let ticker = progress.spawn_ticker();+ progress.phase("finalize", 135, 0);++ let deadline = Instant::now() + Duration::from_secs(20);+ let text = loop {+ let text = captured(&buffer);+ let ticks = text+ .lines()+ .filter(|line| line.starts_with("xerj-progress phase=finalize"))+ .count();+ if ticks >= 3 {+ break text;+ }+ assert!(+ Instant::now() < deadline,+ "a phase that completes no items must still prove liveness; \+ only {ticks} line(s) in 20s:\n{text}"+ );+ std::thread::sleep(Duration::from_millis(20));+ };+ drop(ticker);+ assert!(text.contains("items=0/135"), "{text}");+ }++ #[test]+ fn percent_is_bytes_based_when_bytes_are_known() {+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));+ progress.phase("index", 4, 1000);+ // Three of four files done, but only 10% of the bytes: a file-count+ // percent would claim 75% here. The measured corpus had 40.4% of all+ // bytes in ONE file, which is exactly this shape.+ for _ in 0..3 {+ progress.item_done(33);+ }+ progress.tick();+ let text = captured(&buffer);+ let last = text.lines().last().unwrap();+ assert!(last.contains("basis=bytes"), "{last}");+ assert!(last.contains("pct=9.9"), "{last}");+ assert!(last.contains("items=3/4"), "{last}");+ }++ #[test]+ fn percent_is_unknown_when_no_denominator_exists() {+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));+ progress.phase("walk", 0, 0);+ progress.tick();+ let text = captured(&buffer);+ assert!(text.contains("pct=unknown"), "{text}");+ assert!(text.contains("basis=none"), "{text}");+ assert!(text.contains("eta_s=unknown"), "{text}");+ }++ #[test]+ fn eta_stays_unknown_until_the_quality_gate_is_met() {+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));+ progress.phase("index", 100, 1000);+ // Well past 2% of the bytes, but nowhere near 5s into the phase.+ progress.item_done(500);+ progress.tick();+ let text = captured(&buffer);+ assert!(+ text.lines().last().unwrap().contains("eta_s=unknown"),+ "an ETA from a sub-5s sample is not one we can stand behind: {text}"+ );+ assert!(text.contains("eta_quality=unknown"), "{text}");+ }++ #[test]+ fn stragglers_are_named_when_the_tail_goes_quiet() {+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));+ progress.phase("index", 3, 30_000_000);+ let guard = progress.file("tantivy/benches/hdfs.json", 23_488_102);+ progress.tick();+ let text = captured(&buffer);+ assert!(+ text.contains("waiting_on=tantivy/benches/hdfs.json(22.4MB)"),+ "{text}"+ );+ drop(guard);+ assert_eq!(progress.items_done.load(Ordering::Relaxed), 1);+ assert_eq!(progress.bytes_done.load(Ordering::Relaxed), 23_488_102);+ }++ #[test]+ fn json_surface_emits_one_object_per_line_with_nulls_for_unknowns() {+ let (progress, buffer) = Progress::capture(Surface::Json, Duration::from_secs(3600));+ progress.phase("walk", 0, 0);+ progress.note("resuming from journal");+ progress.finish(true, 3, "completed-with-junk", &[("files", 1922)]);+ let text = captured(&buffer);+ for line in text.lines() {+ let value: serde_json::Value =+ serde_json::from_str(line).unwrap_or_else(|e| panic!("{line}: {e}"));+ assert!(value.get("event").is_some(), "{line}");+ }+ let first: serde_json::Value = serde_json::from_str(text.lines().next().unwrap()).unwrap();+ assert!(first["pct"].is_null(), "{text}");+ assert!(first["eta_s"].is_null(), "{text}");+ let last: serde_json::Value = serde_json::from_str(text.lines().last().unwrap()).unwrap();+ assert_eq!(last["event"], "done");+ assert_eq!(last["exit"], 3);+ assert_eq!(last["reason"], "completed-with-junk");+ assert_eq!(last["files"], 1922);+ }++ /// Exit 3 is success. The stream must say so in words, because an agent+ /// reading a bare `3` reads failure.+ #[test]+ fn terminal_line_states_the_outcome_in_words() {+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));+ progress.finish(+ true,+ 3,+ "completed-with-junk",+ &[("files", 12), ("records", 7)],+ );+ let text = captured(&buffer);+ assert!(+ text.starts_with("xerj-done ok=true exit=3 reason=completed-with-junk"),+ "{text}"+ );+ assert!(text.contains("files=12 records=7"), "{text}");+ }++ #[test]+ fn a_run_that_dies_still_closes_the_stream() {+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_millis(20));+ {+ let _ticker = progress.spawn_ticker();+ progress.phase("index", 10, 100);+ }+ let text = captured(&buffer);+ assert!(+ text.contains("xerj-done ok=false exit=1 reason=aborted"),+ "{text}"+ );+ }++ #[test]+ fn silent_surface_writes_nothing_at_all() {+ let (progress, buffer) = Progress::capture(Surface::Silent, Duration::from_millis(10));+ let ticker = progress.spawn_ticker();+ progress.phase("index", 10, 100);+ progress.note("hello");+ progress.item_done(10);+ progress.tick();+ progress.finish(true, 0, "completed", &[]);+ drop(ticker);+ assert!(captured(&buffer).is_empty());+ }++ #[test]+ fn tty_surface_redraws_in_place_and_never_leaves_a_stale_tail() {+ let (progress, buffer) = Progress::capture(Surface::Tty, Duration::from_secs(3600));+ progress.phase("index", 1000, 1_000_000);+ progress.item_done(500_000);+ progress.tick();+ progress.note("graph: 12 structural edges");+ let text = captured(&buffer);+ assert!(+ text.contains('\r'),+ "a tty surface redraws in place: {text:?}"+ );+ assert!(+ text.ends_with("graph: 12 structural edges\n"),+ "a note must land on its own line: {text:?}"+ );+ }++ /// Caught on a real 185 s run, not in review: with 269 of 270 files done+ /// and one 20 MB file still streaming, bytes stop being credited, so+ /// folding those empty samples into the average sent the ETA from 10 s to+ /// 471 s and climbing while roughly 50 s of work remained. An ETA that+ /// runs away exactly when the user is watching is worse than no ETA, so+ /// the estimate is withdrawn and the straggler named instead.+ #[test]+ fn a_stalled_phase_withdraws_its_eta_instead_of_inventing_one() {+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(1));+ progress.phase("index", 270, 31_130_222);+ let big = progress.file("bench/hdfs.jsonl", 20_881_790);+ progress.bytes_done.store(10_248_432, Ordering::Relaxed);+ progress.items_done.store(269, Ordering::Relaxed);+ {+ let mut state = progress.state.lock().unwrap();+ state.phase_started = Instant::now() - Duration::from_secs(120);+ state.last_sample_at = Instant::now() - Duration::from_secs(100);+ state.last_advance_at = Instant::now() - Duration::from_secs(100);+ state.last_sample_units = 10_248_432;+ state.rate = Some(2_545_880.0);+ state.shown_eta = Some(10.3);+ }+ progress.tick();+ let line = captured(&buffer).lines().last().unwrap().to_string();+ assert!(+ line.contains("eta_s=unknown") && line.contains("eta_quality=stalled"),+ "no completion in 100s is an absence of measurement, not a slow rate: {line}"+ );+ assert!(line.contains("since_progress_s=100."), "{line}");+ assert!(+ line.contains("waiting_on=bench/hdfs.jsonl(19.9MB)"),+ "name what it is actually waiting on: {line}"+ );+ // ...and the percent stays honest: 99.6% of the FILES are done, but+ // only a third of the bytes are.+ assert!(line.contains("pct=32.9"), "{line}");+ drop(big);+ }++ /// An empty sample must not drag the measured rate down either — the rate+ /// is a statement about the work that actually completed.+ #[test]+ fn empty_samples_do_not_decay_the_measured_rate() {+ let (progress, _buffer) = Progress::capture(Surface::Plain, Duration::from_secs(1));+ progress.phase("index", 10, 1000);+ progress.item_done(500);+ {+ let mut state = progress.state.lock().unwrap();+ state.phase_started = Instant::now() - Duration::from_secs(10);+ state.last_sample_at = Instant::now() - Duration::from_secs(1);+ }+ let first = progress.snapshot().rate.expect("a sample was taken");+ for _ in 0..20 {+ assert_eq!(progress.snapshot().rate, Some(first));+ }+ }++ #[test]+ fn displayed_eta_cannot_jump_more_than_the_clamp_per_tick() {+ let (progress, _buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));+ {+ let mut state = progress.state.lock().unwrap();+ state.phase = "index";+ state.items_total = 100;+ state.bytes_total = 1000;+ state.phase_started = Instant::now() - Duration::from_secs(30);+ state.last_sample_at = Instant::now() - Duration::from_secs(1);+ state.rate = Some(10.0);+ state.shown_eta = Some(100.0);+ }+ progress.bytes_done.store(500, Ordering::Relaxed);+ let snapshot = progress.snapshot();+ let eta = snapshot.eta.seconds().expect("gate is satisfied");+ assert!(+ (80.0..=120.0).contains(&eta),+ "a displayed ETA may move at most 20% per tick, got {eta}"+ );+ }+}diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rsindex 5296a227f..3ab287bf8 100644--- a/engine/crates/xerj-server/src/brain.rs+++ b/engine/crates/xerj-server/src/brain.rs@@ -443,6 +443,10 @@ fn index_cfg(cfg: &BrainCfg, brain: &str, api_key: Option<String>) -> IndexCfg {dry_run: false,json: false,quiet: false,+ // `xerj brain` is a foreground command a human watches; it gets the+ // same auto-resolved progress surface as `xerj autoindex` (#241).+ progress: xerj_autoindex::progress::ProgressMode::Auto,+ progress_interval: None,}}diff --git a/landing/llms.txt b/landing/llms.txtindex 443ca92f8..4e98a31a4 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -76,7 +76,11 @@ If you are an AI coding agent driving XERJ on a person's machine, indexing is th* **Fast** — `--no-semantic --no-graph`: skips embedding-backed body fields and relationship detection. You still get typed, queryable BM25 + keyword indices over the same files — a better-than-grep experience for a fraction of the time. (Note: XERJ's default embedder is lexical feature-hashing, not neural, so `--no-semantic` costs you less than the word "semantic" suggests unless the node runs `--embed-mode neural`.)* **Narrower** — index the subdirectory that matters, or skip `node_modules`/`vendor`/`target`/build outputs. Usually the biggest win available.* Also useful when their machine must stay usable: `--workers N` and `--pdf-workers N` (which accepts 1–4 only). They bound the **client-side extractor**, not the server — indexing itself runs in the node process, so do not promise a user these cap the machine's total CPU.-- **Never run a long index as one silent blocking call.** Start it in the background with an explicit `--state-dir <dir>`, then poll `xerj autoindex status --state-dir <dir>` with the *same* directory — it prints `N files done, M records, in progress` (or `FINISHED`) plus live per-index doc counts — and relay that to your user periodically. Passing no `--state-dir` to `status` prints one line per journal under `~/.xerj/autoindex`, including unrelated runs from weeks ago, which is how an agent ends up reporting someone else's progress. `status` has **no** `--json` (only `map` does); parse its text, or poll `GET /_cat/indices` / `GET /ax-*/_count` for machine-readable counts.+- **Read the progress stream — it is designed for you.** `autoindex` writes its RESULT to stdout and its LIVENESS to stderr, so `--json` stdout stays one clean parseable document while stderr narrates. When stderr is not a terminal (you), you get one line per interval, by default every 5 s: `xerj-progress phase=index basis=bytes pct=1.9 items=250/251 bytes=693420/37004502 rate=152775.9 eta_s=unknown eta_quality=unknown since_progress_s=5.0 phase_elapsed_s=9.5 elapsed_s=10.0 waiting_on=big-events.jsonl(34.6MB)` (a real line from the run that verified this). Use `--progress json` for one JSON object per line, `--progress-interval SECS` to change the cadence, `--progress none` (or `--quiet`) to silence it. Relay `pct`, `eta_s` and `waiting_on` to your user as they arrive.+ * **`pct` and `eta_s` are honest or absent.** Both can be the literal string `unknown` (JSON `null`) — before a denominator exists, before the estimate has settled, or when nothing has completed recently (`eta_quality=stalled`, meaning one big file is still streaming). Never invent a number to fill the gap; report `waiting_on` instead, which names the file it is actually on.+ * **Percent is bytes-based, not file-based.** `items=250/251` with `pct=1.9` is normal and correct: in that measured corpus one 34.6 MB file held 98% of the bytes, so 250 of 251 files done really was 1.9% of the work. A file-count percent would have claimed 99.6% and then sat there.+ * **Every run that reaches an exit ends with one terminal line, in every mode** — `xerj-done ok=true exit=3 reason=completed-with-junk wall=57.6s files=1922 records=115139`, and `ok=false … reason=aborted` on the error path. Wait for it rather than inferring completion from silence. A run killed by a signal cannot print one, so a process that is gone with no terminal line died; it did not finish.+- **You can still poll from outside.** Start the run in the background with an explicit `--state-dir <dir>`, then poll `xerj autoindex status --state-dir <dir>` with the *same* directory — it prints `N files done, M records, in progress` (or `FINISHED`) plus live per-index doc counts. Passing no `--state-dir` to `status` prints one line per journal under `~/.xerj/autoindex`, including unrelated runs from weeks ago, which is how an agent ends up reporting someone else's progress. `status` has **no** `--json` (only `map` does); parse its text, or poll `GET /_cat/indices` / `GET /ax-*/_count` for machine-readable counts.- **Stopping is safe, so offer it.** The resume journal plus idempotent `_id`s mean an interrupted run resumes where it stopped and never duplicates. A user may abort a long index without losing the work already done.- **Read the exit code correctly.** `0` complete · **`3` completed-with-junk — this is SUCCESS**, junk records are recorded and never fatal; do not report it as a failure · `2` usage error · `1` **any error at all** — it is the catch-all for every failure, so read the `error:` line on stderr before you act. Endpoint-unreachable is only the most common one; a bulk failure, an index-creation conflict, a journal-config mismatch, a state-dir lock conflict, corruption or a disk error all exit 1 too, and restarting the node "because 1 means it is down" will not fix any of them.- **Close the loop.** When it finishes, tell the user the actual elapsed time, how many records landed, and what they can now ask — `xerj autoindex map` is the briefing to summarize from.