Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(autoindex): real progress, honest percent and ETA — no run is silent (#241)

xerj-orge8d9f5fbf17a
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index 748159478..39ed95faf 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -1,6 +1,8 @@
//! Hand-rolled arg parser (house style of xerj-server — no clap).
+use crate::progress::ProgressMode;
use std::path::PathBuf;
+use std::time::Duration;
/// Largest `--bulk-mb` accepted. Past this a single bulk body stops being a
/// unit of work and starts being a memory incident on the server.
@@ -37,6 +39,12 @@ pub struct IndexCfg {
pub dry_run: bool,
pub json: bool,
pub quiet: bool,
+ /// Progress surface. Orthogonal to `json`: `--json` shapes *stdout* (the
+ /// result), `--progress` shapes *stderr* (liveness).
+ pub progress: ProgressMode,
+ /// Progress cadence. `None` means "the surface's default" — 1 s on a
+ /// terminal, 5 s for a pipe.
+ pub progress_interval: Option<Duration>,
}
#[derive(Debug, Clone)]
@@ -101,8 +109,17 @@ pub fn print_help() {
--no-graph skip relationship detection (wikilinks, local links,\n\
section order, directory chains) — no edges are written\n\
--dry-run walk+sniff+infer, print the plan, index nothing\n\
- --json machine-readable output (map: raw catalog docs)\n\
- --quiet errors only\n\
+ --json machine-readable RESULT on stdout (map: raw catalog docs).\n\
+ Orthogonal to --progress, which owns stderr.\n\
+ --progress <MODE> liveness on stderr: auto|plain|json|none (default auto).\n\
+ auto = live redrawn line when stderr is a terminal,\n\
+ otherwise one parseable line per interval. plain and\n\
+ json force that shape everywhere (CI, pipes, agents).\n\
+ --progress-interval <SECS>\n\
+ progress cadence, 1..=3600 (default 1 on a terminal,\n\
+ 5 otherwise). This is the guaranteed upper bound on\n\
+ silence between phases.\n\
+ --quiet errors only (implies --progress none)\n\
--dataset <SLUG> (map) show a single dataset\n\
--help, -h this help\n\
\n\
@@ -126,6 +143,18 @@ pub fn print_help() {
short/structured datasets may infer none). Use --dry-run or `autoindex map` to\n\
confirm a semantic field before attributing an indexing result to embeddings.\n\
\n\
+ PROGRESS STREAM:\n\
+ stdout is the RESULT, stderr is PROGRESS — pipe them separately.\n\
+ Every run that reaches an exit — success OR error — ends with one\n\
+ terminal line, in every mode, so an outcome never has to be guessed\n\
+ from silence:\n\
+ xerj-done ok=true exit=3 reason=completed-with-junk wall=57.6s …\n\
+ (A run killed by a signal cannot print one; a missing terminal line\n\
+ after the process is gone means it died, not that it finished.)\n\
+ --progress plain emits `xerj-progress phase=… pct=… eta_s=…` lines;\n\
+ `pct`/`eta_s` are the literal word `unknown` (JSON null) whenever they\n\
+ cannot be computed honestly, never a filler number.\n\
+ \n\
EXIT CODES: 0 complete; 3 completed-with-junk (junk recorded, never fatal);\n\
2 usage; 1 endpoint unreachable / journal-config mismatch\n"
);
@@ -159,6 +188,8 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
let mut dry_run = false;
let mut json = false;
let mut quiet = false;
+ let mut progress: Option<ProgressMode> = None;
+ let mut progress_interval: Option<Duration> = None;
let mut dataset: Option<String> = None;
while let Some(arg) = it.next() {
@@ -250,6 +281,25 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
"--dry-run" => dry_run = true,
"--json" => json = true,
"--md" => json = false,
+ "--progress" => {
+ let raw = it
+ .next()
+ .ok_or("--progress needs a value: auto, plain, json or none")?;
+ progress = Some(ProgressMode::parse(&raw)?);
+ }
+ "--progress-interval" => {
+ let secs: u64 = it
+ .next()
+ .ok_or("--progress-interval needs a value in seconds")?
+ .parse()
+ .map_err(|_| {
+ "--progress-interval needs an integer in the range 1..=3600".to_string()
+ })?;
+ if !(1..=3_600).contains(&secs) {
+ return Err("--progress-interval must be in the range 1..=3600 seconds".into());
+ }
+ progress_interval = Some(Duration::from_secs(secs));
+ }
"--quiet" => quiet = true,
"--dataset" => dataset = it.next(),
"--help" | "-h" => return Ok(Cmd::Help),
@@ -267,7 +317,41 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
return Err("--prefix must contain at least one [a-z0-9] character".into());
}
+ // Contradictory progress requests are refused, never silently resolved in
+ // one side's favour: accepting a flag we will not honour is the defect
+ // class tracked in #204, and "I asked for progress and got none" is the
+ // exact complaint this option exists to answer.
+ let progress_explicit = progress.is_some() || progress_interval.is_some();
+ if quiet {
+ match progress {
+ Some(ProgressMode::None) | None => {}
+ Some(mode) => {
+ return Err(format!(
+ "--quiet and --progress {} contradict each other: --quiet means no progress \
+ output. Drop one of the two",
+ mode.as_str()
+ ))
+ }
+ }
+ }
+ let progress = if quiet {
+ ProgressMode::None
+ } else {
+ progress.unwrap_or(ProgressMode::Auto)
+ };
+ if progress_interval.is_some() && progress == ProgressMode::None {
+ return Err(
+ "--progress-interval sets the cadence of a progress stream that --progress none / \
+ --quiet turns off. Drop one of the two"
+ .into(),
+ );
+ }
+
match (sub.as_deref(), folder) {
+ (Some("map"), _) | (Some("status"), _) if progress_explicit => Err(format!(
+ "--progress/--progress-interval apply only to indexing, not `autoindex {}`",
+ sub.as_deref().unwrap_or_default()
+ )),
(Some("map"), _) if bulk_timeout_explicit => {
Err("--bulk-timeout-secs applies only to indexing, not `autoindex map`".into())
}
@@ -312,6 +396,8 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
dry_run,
json,
quiet,
+ progress,
+ progress_interval,
}))
}
_ => Ok(Cmd::Help),
@@ -320,7 +406,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {
#[cfg(test)]
mod tests {
- use super::{parse, Cmd};
+ use super::{parse, Cmd, Duration, ProgressMode};
fn index(args: &[&str]) -> super::IndexCfg {
match parse(args.iter().map(|s| s.to_string()).collect()).unwrap() {
@@ -413,6 +499,73 @@ mod tests {
}
}
+ #[test]
+ fn progress_defaults_to_auto_and_quiet_turns_it_off() {
+ let cfg = index(&["data"]);
+ assert_eq!(cfg.progress, ProgressMode::Auto);
+ assert_eq!(cfg.progress_interval, None);
+ assert_eq!(index(&["data", "--quiet"]).progress, ProgressMode::None);
+ assert_eq!(
+ index(&["data", "--progress", "json"]).progress,
+ ProgressMode::Json
+ );
+ // --json shapes stdout, --progress shapes stderr; they never collide.
+ let both = index(&["data", "--json", "--progress", "plain"]);
+ assert!(both.json);
+ assert_eq!(both.progress, ProgressMode::Plain);
+ assert_eq!(
+ index(&["data", "--progress-interval", "30"]).progress_interval,
+ Some(Duration::from_secs(30))
+ );
+ }
+
+ #[test]
+ fn contradictory_and_malformed_progress_requests_are_refused() {
+ for args in [
+ vec!["data", "--progress"],
+ vec!["data", "--progress", "tty"],
+ vec!["data", "--progress", "yes"],
+ vec!["data", "--progress-interval"],
+ vec!["data", "--progress-interval", "0"],
+ vec!["data", "--progress-interval", "3601"],
+ vec!["data", "--progress-interval", "soon"],
+ // Accepting either of these and honouring only one half would be a
+ // silent lie about what the run will print.
+ vec!["data", "--quiet", "--progress", "plain"],
+ vec!["data", "--progress", "none", "--progress-interval", "5"],
+ vec!["data", "--quiet", "--progress-interval", "5"],
+ ] {
+ let rendered = args.join(" ");
+ let err = match parse(args.into_iter().map(str::to_string).collect()) {
+ Err(err) => err,
+ Ok(other) => panic!("`{rendered}` must not be accepted, got {other:?}"),
+ };
+ assert!(
+ err.contains("--progress"),
+ "`{rendered}` must explain itself: {err}"
+ );
+ }
+ // The one benign combination stays benign.
+ assert_eq!(
+ index(&["data", "--quiet", "--progress", "none"]).progress,
+ ProgressMode::None
+ );
+ }
+
+ #[test]
+ fn progress_flags_are_rejected_for_non_index_subcommands() {
+ for args in [
+ vec!["map", "--progress", "json"],
+ vec!["--progress", "json", "map"],
+ vec!["status", "--progress-interval", "5"],
+ ] {
+ let err = parse(args.into_iter().map(str::to_string).collect()).unwrap_err();
+ assert!(err.contains("apply only to indexing"), "{err}");
+ }
+ // --quiet keeps its historical no-op acceptance on map/status.
+ assert!(parse(["map", "--quiet"].into_iter().map(str::to_string).collect()).is_ok());
+ }
+
#[test]
fn bulk_timeout_is_rejected_for_non_index_subcommands_in_any_position() {
for args in [
diff --git a/engine/crates/xerj-autoindex/src/content.rs b/engine/crates/xerj-autoindex/src/content.rs
index 9793b7c13..f934ca238 100644
--- a/engine/crates/xerj-autoindex/src/content.rs
+++ b/engine/crates/xerj-autoindex/src/content.rs
@@ -35,9 +35,20 @@ pub(crate) static DIGEST_RAN_OFF_THE_SCAN_POOL: std::sync::atomic::AtomicBool =
///
/// This is linear in corpus bytes for adversarial common-prefix inputs rather
/// than pairwise O(n² × file_size).
-pub fn resolve(files: Vec<FileEntry>) -> Result<Inventory> {
+///
+/// `hashed` is called with each file's byte count as it completes.
+/// Hashing reads the whole corpus, so on a large tree this is minutes of work
+/// that reported nothing before #241. The callback runs on the scan pool's
+/// workers and must therefore be `Sync`; it does no I/O of its own — it only
+/// bumps the progress counters that the ticker thread reads.
+pub fn resolve_reporting(
+ files: Vec<FileEntry>,
+ hashed: &(dyn Fn(u64) + Sync),
+) -> Result<Inventory> {
// Phase A belongs to the run's scan pool, not to rayon's global pool:
// `--workers` has to bound the CPU-bound phase to mean anything (#240 §2).
+ // The progress callback therefore fires from scan-pool threads, which is
+ // also the pool whose width the ETA is computed against (#241).
let digests: Vec<Result<String>> = crate::pool::install(|| {
files
.par_iter()
@@ -56,7 +67,9 @@ pub fn resolve(files: Vec<FileEntry>) -> Result<Inventory> {
.store(true, std::sync::atomic::Ordering::Relaxed);
}
}
- full_digest(&file.path, file.size)
+ let digest = full_digest(&file.path, file.size);
+ hashed(file.size);
+ digest
})
.collect()
});
@@ -214,11 +227,20 @@ mod tests {
use super::*;
use std::fs;
+ /// These tests are about identity resolution, not progress reporting.
+ fn resolve(files: Vec<FileEntry>) -> Result<Inventory> {
+ resolve_reporting(files, &|_| {})
+ }
+
/// #240 §2: the digest phase ran on rayon's default global pool — every
/// core on the machine — so `--workers` never bounded it. It must run on
/// the run's own scan pool, whose width the CLI sets.
+ ///
+ /// #241 rides on the same closure: moving phase A into a private pool must
+ /// not cost the progress stream a single completion, or the percent the
+ /// ticker prints would stall below 100 for the rest of the phase.
#[test]
- fn the_digest_phase_runs_on_the_runs_scan_pool() {
+ fn the_digest_phase_runs_on_the_runs_scan_pool_and_still_reports_every_file() {
let dir = tempfile::tempdir().unwrap();
for i in 0..32 {
fs::write(dir.path().join(format!("f{i}.txt")), format!("body {i}\n")).unwrap();
@@ -226,7 +248,16 @@ mod tests {
OBSERVED_DIGEST_POOL_THREADS.store(0, std::sync::atomic::Ordering::Relaxed);
DIGEST_RAN_OFF_THE_SCAN_POOL.store(false, std::sync::atomic::Ordering::Relaxed);
- resolve(crate::walk::walk(dir.path(), false).unwrap()).unwrap();
+ let files = crate::walk::walk(dir.path(), false).unwrap();
+ let expected_files = files.len() as u64;
+ let expected_bytes: u64 = files.iter().map(|f| f.size).sum();
+ let reported_files = std::sync::atomic::AtomicU64::new(0);
+ let reported_bytes = std::sync::atomic::AtomicU64::new(0);
+ resolve_reporting(files, &|bytes| {
+ reported_files.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ reported_bytes.fetch_add(bytes, std::sync::atomic::Ordering::Relaxed);
+ })
+ .unwrap();
assert!(
!DIGEST_RAN_OFF_THE_SCAN_POOL.load(std::sync::atomic::Ordering::Relaxed),
@@ -237,6 +268,16 @@ mod tests {
crate::pool::scan_pool().current_num_threads(),
"the digest phase must see exactly the scan pool the run configured"
);
+ assert_eq!(
+ reported_files.load(std::sync::atomic::Ordering::Relaxed),
+ expected_files,
+ "every hashed file must still be credited to the progress stream"
+ );
+ assert_eq!(
+ reported_bytes.load(std::sync::atomic::Ordering::Relaxed),
+ expected_bytes,
+ "the bytes-based percent must see the whole corpus"
+ );
}
#[test]
diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rs
index 72c9446ae..8a228f0de 100644
--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs
+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs
@@ -275,6 +275,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
dry_run: false,
json: false,
quiet: true,
+ progress: crate::progress::ProgressMode::None,
+ progress_interval: None,
}
}
diff --git a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
index fe0c6a816..401998ee6 100644
--- a/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
+++ b/engine/crates/xerj-autoindex/src/failure_resume_http_tests.rs
@@ -276,6 +276,8 @@ fn cfg(root: &Path, state_dir: &Path, url: &str) -> IndexCfg {
dry_run: false,
json: false,
quiet: true,
+ progress: crate::progress::ProgressMode::None,
+ progress_interval: None,
}
}
@@ -1100,3 +1102,112 @@ fn zero_live_documents_after_journaled_records_fails_the_run() {
assert!(error.contains("0 documents are live"), "{error}");
assert!(error.contains("--fresh"), "{error}");
}
+
+/// #241 regression: a run must never be silent.
+///
+/// Before this fix `autoindex` printed a handful of phase banners, one line
+/// per 200 completed files, and then nothing at all through the entire
+/// `finalize` block — measured at 47-64% of wall in the issue and at 45% (100
+/// of 222 s) on the corpus used to reproduce it here. An agent watching the
+/// stream could not tell a running index from a hung one.
+///
+/// This asserts the three properties that make the stream readable, through
+/// the REAL progress surface a production run uses:
+/// 1. phase B reports as it goes, with a percent and an ETA field;
+/// 2. the previously-silent finalize block reports at all;
+/// 3. every run ends with a terminal line that states the outcome in words,
+/// because exit 3 is a success an agent otherwise reads as failure.
+#[test]
+fn a_run_reports_progress_through_every_phase_and_closes_the_stream() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _sink_guard = crate::progress::SINK_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ // Distinct bytes per file: byte-identical files collapse into one
+ // canonical file plus aliases, which would make `files=8` a lie.
+ for n in 0..8 {
+ fs::write(
+ corpus.path().join(format!("rows-{n}.csv")),
+ format!("id,value\n{n}0,alpha\n{n}1,beta\n{n}2,gamma\n"),
+ )
+ .unwrap();
+ }
+
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let mut config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ config.quiet = false;
+ config.progress = crate::progress::ProgressMode::Plain;
+ config.progress_interval = Some(std::time::Duration::from_secs(1));
+
+ let buffer = Arc::new(Mutex::new(Vec::new()));
+ let (code, _report) = {
+ let _sink = crate::progress::install_test_sink(&buffer);
+ run_index_report(config).unwrap()
+ };
+ let stream = String::from_utf8(buffer.lock().unwrap().clone()).unwrap();
+
+ let progress_lines: Vec<&str> = stream
+ .lines()
+ .filter(|line| line.starts_with("xerj-progress "))
+ .collect();
+ assert!(
+ progress_lines.len() >= 5,
+ "a run must narrate its phases, got {} line(s):\n{stream}",
+ progress_lines.len()
+ );
+ assert!(
+ progress_lines
+ .iter()
+ .any(|line| line.contains("phase=index")),
+ "phase B must report:\n{stream}"
+ );
+ assert!(
+ progress_lines
+ .iter()
+ .any(|line| line.contains("phase=finalize")),
+ "the finalize block was the longest silence in the tool; it must report now:\n{stream}"
+ );
+ assert!(
+ progress_lines
+ .iter()
+ .all(|line| line.contains(" pct=") && line.contains(" eta_s=")),
+ "every progress line carries a percent and an ETA field (possibly `unknown`):\n{stream}"
+ );
+
+ let done = stream
+ .lines()
+ .find(|line| line.starts_with("xerj-done "))
+ .unwrap_or_else(|| panic!("a run must close its own stream:\n{stream}"));
+ assert!(
+ done.contains("ok=true") && done.contains(&format!("exit={code}")),
+ "{done}"
+ );
+ assert!(done.contains("reason=completed"), "{done}");
+ assert!(done.contains("files=8"), "{done}");
+}
+
+/// The other half of the contract: `--quiet` still means silence, so nothing
+/// here can spam a caller that asked for none.
+#[test]
+fn quiet_runs_emit_no_progress_stream_at_all() {
+ let _guard = FAILPOINT_TEST_LOCK.lock().unwrap();
+ let _sink_guard = crate::progress::SINK_TEST_LOCK.lock().unwrap();
+ let corpus = tempfile::tempdir().unwrap();
+ let state_dir = tempfile::tempdir().unwrap();
+ fs::write(corpus.path().join("rows.csv"), "id,value\n0,alpha\n").unwrap();
+
+ let endpoint = MockEndpoint::start(usize::MAX);
+ let config = cfg(corpus.path(), state_dir.path(), &endpoint.url);
+ assert_eq!(config.progress, crate::progress::ProgressMode::None);
+
+ let buffer = Arc::new(Mutex::new(Vec::new()));
+ {
+ let _sink = crate::progress::install_test_sink(&buffer);
+ run_index_report(config).unwrap();
+ }
+ assert!(
+ buffer.lock().unwrap().is_empty(),
+ "--quiet asked for nothing: {}",
+ String::from_utf8_lossy(&buffer.lock().unwrap())
+ );
+}
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index e4d255704..1c3316034 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -15,6 +15,7 @@ pub mod extract;
pub mod ids;
pub mod infer;
pub mod pool;
+pub mod progress;
pub mod resources;
pub mod sniff;
pub mod state;
@@ -34,6 +35,7 @@ use cli::{Cmd, IndexCfg, MapCfg, StatusCfg};
// `Box<dyn EdgeDetector>` like the registry entries).
use detect::EdgeDetector as _;
use esclient::Es;
+use progress::Progress;
use sniff::{Family, Sniffed};
use state::{DuplicateFile, FileAssignment, FileDone, JunkFile, Plan, PlanDataset};
@@ -411,6 +413,8 @@ mod phase_a_grouping_tests {
dry_run: true,
json: false,
quiet: true,
+ progress: crate::progress::ProgressMode::None,
+ progress_interval: None,
}
}
@@ -421,7 +425,15 @@ mod phase_a_grouping_tests {
.map(|f| ids::file_key(&f.path, f.size).unwrap())
.collect();
let digests: Vec<String> = (0..files.len()).map(|i| format!("d{i}")).collect();
- let (plan, _) = build_phase_a(root, &files, &keys, &digests, Vec::new(), &cfg_for(root));
+ let (plan, _) = build_phase_a(
+ root,
+ &files,
+ &keys,
+ &digests,
+ Vec::new(),
+ &cfg_for(root),
+ &Progress::silent(),
+ );
plan
}
@@ -593,14 +605,20 @@ fn build_phase_a(
digests: &[String],
duplicate_files: Vec<DuplicateFile>,
cfg: &IndexCfg,
+ pr: &Progress,
) -> (Plan, Vec<dataset::Cluster>) {
use rayon::prelude::*;
// Same pool as the digest phase: sniffing and sampling are the other half
- // of the CPU-bound phase `--workers` has to bound (#240 §2).
+ // of the CPU-bound phase `--workers` has to bound (#240 §2). Progress is
+ // reported from inside that pool, so the straggler the ticker names is the
+ // file a scan-pool thread is genuinely sitting on (#241).
let scans: Vec<FileScan> = crate::pool::install(|| {
files
.par_iter()
- .map(|f| scan_file(&f.path, f.size, cfg.sample, cfg.max_file_gb))
+ .map(|f| {
+ let _in_flight = pr.file(&f.rel, f.size);
+ scan_file(&f.path, f.size, cfg.sample, cfg.max_file_gb)
+ })
.collect()
});
@@ -836,7 +854,8 @@ fn select_resume_plan_keys(
// replacement transaction on one ax_file key and delete each
// other's freshly published documents. Divert this file to a
// deterministic path-derived key, the same discriminator scheme
- // content::resolve uses for byte-proven digest collisions.
+ // content::resolve_reporting uses for byte-proven digest
+ // collisions.
Some(format!(
"{content_key}-claimed-{:032x}",
xxhash_rust::xxh3::xxh3_128(file.rel_id.as_bytes())
@@ -917,55 +936,76 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// sniffing are the CPU-bound phase, and they used to take every core no
// matter what the caller asked for (#240 §2).
pool::configure(cfg.scan_workers);
+ // What phase A is *actually* running on. `pool::configure` is first-call-wins
+ // (rayon pools cannot be resized), so in a process that already indexed once
+ // — `xerj-server`'s brain endpoint does exactly that — the installed width
+ // can differ from what this run's plan asked for. Progress must state the
+ // number the policy got, not the one it requested (#240 + #241).
+ let scan_threads = pool::scan_pool().current_num_threads();
extract::pdf::configure_workers(cfg.pdf_workers);
extract::pdf::configure_timeout(cfg.pdf_timeout_secs);
let t0 = Instant::now();
- if !cfg.quiet {
- // What this run decided to take from the machine, before it takes it.
- eprintln!(
- "autoindex: {} scan threads, {} index workers, {} pdf workers, --bulk-mb {} \
- [{}]",
- cfg.scan_workers,
- cfg.workers,
- cfg.pdf_workers,
- cfg.bulk_mb,
- xerj_common::resource::describe(),
- );
- for note in &cfg.resource_notes {
- eprintln!("autoindex: {note}");
- }
- eprintln!(
- "autoindex: bulk HTTP request timeout: {}s",
- cfg.bulk_timeout_secs
- );
- }
+ // The progress surface and its ticker are the FIRST things built: every
+ // later phase reports through them, and the ticker guarantees the stream
+ // closes with a terminal line even if this function bails (#241). The
+ // resource plan is announced through the same surface rather than through a
+ // bare `eprintln!`, so `--quiet` / `--progress none` still means silent and
+ // `--progress json` still means one machine-readable stream (#240 + #241).
+ let surface = progress::detect(cfg.progress);
+ let pr = Progress::new(
+ surface,
+ cfg.progress_interval
+ .unwrap_or_else(|| progress::default_interval(surface)),
+ );
+ let ticker = pr.spawn_ticker();
+ // What this run decided to take from the machine, before it takes it.
+ pr.note(&format!(
+ "autoindex: {} scan threads, {} index workers, {} pdf workers, --bulk-mb {} [{}]",
+ scan_threads,
+ cfg.workers,
+ cfg.pdf_workers,
+ cfg.bulk_mb,
+ xerj_common::resource::describe(),
+ ));
+ for note in &cfg.resource_notes {
+ pr.note(&format!("autoindex: {note}"));
+ }
+ pr.note(&format!(
+ "autoindex: bulk HTTP request timeout: {}s",
+ cfg.bulk_timeout_secs
+ ));
// The run's bulk load is admitted through one window `--workers` wide, so
// a 429 can shrink what the run offers instead of only delaying it
// (#240 §8). Enabled here and nowhere else: probes have nothing to
- // throttle.
+ // throttle. Its shrink/recover announcements go to stderr, which the
+ // progress surface owns — so they are emitted exactly when that surface is
+ // enabled, not merely when `--quiet` is absent.
let es = Es::with_bulk_timeout(&cfg.url, cfg.api_key.clone(), cfg.bulk_timeout_secs)?
- .with_bulk_concurrency(cfg.workers, !cfg.quiet);
+ .with_bulk_concurrency(cfg.workers, pr.enabled());
es.ping()?;
+ // Totals are unknown until the walk returns, so this phase honestly
+ // reports `pct=unknown` and proves liveness with the clock alone.
+ pr.phase("walk", 0, 0);
let discovered_files = walk::walk(&cfg.root, cfg.follow_symlinks)?;
if discovered_files.is_empty() {
println!("no files found under {}", cfg.root.display());
+ pr.finish(true, 0, "no-files", &[]);
return Ok((0, None));
}
+ let discovered_bytes: u64 = discovered_files.iter().map(|f| f.size).sum();
let root_str = cfg
.root
.canonicalize()
.unwrap_or_else(|_| cfg.root.clone())
.to_string_lossy()
.to_string();
- if !cfg.quiet {
- eprintln!(
- "autoindex: {} files ({} MB) under {}",
- discovered_files.len(),
- discovered_files.iter().map(|f| f.size).sum::<u64>() / (1 << 20),
- root_str
- );
- }
+ pr.note(&format!(
+ "autoindex: {} files ({} MB) under {}",
+ discovered_files.len(),
+ discovered_bytes / (1 << 20),
+ root_str
+ ));
let state_dir = cfg
.state_dir
@@ -981,18 +1021,21 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
)?;
let resumed_with_plan = journal.plan.is_some();
let run_id = journal.run_id.clone();
- if journal.resumed && !cfg.quiet {
- eprintln!(
+ if journal.resumed {
+ pr.note(&format!(
"resuming from journal {} ({} files already done)",
journal.path().display(),
journal.done.len()
- );
+ ));
}
// Full hashing on every run is deliberate: size/mtime/inode fingerprints
// cannot prove byte identity across all supported local and network
// filesystems. A metadata-only shortcut could leave stale live documents
// forever after a same-size rewrite with restored or stale timestamps.
- let mut inventory = content::resolve(discovered_files)?;
+ // Hashing reads every byte of the corpus. On a large tree it is minutes of
+ // real work, and before #241 it was minutes with no output at all.
+ pr.phase("hash", discovered_files.len() as u64, discovered_bytes);
+ let mut inventory = content::resolve_reporting(discovered_files, &|bytes| pr.item_done(bytes))?;
let journal_path = journal.path().to_path_buf();
let mut content_changed = std::collections::HashSet::new();
let mut stale_alias_ids = Vec::new();
@@ -1121,19 +1164,20 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
let digests = inventory.digests;
let duplicate_files = inventory.duplicates;
let paths_discovered = files.len() + duplicate_files.len();
- if !duplicate_files.is_empty() && !cfg.quiet {
- eprintln!(
+ let planned_bytes: u64 = files.iter().map(|f| f.size).sum();
+ if !duplicate_files.is_empty() {
+ pr.note(&format!(
"autoindex: {} byte-identical duplicate path(s) will reuse canonical content",
duplicate_files.len()
- );
+ ));
for duplicate in duplicate_files.iter().take(10) {
- eprintln!(
+ pr.note(&format!(
" duplicate: {} → {}",
duplicate.rel, duplicate.duplicate_of
- );
+ ));
}
if duplicate_files.len() > 10 {
- eprintln!(" … and {} more", duplicate_files.len() - 10);
+ pr.note(&format!(" … and {} more", duplicate_files.len() - 10));
}
}
@@ -1142,9 +1186,14 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
let mut plan: Plan = if let Some(p) = journal.plan.clone() {
p
} else {
- if !cfg.quiet {
- eprintln!("phase A: sniffing + sampling {} files…", files.len());
- }
+ pr.phase("scan", files.len() as u64, planned_bytes);
+ // Named with the width phase A is really running at: `--workers` only
+ // started governing this phase in #240, so "how wide is it right now"
+ // is exactly the question the reader has.
+ pr.note(&format!(
+ "phase A: sniffing + sampling {} files with {scan_threads} threads…",
+ files.len()
+ ));
let (plan, clusters) = build_phase_a(
&cfg.root,
&files,
@@ -1152,14 +1201,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
&digests,
duplicate_files.clone(),
&cfg,
+ &pr,
);
- if !cfg.quiet {
- eprintln!(
- "phase A: {} datasets inferred, {} junk/skipped files",
- plan.datasets.len(),
- plan.junk_files.len()
- );
- }
+ pr.note(&format!(
+ "phase A: {} datasets inferred, {} junk/skipped files",
+ plan.datasets.len(),
+ plan.junk_files.len()
+ ));
clusters_rt = Some(clusters);
plan
};
@@ -1197,7 +1245,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
if cfg.dry_run {
println!("{}", serde_json::to_string_pretty(&plan)?);
- eprintln!("(dry run — nothing indexed)");
+ pr.note("(dry run — nothing indexed)");
+ pr.finish(true, 0, "dry-run", &[("files", files.len() as u64)]);
return Ok((0, None));
}
@@ -1217,6 +1266,8 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
}
// ── create indices with explicit mappings ────────────────────────────
+ // Two round trips per dataset; a 135-dataset plan is a real wait.
+ pr.phase("prepare", plan.datasets.len() as u64, 0);
for d in &plan.datasets {
es.ensure_index(&d.index, &build_mapping(&d.specs))
.with_context(|| format!("create index {}", d.index))?;
@@ -1225,6 +1276,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
&json!({"properties": {"ax_paths": {"type": "keyword"}}}),
)
.with_context(|| format!("upgrade alias-path mapping for {}", d.index))?;
+ pr.item_done(0);
}
es.ensure_index(catalog::CATALOG_INDEX, &catalog::catalog_mapping())?;
es.update_mapping(
@@ -1365,6 +1417,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
let graph: Option<GraphRt> = if cfg.no_graph {
None
} else {
+ pr.phase("graph", files.len() as u64, 0);
let brain = match &cfg.brain {
Some(b) => b.clone(),
None => derive_brain_name(&cfg.root),
@@ -1404,6 +1457,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
corpus_files.push(detect::corpus_file(
&f.rel, key, &slug, &fa.family, mtime_ms,
));
+ pr.item_done(0);
}
let corpus = detect::CorpusIndex::build(corpus_files);
let detectors = detect::default_detectors();
@@ -1476,15 +1530,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
anyhow::bail!("write structural graph edges to {edges_index}: {e}");
}
}
- if !cfg.quiet {
- eprintln!(
- "graph: brain '{brain}' → {edges_index}; {} structural edges, {} prior edges \
- invalidated ({} detectors live)",
- assembled.edges.len(),
- invalidated,
- detectors.len()
- );
- }
+ pr.note(&format!(
+ "graph: brain '{brain}' → {edges_index}; {} structural edges, {} prior edges \
+ invalidated ({} detectors live)",
+ assembled.edges.len(),
+ invalidated,
+ detectors.len()
+ ));
Some(GraphRt {
corpus,
detectors,
@@ -1502,12 +1554,15 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// start first and can't serialize the end of the run.
todo.sort_by_key(|&i| files[i].size);
let n_todo = todo.len();
- if !cfg.quiet {
- eprintln!(
- "phase B: indexing {} files with {} workers → {}",
- n_todo, cfg.workers, cfg.url
- );
- }
+ // Percent and ETA are bytes-based, never file-count-based. The queue is
+ // biggest-first, so a files-done percent races to ~100% and then sits
+ // there for minutes on the one big file still in flight (#241 §5/§6).
+ let todo_bytes: u64 = todo.iter().map(|&i| files[i].size).sum();
+ pr.phase("index", n_todo as u64, todo_bytes);
+ pr.note(&format!(
+ "phase B: indexing {} files with {} workers → {}",
+ n_todo, cfg.workers, cfg.url
+ ));
let queue = Mutex::new(todo);
let mut paths_by_key: HashMap<String, Vec<String>> = files
@@ -1539,6 +1594,11 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
None => break,
};
let f = &files[i];
+ // Counts this file done on EVERY exit path below, junk
+ // included: progress measures work drained from the queue,
+ // and a `continue` that skipped the count would park the
+ // bar short of 100% forever.
+ let _in_flight = pr.file(&f.rel, f.size);
let key = &keys[i];
let expected_digest = &digests[i];
let fa = plan.files.get(key).unwrap();
@@ -1975,10 +2035,10 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
continue;
}
}
- let dn = files_done.fetch_add(1, Ordering::Relaxed) + 1;
- if !cfg.quiet && (dn.is_multiple_of(200) || f.size > 5 * (1 << 20)) {
- eprintln!(" [{dn}/{n_todo}] {} ({} records)", f.rel, file_records);
- }
+ // Workers no longer print. A worker can block for minutes
+ // inside one file, so a worker-driven heartbeat cannot
+ // bound silence; the ticker thread can, and does.
+ files_done.fetch_add(1, Ordering::Relaxed);
}
});
}
@@ -1993,9 +2053,11 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// and edges over a half-read corpus would be edges over a lie.
if let Some(gr) = &graph {
if bulk_errors.lock().unwrap().is_empty() {
+ pr.phase("graph-corpus", gr.detectors.len() as u64, 0);
let mut drafts = Vec::new();
for det in &gr.detectors {
det.detect_corpus(&gr.corpus, &mut drafts);
+ pr.item_done(0);
}
if !drafts.is_empty() {
let out = detect::assemble(&drafts, &gr.edges_index, gr.created_at_ms);
@@ -2047,16 +2109,27 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
}
// ── finalize: refresh, verify, correlate, catalog ────────────────────
+ //
+ // This block was 47-64% of every measured run and emitted NOTHING between
+ // the last phase-B line and the final summary (#241 §1). Its work is fully
+ // countable before each loop starts, so it is reported like any other
+ // phase — split into named sub-phases because the mix of one-shot refreshes
+ // and per-dataset round trips has no single honest denominator.
+ pr.phase("finalize-refresh", 1 + graph.is_some() as u64, 0);
es.refresh(&format!("{}-*", cfg.prefix)).ok();
+ pr.item_done(0);
// The dot-prefixed edges index is outside the {prefix}-* pattern.
if let Some(gr) = &graph {
es.refresh(&gr.edges_index).ok();
+ pr.item_done(0);
}
// live per-dataset counts + time ranges (every claim traces to a run)
+ pr.phase("finalize-count", plan.datasets.len() as u64, 0);
let mut ds_counts: HashMap<String, u64> = HashMap::new();
let mut ds_timerange: HashMap<String, (Option<String>, Option<String>)> = HashMap::new();
for d in &plan.datasets {
+ let _counted = pr.file(&d.index, 0);
let cnt = es.count(&d.index).unwrap_or(0);
ds_counts.insert(d.slug.clone(), cnt);
if let Some(t) = &d.time_field {
@@ -2140,18 +2213,33 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
}
}
key_corrs = correlate::key_overlaps(&cands);
+ // One live query per candidate overlap, and the bound is known before
+ // the loop starts — `phase=finalize-correlate items=37/135` needs no
+ // new bookkeeping at all (#241 §8).
+ pr.phase("finalize-correlate", key_corrs.len() as u64, 0);
for c in key_corrs.iter_mut() {
+ let _query = pr.file(
+ &format!("{}.{} ~ {}.{}", c.a_slug, c.a_field, c.b_slug, c.b_field),
+ 0,
+ );
correlate::confirm(&es, c, 20).ok();
}
// keep only live-confirmed overlaps in the report
key_corrs.retain(|c| c.confirmed.map(|(n, _)| n > 0).unwrap_or(false));
- } else if !cfg.quiet {
- eprintln!("(resumed run: key-overlap correlations kept from the original run's catalog)");
+ } else {
+ pr.note("(resumed run: key-overlap correlations kept from the original run's catalog)");
}
+ let timed: Vec<&PlanDataset> = plan
+ .datasets
+ .iter()
+ .filter(|d| d.time_field.is_some())
+ .collect();
+ pr.phase("finalize-histogram", timed.len() as u64, 0);
let mut series = Vec::new();
- for d in &plan.datasets {
+ for d in timed {
if let Some(t) = &d.time_field {
+ let _query = pr.file(&d.index, 0);
if let Ok(Some(s)) = correlate::fetch_histogram(&es, &d.slug, &d.index, t) {
series.push(s);
}
@@ -2163,7 +2251,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
// Alias IDs changed as identity evolved. Remove by logical path first so
// catalogs created by any previous identity scheme cannot survive beside
// the one current alias document.
+ pr.phase(
+ "finalize-catalog",
+ alias_paths_to_replace.len() as u64 + 1,
+ 0,
+ );
for path in &alias_paths_to_replace {
+ let _delete = pr.file(path, 0);
es.delete_by_query(
catalog::CATALOG_INDEX,
&json!({
@@ -2405,7 +2499,11 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
"workers": cfg.workers,
// The whole resource decision, so a run can be explained after the
// fact from its own summary rather than from the machine it ran on.
- "scan_workers": cfg.scan_workers,
+ // `scan_workers` is the width phase A actually ran at — see the
+ // first-call-wins note at the top of this function; a requested width
+ // that lost that race would make the summary a record of an intention
+ // rather than of a run.
+ "scan_workers": scan_threads,
"pdf_workers": cfg.pdf_workers,
"bulk_mb": cfg.bulk_mb,
"cores_available": xerj_common::resource::cores(),
@@ -2445,6 +2543,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
}
}
es.refresh(catalog::CATALOG_INDEX).ok();
+ pr.item_done(0);
// ── durable junk record, written last (#238) ─────────────────────────
//
// Only now is the plan allowed to claim what the catalog holds: the
@@ -2523,6 +2622,25 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
} else {
0
};
+ // Terminal line, in every mode. Exit 3 means "completed, some files were
+ // unparseable" — success — and an agent reading a bare `3` off a silent
+ // stream reads failure (#241 §9). Say it in words.
+ pr.finish(
+ true,
+ code,
+ if code == 3 {
+ "completed-with-junk"
+ } else {
+ "completed"
+ },
+ &[
+ ("files", files_done.load(Ordering::Relaxed)),
+ ("records", total_records),
+ ("datasets", plan.datasets.len() as u64),
+ ("junk_files", junk_file_count as u64),
+ ],
+ );
+ drop(ticker);
Ok((code, Some(run_doc)))
}
@@ -2820,7 +2938,7 @@ mod duplicate_integration_tests {
fs::write(corpus.path().join("a.txt"), a).unwrap();
fs::write(corpus.path().join("b.txt"), b).unwrap();
let files = walk::walk(corpus.path(), false).unwrap();
- let inventory = content::resolve(files.clone()).unwrap();
+ let inventory = content::resolve_reporting(files.clone(), &|_| {}).unwrap();
let legacy = ids::file_key(&files[0].path, files[0].size).unwrap();
assert_eq!(
legacy,
@@ -2856,7 +2974,8 @@ mod duplicate_integration_tests {
// with — so b.txt's content key IS the planned key a.txt claims by rel.
fs::write(corpus.path().join("a.txt"), b"rewritten content\n").unwrap();
fs::write(corpus.path().join("b.txt"), b"original planned content\n").unwrap();
- let inventory = content::resolve(walk::walk(corpus.path(), false).unwrap()).unwrap();
+ let inventory =
+ content::resolve_reporting(walk::walk(corpus.path(), false).unwrap(), &|_| {}).unwrap();
let planned_key = inventory.keys[1].clone();
let mut plan = Plan::default();
plan.files
@@ -2927,7 +3046,7 @@ mod duplicate_integration_tests {
fs::write(corpus.path().join("report-copy.txt"), body).unwrap();
let discovered = walk::walk(corpus.path(), false).unwrap();
- let inventory = content::resolve(discovered).unwrap();
+ let inventory = content::resolve_reporting(discovered, &|_| {}).unwrap();
assert_eq!(inventory.files.len(), 1);
assert_eq!(inventory.duplicates.len(), 1);
@@ -2999,7 +3118,8 @@ mod duplicate_integration_tests {
}
fs::write(&path, csv).unwrap();
- let inventory = content::resolve(walk::walk(corpus.path(), false).unwrap()).unwrap();
+ let inventory =
+ content::resolve_reporting(walk::walk(corpus.path(), false).unwrap(), &|_| {}).unwrap();
let expected_size = inventory.files[0].size;
let expected_digest = inventory.digests[0].clone();
let sniffed = sniff::sniff(&path).unwrap();
diff --git a/engine/crates/xerj-autoindex/src/progress.rs b/engine/crates/xerj-autoindex/src/progress.rs
new file mode 100644
index 000000000..bbec0c515
--- /dev/null
+++ b/engine/crates/xerj-autoindex/src/progress.rs
@@ -0,0 +1,1208 @@
+//! Progress, percent and ETA for `xerj autoindex` (#241).
+//!
+//! The contract this module implements:
+//!
+//! * **stdout is the RESULT, stderr is PROGRESS.** `--json` keeps stdout a
+//! single parseable document; liveness never contaminates it.
+//! * **One ticker thread owns the progress surface.** Workers only bump
+//! atomics. A worker can block for many seconds inside one large file, so a
+//! worker-driven heartbeat cannot bound silence — a timer can.
+//! * **Liveness is time-based, never item-based.** `--progress-interval`
+//! seconds is the guaranteed upper bound on silence between the first phase
+//! and the terminal line.
+//! * **Nothing is displayed that cannot be computed honestly.** A percent is
+//! emitted only when a real denominator exists, and an ETA only after the
+//! quality gate below is satisfied; otherwise the field is literally
+//! `unknown` (`null` in JSON) rather than a comforting number.
+//!
+//! ETA is derived from **bytes**, not from file count. File count is a badly
+//! skewed proxy for work — in the corpus that produced #241 the largest single
+//! file held 40.4% of all bytes, so a files-done ETA collapses to ~0 while
+//! minutes of work remain.
+//!
+//! Prior art, consulted and adapted rather than copied (both Apache-2.0/MIT and
+//! cited per the repo's reference-coding rule):
+//!
+//! * `quickwit-common/src/progress.rs:78` `record_progress()`, `:90`
+//! `protect_zone()`, `:117` `registered_activity_since_last_call()` — a
+//! supervisor polls an atomic, and work that will block for a long time
+//! declares a *protected zone* so the supervisor does not judge it hung. We
+//! need the same property but cannot ask each call site to declare a zone, so
+//! the ticker here runs unconditionally and [`Progress::file`] names what is
+//! being waited on — the in-flight table is this design's answer to the
+//! question a protected zone answers ("it is not hung, it is doing *this*").
+//! * `meilisearch/crates/milli/src/progress.rs:21` `Progress`, `:137` the
+//! nested overall percentage, `:331` `ProgressStepView { current_step,
+//! finished, total }` — the shape of an honest cross-phase percentage. We
+//! keep the per-phase denominator explicit instead of composing one global
+//! number, because autoindex's phases have no common unit.
+
+use std::io::{IsTerminal, Write};
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::sync::{Arc, Condvar, Mutex};
+use std::time::{Duration, Instant};
+
+/// Smoothing factor on the per-tick throughput sample.
+const EWMA_ALPHA: f64 = 0.2;
+/// No ETA until the phase has run at least this long.
+const ETA_MIN_PHASE_SECS: f64 = 5.0;
+/// No ETA until this fraction of the phase is complete.
+const ETA_MIN_FRACTION: f64 = 0.02;
+/// Below this fraction a computed ETA is labelled `rough`, not `good`.
+const ETA_GOOD_FRACTION: f64 = 0.10;
+/// A displayed ETA may not move by more than this fraction per tick.
+const ETA_MAX_STEP: f64 = 0.20;
+/// Name at most this many still-running files when the tail goes quiet.
+const STRAGGLER_MAX: usize = 3;
+/// Fallback terminal width when `COLUMNS` is unset.
+const DEFAULT_COLUMNS: usize = 100;
+
+/// Live-redraw cadence on a terminal.
+pub const TTY_INTERVAL: Duration = Duration::from_secs(1);
+/// Line cadence for pipes, agents and CI.
+pub const STREAM_INTERVAL: Duration = Duration::from_secs(5);
+
+/// `--progress` as the user wrote it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ProgressMode {
+ /// Terminal → live line; anything else → `plain`.
+ Auto,
+ /// One periodic key=value line, always, whatever stderr is.
+ Plain,
+ /// One periodic JSON object per line.
+ Json,
+ /// Emit nothing (what `--quiet` selects).
+ None,
+}
+
+impl ProgressMode {
+ /// Parse the flag value. An unrecognised mode is an error, never a silent
+ /// fallback to `auto`: accepting a value we will not honour is exactly the
+ /// class of defect tracked in #204.
+ pub fn parse(raw: &str) -> Result<Self, String> {
+ match raw {
+ "auto" => Ok(Self::Auto),
+ "plain" => Ok(Self::Plain),
+ "json" => Ok(Self::Json),
+ "none" => Ok(Self::None),
+ other => Err(format!(
+ "--progress {other}: expected one of auto, plain, json, none"
+ )),
+ }
+ }
+
+ pub fn as_str(self) -> &'static str {
+ match self {
+ Self::Auto => "auto",
+ Self::Plain => "plain",
+ Self::Json => "json",
+ Self::None => "none",
+ }
+ }
+}
+
+/// What `auto` actually resolved to.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Surface {
+ /// In-place redraw with `\r`, for a human at a terminal.
+ Tty,
+ /// One `xerj-progress key=value …` line per tick.
+ Plain,
+ /// One JSON object per tick.
+ Json,
+ /// Nothing at all.
+ Silent,
+}
+
+/// Resolve a mode against the environment. Pure so it can be tested without a
+/// terminal: `auto` is a terminal surface only when stderr is a tty, `TERM` is
+/// not `dumb`, and `CI` is unset.
+pub fn resolve(mode: ProgressMode, stderr_tty: bool, term: Option<&str>, ci: bool) -> Surface {
+ match mode {
+ ProgressMode::None => Surface::Silent,
+ ProgressMode::Plain => Surface::Plain,
+ ProgressMode::Json => Surface::Json,
+ ProgressMode::Auto => {
+ if stderr_tty && term != Some("dumb") && !ci {
+ Surface::Tty
+ } else {
+ Surface::Plain
+ }
+ }
+ }
+}
+
+/// `resolve` against this process's real environment.
+pub fn detect(mode: ProgressMode) -> Surface {
+ let term = std::env::var("TERM").ok();
+ let ci = std::env::var_os("CI").is_some_and(|value| !value.is_empty());
+ resolve(mode, std::io::stderr().is_terminal(), term.as_deref(), ci)
+}
+
+/// Default cadence for a surface: a human wants a second, a log wants five.
+pub fn default_interval(surface: Surface) -> Duration {
+ match surface {
+ Surface::Tty => TTY_INTERVAL,
+ _ => STREAM_INTERVAL,
+ }
+}
+
+/// Serialises the tests that redirect the process-wide progress surface.
+#[cfg(test)]
+pub static SINK_TEST_LOCK: Mutex<()> = Mutex::new(());
+
+#[cfg(test)]
+static TEST_SINK: Mutex<Option<Arc<Mutex<Vec<u8>>>>> = Mutex::new(None);
+
+/// Redirect every `Progress::new` in this process into `buffer` until the
+/// returned guard drops. Hold `SINK_TEST_LOCK` for the whole window.
+#[cfg(test)]
+pub fn install_test_sink(buffer: &Arc<Mutex<Vec<u8>>>) -> TestSinkGuard {
+ *TEST_SINK.lock().unwrap() = Some(Arc::clone(buffer));
+ TestSinkGuard
+}
+
+#[cfg(test)]
+pub struct TestSinkGuard;
+
+#[cfg(test)]
+impl Drop for TestSinkGuard {
+ fn drop(&mut self) {
+ *TEST_SINK.lock().unwrap() = None;
+ }
+}
+
+enum Sink {
+ Stderr,
+ /// Test-only: the identical render path, captured in memory.
+ #[cfg(test)]
+ Buffer(Arc<Mutex<Vec<u8>>>),
+}
+
+impl Sink {
+ fn write(&self, bytes: &[u8]) {
+ match self {
+ Self::Stderr => {
+ let stderr = std::io::stderr();
+ let mut handle = stderr.lock();
+ let _ = handle.write_all(bytes);
+ let _ = handle.flush();
+ }
+ #[cfg(test)]
+ Self::Buffer(buffer) => buffer.lock().unwrap().extend_from_slice(bytes),
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+enum Eta {
+ Unknown,
+ /// Work is still running but nothing has completed for a long time, so
+ /// there is no current throughput measurement to extrapolate from.
+ Stalled,
+ Rough(f64),
+ Good(f64),
+}
+
+impl Eta {
+ fn seconds(self) -> Option<f64> {
+ match self {
+ Self::Unknown | Self::Stalled => None,
+ Self::Rough(s) | Self::Good(s) => Some(s),
+ }
+ }
+
+ fn quality(self) -> &'static str {
+ match self {
+ Self::Unknown => "unknown",
+ Self::Stalled => "stalled",
+ Self::Rough(_) => "rough",
+ Self::Good(_) => "good",
+ }
+ }
+}
+
+/// Whether the percent/ETA denominator is bytes of work or a count of items.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum Basis {
+ Bytes,
+ Items,
+ None,
+}
+
+impl Basis {
+ fn as_str(self) -> &'static str {
+ match self {
+ Self::Bytes => "bytes",
+ Self::Items => "items",
+ Self::None => "none",
+ }
+ }
+}
+
+struct State {
+ phase: &'static str,
+ items_total: u64,
+ bytes_total: u64,
+ phase_started: Instant,
+ last_sample_at: Instant,
+ last_sample_units: u64,
+ /// When a unit last actually completed. Progress inside a single large
+ /// file is not observable from here, so this is the only honest signal
+ /// that the throughput estimate is still current.
+ last_advance_at: Instant,
+ rate: Option<f64>,
+ shown_eta: Option<f64>,
+ in_flight: Vec<(u64, String, u64)>,
+ next_seq: u64,
+ tty_width: usize,
+}
+
+struct Snapshot {
+ phase: &'static str,
+ basis: Basis,
+ items_done: u64,
+ items_total: u64,
+ bytes_done: u64,
+ bytes_total: u64,
+ percent: Option<f64>,
+ rate: Option<f64>,
+ eta: Eta,
+ elapsed: f64,
+ phase_elapsed: f64,
+ since_progress: f64,
+ waiting_on: Vec<(String, u64)>,
+}
+
+/// Shared progress state. Cheap for workers: the hot path is two relaxed
+/// atomic adds plus one short mutex for the in-flight table.
+pub struct Progress {
+ surface: Surface,
+ interval: Duration,
+ started: Instant,
+ items_done: AtomicU64,
+ bytes_done: AtomicU64,
+ reported: AtomicBool,
+ state: Mutex<State>,
+ stopped: Mutex<bool>,
+ wake: Condvar,
+ sink: Sink,
+}
+
+impl Progress {
+ fn build(surface: Surface, interval: Duration, sink: Sink) -> Arc<Self> {
+ let now = Instant::now();
+ Arc::new(Self {
+ surface,
+ interval,
+ started: now,
+ items_done: AtomicU64::new(0),
+ bytes_done: AtomicU64::new(0),
+ reported: AtomicBool::new(false),
+ state: Mutex::new(State {
+ phase: "starting",
+ items_total: 0,
+ bytes_total: 0,
+ phase_started: now,
+ last_sample_at: now,
+ last_sample_units: 0,
+ last_advance_at: now,
+ rate: None,
+ shown_eta: None,
+ in_flight: Vec::new(),
+ next_seq: 0,
+ tty_width: 0,
+ }),
+ stopped: Mutex::new(false),
+ wake: Condvar::new(),
+ sink,
+ })
+ }
+
+ pub fn new(surface: Surface, interval: Duration) -> Arc<Self> {
+ // Test hook, same shape as `REPLACEMENT_FAILPOINT` in lib.rs: an
+ // end-to-end run must be observable through the REAL surface it uses
+ // in production, not through a parallel one built for tests.
+ #[cfg(test)]
+ if let Some(buffer) = TEST_SINK.lock().unwrap().as_ref() {
+ return Self::build(surface, interval, Sink::Buffer(Arc::clone(buffer)));
+ }
+ Self::build(surface, interval, Sink::Stderr)
+ }
+
+ /// A progress handle that emits nothing — `--quiet` / `--progress=none`,
+ /// and the default for library callers that do not want a surface.
+ pub fn silent() -> Arc<Self> {
+ Self::new(Surface::Silent, STREAM_INTERVAL)
+ }
+
+ /// Test constructor: same code path, output captured in memory.
+ #[cfg(test)]
+ pub fn capture(surface: Surface, interval: Duration) -> (Arc<Self>, Arc<Mutex<Vec<u8>>>) {
+ let buffer = Arc::new(Mutex::new(Vec::new()));
+ let progress = Self::build(surface, interval, Sink::Buffer(Arc::clone(&buffer)));
+ (progress, buffer)
+ }
+
+ pub fn surface(&self) -> Surface {
+ self.surface
+ }
+
+ pub fn interval(&self) -> Duration {
+ self.interval
+ }
+
+ pub fn enabled(&self) -> bool {
+ self.surface != Surface::Silent
+ }
+
+ /// Enter a phase. Counters reset, the ETA estimator restarts (throughput
+ /// in one phase says nothing about the next), and a line is emitted at
+ /// once so a phase transition is never hidden behind the tick cadence.
+ ///
+ /// `items_total` / `bytes_total` of 0 mean "not knowable here" and produce
+ /// `pct=unknown` rather than an invented denominator.
+ pub fn phase(&self, name: &'static str, items_total: u64, bytes_total: u64) {
+ if !self.enabled() {
+ return;
+ }
+ let now = Instant::now();
+ {
+ let mut state = self.state.lock().unwrap();
+ state.phase = name;
+ state.items_total = items_total;
+ state.bytes_total = bytes_total;
+ state.phase_started = now;
+ state.last_sample_at = now;
+ state.last_sample_units = 0;
+ state.last_advance_at = now;
+ state.rate = None;
+ state.shown_eta = None;
+ state.in_flight.clear();
+ }
+ self.items_done.store(0, Ordering::Relaxed);
+ self.bytes_done.store(0, Ordering::Relaxed);
+ self.emit();
+ }
+
+ /// One unit of the current phase finished.
+ pub fn item_done(&self, bytes: u64) {
+ self.items_done.fetch_add(1, Ordering::Relaxed);
+ if bytes > 0 {
+ self.bytes_done.fetch_add(bytes, Ordering::Relaxed);
+ }
+ }
+
+ /// Register a file as in flight. The returned guard counts it done exactly
+ /// once, on every exit path — including the `continue`s that skip a junk
+ /// file — because progress measures work *drained from the queue*, not
+ /// work that succeeded. Forgetting one would stall the bar below 100%.
+ pub fn file(&self, rel: &str, bytes: u64) -> FileGuard<'_> {
+ let seq = if self.enabled() {
+ let mut state = self.state.lock().unwrap();
+ let seq = state.next_seq;
+ state.next_seq += 1;
+ state.in_flight.push((seq, rel.to_string(), bytes));
+ seq
+ } else {
+ 0
+ };
+ FileGuard {
+ progress: self,
+ seq,
+ bytes,
+ }
+ }
+
+ fn file_finished(&self, seq: u64, bytes: u64) {
+ if self.enabled() {
+ let mut state = self.state.lock().unwrap();
+ state
+ .in_flight
+ .retain(|(candidate, _, _)| *candidate != seq);
+ }
+ self.item_done(bytes);
+ }
+
+ /// A one-off human line. Routed through the same surface so it cannot
+ /// scramble the terminal's in-place line, and so `--progress=json` keeps
+ /// stderr machine-readable end to end.
+ pub fn note(&self, message: &str) {
+ match self.surface {
+ Surface::Silent => {}
+ Surface::Json => {
+ let line = serde_json::json!({"event": "note", "message": message});
+ self.sink.write(format!("{line}\n").as_bytes());
+ }
+ Surface::Plain => self.sink.write(format!("{message}\n").as_bytes()),
+ Surface::Tty => {
+ let mut state = self.state.lock().unwrap();
+ let mut out = clear_tty(state.tty_width);
+ state.tty_width = 0;
+ drop(state);
+ out.push_str(message);
+ out.push('\n');
+ self.sink.write(out.as_bytes());
+ }
+ }
+ }
+
+ /// Render one progress line now.
+ pub fn tick(&self) {
+ self.emit();
+ }
+
+ /// The terminal line, in every mode. An exit code alone is ambiguous —
+ /// autoindex exits 3 on success-with-junk — so the reason is spelled out.
+ ///
+ /// Every run that *reaches an exit* emits exactly one of these: on success
+ /// from the call sites below, and on any error path from [`Ticker::drop`].
+ /// A run killed by a signal, or aborted by a panic (the release profile is
+ /// `panic = "abort"`, so unwinding never reaches `Drop`), cannot print one
+ /// — which is itself the signal that the process died abnormally.
+ pub fn finish(&self, ok: bool, exit: i32, reason: &str, extra: &[(&str, u64)]) {
+ if self.reported.swap(true, Ordering::SeqCst) || !self.enabled() {
+ return;
+ }
+ let wall = self.started.elapsed().as_secs_f64();
+ match self.surface {
+ Surface::Json => {
+ let mut doc = serde_json::Map::new();
+ doc.insert("event".into(), "done".into());
+ doc.insert("ok".into(), ok.into());
+ doc.insert("exit".into(), exit.into());
+ doc.insert("reason".into(), reason.into());
+ doc.insert("wall_s".into(), serde_json::json!(round1(wall)));
+ for (key, value) in extra {
+ doc.insert((*key).to_string(), (*value).into());
+ }
+ let line = serde_json::Value::Object(doc);
+ self.sink.write(format!("{line}\n").as_bytes());
+ }
+ _ => {
+ let mut line = String::new();
+ if self.surface == Surface::Tty {
+ let mut state = self.state.lock().unwrap();
+ line.push_str(&clear_tty(state.tty_width));
+ state.tty_width = 0;
+ }
+ line.push_str(&format!(
+ "xerj-done ok={ok} exit={exit} reason={reason} wall={wall:.1}s"
+ ));
+ for (key, value) in extra {
+ line.push_str(&format!(" {key}={value}"));
+ }
+ line.push('\n');
+ self.sink.write(line.as_bytes());
+ }
+ }
+ }
+
+ /// Terminal line for a run that returned an error before `finish` — so an
+ /// agent never has to infer an outcome from silence on the failure path
+ /// either.
+ fn finish_aborted(&self) {
+ self.finish(false, 1, "aborted", &[]);
+ }
+
+ fn snapshot(&self) -> Snapshot {
+ let now = Instant::now();
+ let items_done = self.items_done.load(Ordering::Relaxed);
+ let bytes_done = self.bytes_done.load(Ordering::Relaxed);
+ let mut state = self.state.lock().unwrap();
+
+ let basis = if state.bytes_total > 0 {
+ Basis::Bytes
+ } else if state.items_total > 0 {
+ Basis::Items
+ } else {
+ Basis::None
+ };
+ let (done, total) = match basis {
+ Basis::Bytes => (bytes_done.min(state.bytes_total), state.bytes_total),
+ Basis::Items => (items_done.min(state.items_total), state.items_total),
+ Basis::None => (0, 0),
+ };
+
+ // Throughput sample → EWMA. Guarded on dt so a burst of emits cannot
+ // divide by a near-zero interval and produce an absurd rate.
+ //
+ // A sample where NOTHING completed is deliberately not fed in. Bytes
+ // are credited when a file finishes, so a worker grinding through one
+ // 20 MB file looks like zero throughput from out here — folding those
+ // zeroes into the average drags the rate toward 0 and the ETA toward
+ // infinity, which is precisely the moment a user is staring at the
+ // screen. Nothing completed is an absence of measurement, not a
+ // measurement of zero.
+ let dt = now.duration_since(state.last_sample_at).as_secs_f64();
+ let advanced = done > state.last_sample_units;
+ if total > 0 && dt >= 0.05 && advanced {
+ let instant = (done - state.last_sample_units) as f64 / dt;
+ state.rate = Some(match state.rate {
+ Some(previous) => EWMA_ALPHA * instant + (1.0 - EWMA_ALPHA) * previous,
+ None => instant,
+ });
+ state.last_sample_at = now;
+ state.last_sample_units = done;
+ state.last_advance_at = now;
+ }
+
+ let phase_elapsed = now.duration_since(state.phase_started).as_secs_f64();
+ let since_progress = now.duration_since(state.last_advance_at).as_secs_f64();
+ let percent = (total > 0).then(|| (done as f64 / total as f64) * 100.0);
+ let fraction = if total > 0 {
+ done as f64 / total as f64
+ } else {
+ 0.0
+ };
+ let eta = if total == 0 || phase_elapsed < ETA_MIN_PHASE_SECS || fraction < ETA_MIN_FRACTION
+ {
+ Eta::Unknown
+ } else if since_progress > self.stall_threshold().as_secs_f64() {
+ // The last measurement is too old to extrapolate from. The named
+ // straggler below is the real answer to "what is it doing?" — an
+ // extrapolated number here would be pure invention.
+ Eta::Stalled
+ } else {
+ match state.rate {
+ Some(rate) if rate > 0.0 => {
+ let raw = (total - done) as f64 / rate;
+ // Clamp the *displayed* movement. Instantaneous throughput
+ // swung 34.8x within one measured run; an unclamped ETA
+ // jumps around so much it reads as noise.
+ let shown = match state.shown_eta {
+ Some(previous) if previous > 0.001 => raw.clamp(
+ previous * (1.0 - ETA_MAX_STEP),
+ previous * (1.0 + ETA_MAX_STEP),
+ ),
+ _ => raw,
+ };
+ state.shown_eta = Some(shown);
+ if fraction < ETA_GOOD_FRACTION {
+ Eta::Rough(shown)
+ } else {
+ Eta::Good(shown)
+ }
+ }
+ _ => Eta::Unknown,
+ }
+ };
+
+ // Name the tail. Once the queue is nearly drained the run looks hung
+ // while one worker chews through the biggest file in the corpus.
+ let mut waiting_on = Vec::new();
+ if !state.in_flight.is_empty() && state.in_flight.len() <= STRAGGLER_MAX {
+ waiting_on = state
+ .in_flight
+ .iter()
+ .map(|(_, rel, bytes)| (rel.clone(), *bytes))
+ .collect();
+ }
+
+ Snapshot {
+ phase: state.phase,
+ basis,
+ items_done,
+ items_total: state.items_total,
+ bytes_done,
+ bytes_total: state.bytes_total,
+ percent,
+ rate: state.rate,
+ eta,
+ elapsed: now.duration_since(self.started).as_secs_f64(),
+ phase_elapsed,
+ since_progress,
+ waiting_on,
+ }
+ }
+
+ /// How long nothing may complete before the ETA is withdrawn. Three ticks,
+ /// and never less than 15 s so a slow-but-honest phase is not flagged by a
+ /// short `--progress-interval`.
+ fn stall_threshold(&self) -> Duration {
+ (self.interval * 3).max(Duration::from_secs(15))
+ }
+
+ fn emit(&self) {
+ if !self.enabled() {
+ return;
+ }
+ let snapshot = self.snapshot();
+ match self.surface {
+ Surface::Silent => {}
+ Surface::Json => self
+ .sink
+ .write(format!("{}\n", render_json(&snapshot)).as_bytes()),
+ Surface::Plain => self
+ .sink
+ .write(format!("{}\n", render_plain(&snapshot)).as_bytes()),
+ Surface::Tty => {
+ let body = render_tty(&snapshot);
+ let width = columns();
+ let body = truncate(&body, width);
+ let mut state = self.state.lock().unwrap();
+ let pad = state.tty_width.saturating_sub(body.chars().count());
+ state.tty_width = body.chars().count();
+ drop(state);
+ let line = format!("\r{body}{}\r", " ".repeat(pad));
+ self.sink.write(line.as_bytes());
+ }
+ }
+ }
+
+ /// Start the ticker. Nothing is spawned for a silent surface.
+ pub fn spawn_ticker(self: &Arc<Self>) -> Ticker {
+ if !self.enabled() {
+ return Ticker {
+ progress: Arc::clone(self),
+ handle: None,
+ };
+ }
+ let progress = Arc::clone(self);
+ let interval = self.interval;
+ let handle = std::thread::Builder::new()
+ .name("autoindex-progress".into())
+ .spawn(move || loop {
+ let stopped = progress.stopped.lock().unwrap();
+ let (stopped, _timeout) = progress
+ .wake
+ .wait_timeout(stopped, interval)
+ .expect("progress ticker mutex poisoned");
+ if *stopped {
+ break;
+ }
+ drop(stopped);
+ progress.emit();
+ })
+ .ok();
+ Ticker {
+ progress: Arc::clone(self),
+ handle,
+ }
+ }
+
+ fn stop(&self) {
+ *self.stopped.lock().unwrap() = true;
+ self.wake.notify_all();
+ }
+}
+
+/// Counts its file done on drop, on every path.
+pub struct FileGuard<'a> {
+ progress: &'a Progress,
+ seq: u64,
+ bytes: u64,
+}
+
+impl Drop for FileGuard<'_> {
+ fn drop(&mut self) {
+ self.progress.file_finished(self.seq, self.bytes);
+ }
+}
+
+/// Owns the ticker thread. Dropping it stops the thread and — if the run never
+/// called `finish` — closes the stream with an honest `ok=false` line.
+pub struct Ticker {
+ progress: Arc<Progress>,
+ handle: Option<std::thread::JoinHandle<()>>,
+}
+
+impl Drop for Ticker {
+ fn drop(&mut self) {
+ self.progress.stop();
+ if let Some(handle) = self.handle.take() {
+ let _ = handle.join();
+ }
+ self.progress.finish_aborted();
+ }
+}
+
+fn clear_tty(width: usize) -> String {
+ if width == 0 {
+ String::new()
+ } else {
+ format!("\r{}\r", " ".repeat(width))
+ }
+}
+
+fn columns() -> usize {
+ std::env::var("COLUMNS")
+ .ok()
+ .and_then(|value| value.parse::<usize>().ok())
+ .filter(|value| *value >= 20)
+ .unwrap_or(DEFAULT_COLUMNS)
+}
+
+fn truncate(text: &str, width: usize) -> String {
+ if text.chars().count() <= width {
+ return text.to_string();
+ }
+ text.chars().take(width.saturating_sub(1)).collect()
+}
+
+fn round1(value: f64) -> f64 {
+ (value * 10.0).round() / 10.0
+}
+
+fn fmt_bytes(bytes: u64) -> String {
+ const GB: u64 = 1 << 30;
+ const MB: u64 = 1 << 20;
+ const KB: u64 = 1 << 10;
+ if bytes >= GB {
+ format!("{:.1}GB", bytes as f64 / GB as f64)
+ } else if bytes >= MB {
+ format!("{:.1}MB", bytes as f64 / MB as f64)
+ } else if bytes >= KB {
+ format!("{:.1}KB", bytes as f64 / KB as f64)
+ } else {
+ format!("{bytes}B")
+ }
+}
+
+fn fmt_secs(seconds: f64) -> String {
+ let seconds = seconds.max(0.0);
+ if seconds < 60.0 {
+ format!("{seconds:.0}s")
+ } else if seconds < 3600.0 {
+ format!(
+ "{}m{:02}s",
+ (seconds / 60.0) as u64,
+ (seconds % 60.0) as u64
+ )
+ } else {
+ format!(
+ "{}h{:02}m",
+ (seconds / 3600.0) as u64,
+ ((seconds % 3600.0) / 60.0) as u64
+ )
+ }
+}
+
+fn render_tty(snapshot: &Snapshot) -> String {
+ let mut line = format!("{} ", snapshot.phase);
+ match snapshot.percent {
+ Some(percent) => line.push_str(&format!("{percent:5.1}% ")),
+ None => line.push_str(" ..% "),
+ }
+ if snapshot.items_total > 0 {
+ line.push_str(&format!(
+ "| {}/{} items ",
+ snapshot.items_done, snapshot.items_total
+ ));
+ } else if snapshot.items_done > 0 {
+ line.push_str(&format!("| {} items ", snapshot.items_done));
+ }
+ if snapshot.bytes_total > 0 {
+ line.push_str(&format!(
+ "| {}/{} ",
+ fmt_bytes(snapshot.bytes_done),
+ fmt_bytes(snapshot.bytes_total)
+ ));
+ }
+ if let Some(rate) = snapshot.rate {
+ match snapshot.basis {
+ Basis::Bytes => line.push_str(&format!("| {}/s ", fmt_bytes(rate as u64))),
+ Basis::Items => line.push_str(&format!("| {rate:.1} items/s ")),
+ Basis::None => {}
+ }
+ }
+ match snapshot.eta.seconds() {
+ Some(seconds) => line.push_str(&format!("| eta {}", fmt_secs(seconds))),
+ None => line.push_str("| eta unknown"),
+ }
+ if snapshot.waiting_on.len() == 1 {
+ line.push_str(&format!(
+ " | waiting on {}",
+ straggler(&snapshot.waiting_on[0])
+ ));
+ } else if snapshot.waiting_on.len() > 1 {
+ line.push_str(&format!(
+ " | waiting on {} items",
+ snapshot.waiting_on.len()
+ ));
+ }
+ line
+}
+
+/// `path(22.4MB)`, or just `path` for an item with no meaningful byte size
+/// (a catalog delete, a correlation query) — `(0B)` would read as a defect.
+fn straggler((name, bytes): &(String, u64)) -> String {
+ if *bytes > 0 {
+ format!("{name}({})", fmt_bytes(*bytes))
+ } else {
+ name.clone()
+ }
+}
+
+fn render_plain(snapshot: &Snapshot) -> String {
+ let percent = match snapshot.percent {
+ Some(percent) => format!("{percent:.1}"),
+ None => "unknown".to_string(),
+ };
+ let rate = match snapshot.rate {
+ Some(rate) => format!("{rate:.1}"),
+ None => "unknown".to_string(),
+ };
+ let eta = match snapshot.eta.seconds() {
+ Some(seconds) => format!("{seconds:.1}"),
+ None => "unknown".to_string(),
+ };
+ let mut line = format!(
+ "xerj-progress phase={} basis={} pct={} items={}/{} bytes={}/{} rate={} eta_s={} \
+ eta_quality={} since_progress_s={:.1} phase_elapsed_s={:.1} elapsed_s={:.1}",
+ snapshot.phase,
+ snapshot.basis.as_str(),
+ percent,
+ snapshot.items_done,
+ snapshot.items_total,
+ snapshot.bytes_done,
+ snapshot.bytes_total,
+ rate,
+ eta,
+ snapshot.eta.quality(),
+ snapshot.since_progress,
+ snapshot.phase_elapsed,
+ snapshot.elapsed,
+ );
+ if !snapshot.waiting_on.is_empty() {
+ let names: Vec<String> = snapshot.waiting_on.iter().map(straggler).collect();
+ line.push_str(&format!(" waiting_on={}", names.join(",")));
+ }
+ line
+}
+
+fn render_json(snapshot: &Snapshot) -> String {
+ let waiting: Vec<serde_json::Value> = snapshot
+ .waiting_on
+ .iter()
+ .map(|(rel, bytes)| serde_json::json!({"path": rel, "bytes": bytes}))
+ .collect();
+ serde_json::json!({
+ "event": "progress",
+ "phase": snapshot.phase,
+ "basis": snapshot.basis.as_str(),
+ "pct": snapshot.percent.map(round1),
+ "items_done": snapshot.items_done,
+ "items_total": snapshot.items_total,
+ "bytes_done": snapshot.bytes_done,
+ "bytes_total": snapshot.bytes_total,
+ "rate": snapshot.rate.map(round1),
+ "eta_s": snapshot.eta.seconds().map(round1),
+ "eta_quality": snapshot.eta.quality(),
+ "since_progress_s": round1(snapshot.since_progress),
+ "phase_elapsed_s": round1(snapshot.phase_elapsed),
+ "elapsed_s": round1(snapshot.elapsed),
+ "waiting_on": waiting,
+ })
+ .to_string()
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn captured(buffer: &Arc<Mutex<Vec<u8>>>) -> String {
+ String::from_utf8(buffer.lock().unwrap().clone()).unwrap()
+ }
+
+ #[test]
+ fn auto_is_a_terminal_surface_only_on_a_real_terminal() {
+ assert_eq!(
+ resolve(ProgressMode::Auto, true, Some("xterm"), false),
+ Surface::Tty
+ );
+ // The three ways `auto` must degrade to parseable lines.
+ assert_eq!(
+ resolve(ProgressMode::Auto, false, Some("xterm"), false),
+ Surface::Plain
+ );
+ assert_eq!(
+ resolve(ProgressMode::Auto, true, Some("dumb"), false),
+ Surface::Plain
+ );
+ assert_eq!(
+ resolve(ProgressMode::Auto, true, Some("xterm"), true),
+ Surface::Plain
+ );
+ // Explicit modes ignore the environment entirely.
+ assert_eq!(
+ resolve(ProgressMode::Json, true, Some("xterm"), false),
+ Surface::Json
+ );
+ assert_eq!(
+ resolve(ProgressMode::None, true, Some("xterm"), false),
+ Surface::Silent
+ );
+ }
+
+ #[test]
+ fn unknown_progress_mode_is_rejected_not_defaulted() {
+ assert!(ProgressMode::parse("tty").is_err());
+ assert!(ProgressMode::parse("").is_err());
+ assert_eq!(ProgressMode::parse("json").unwrap(), ProgressMode::Json);
+ }
+
+ /// The defect in #241: a blocked phase emitted nothing. Liveness must come
+ /// from the clock, not from item completions — a worker can sit inside one
+ /// 22 MB file for many seconds while every counter stands still.
+ ///
+ /// Deliberately written as "wait until it happens, fail on a deadline"
+ /// rather than "sleep N and count": a rate assertion measures the machine's
+ /// scheduler, not this code. An earlier `sleep(400ms)` + `ticks >= 4` form
+ /// of this test failed once at load average 357 with 2 lines — where the
+ /// ticker had in fact fired, just twice instead of ten times. The property
+ /// under test is *repeated emission with nothing completing*, so three
+ /// lines (one from `phase`, two from independent ticker wakeups) prove it
+ /// and no upper bound on the machine's slowness can make that false.
+ #[test]
+ fn ticker_keeps_emitting_while_no_item_completes() {
+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_millis(40));
+ let ticker = progress.spawn_ticker();
+ progress.phase("finalize", 135, 0);
+
+ let deadline = Instant::now() + Duration::from_secs(20);
+ let text = loop {
+ let text = captured(&buffer);
+ let ticks = text
+ .lines()
+ .filter(|line| line.starts_with("xerj-progress phase=finalize"))
+ .count();
+ if ticks >= 3 {
+ break text;
+ }
+ assert!(
+ Instant::now() < deadline,
+ "a phase that completes no items must still prove liveness; \
+ only {ticks} line(s) in 20s:\n{text}"
+ );
+ std::thread::sleep(Duration::from_millis(20));
+ };
+ drop(ticker);
+ assert!(text.contains("items=0/135"), "{text}");
+ }
+
+ #[test]
+ fn percent_is_bytes_based_when_bytes_are_known() {
+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));
+ progress.phase("index", 4, 1000);
+ // Three of four files done, but only 10% of the bytes: a file-count
+ // percent would claim 75% here. The measured corpus had 40.4% of all
+ // bytes in ONE file, which is exactly this shape.
+ for _ in 0..3 {
+ progress.item_done(33);
+ }
+ progress.tick();
+ let text = captured(&buffer);
+ let last = text.lines().last().unwrap();
+ assert!(last.contains("basis=bytes"), "{last}");
+ assert!(last.contains("pct=9.9"), "{last}");
+ assert!(last.contains("items=3/4"), "{last}");
+ }
+
+ #[test]
+ fn percent_is_unknown_when_no_denominator_exists() {
+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));
+ progress.phase("walk", 0, 0);
+ progress.tick();
+ let text = captured(&buffer);
+ assert!(text.contains("pct=unknown"), "{text}");
+ assert!(text.contains("basis=none"), "{text}");
+ assert!(text.contains("eta_s=unknown"), "{text}");
+ }
+
+ #[test]
+ fn eta_stays_unknown_until_the_quality_gate_is_met() {
+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));
+ progress.phase("index", 100, 1000);
+ // Well past 2% of the bytes, but nowhere near 5s into the phase.
+ progress.item_done(500);
+ progress.tick();
+ let text = captured(&buffer);
+ assert!(
+ text.lines().last().unwrap().contains("eta_s=unknown"),
+ "an ETA from a sub-5s sample is not one we can stand behind: {text}"
+ );
+ assert!(text.contains("eta_quality=unknown"), "{text}");
+ }
+
+ #[test]
+ fn stragglers_are_named_when_the_tail_goes_quiet() {
+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));
+ progress.phase("index", 3, 30_000_000);
+ let guard = progress.file("tantivy/benches/hdfs.json", 23_488_102);
+ progress.tick();
+ let text = captured(&buffer);
+ assert!(
+ text.contains("waiting_on=tantivy/benches/hdfs.json(22.4MB)"),
+ "{text}"
+ );
+ drop(guard);
+ assert_eq!(progress.items_done.load(Ordering::Relaxed), 1);
+ assert_eq!(progress.bytes_done.load(Ordering::Relaxed), 23_488_102);
+ }
+
+ #[test]
+ fn json_surface_emits_one_object_per_line_with_nulls_for_unknowns() {
+ let (progress, buffer) = Progress::capture(Surface::Json, Duration::from_secs(3600));
+ progress.phase("walk", 0, 0);
+ progress.note("resuming from journal");
+ progress.finish(true, 3, "completed-with-junk", &[("files", 1922)]);
+ let text = captured(&buffer);
+ for line in text.lines() {
+ let value: serde_json::Value =
+ serde_json::from_str(line).unwrap_or_else(|e| panic!("{line}: {e}"));
+ assert!(value.get("event").is_some(), "{line}");
+ }
+ let first: serde_json::Value = serde_json::from_str(text.lines().next().unwrap()).unwrap();
+ assert!(first["pct"].is_null(), "{text}");
+ assert!(first["eta_s"].is_null(), "{text}");
+ let last: serde_json::Value = serde_json::from_str(text.lines().last().unwrap()).unwrap();
+ assert_eq!(last["event"], "done");
+ assert_eq!(last["exit"], 3);
+ assert_eq!(last["reason"], "completed-with-junk");
+ assert_eq!(last["files"], 1922);
+ }
+
+ /// Exit 3 is success. The stream must say so in words, because an agent
+ /// reading a bare `3` reads failure.
+ #[test]
+ fn terminal_line_states_the_outcome_in_words() {
+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));
+ progress.finish(
+ true,
+ 3,
+ "completed-with-junk",
+ &[("files", 12), ("records", 7)],
+ );
+ let text = captured(&buffer);
+ assert!(
+ text.starts_with("xerj-done ok=true exit=3 reason=completed-with-junk"),
+ "{text}"
+ );
+ assert!(text.contains("files=12 records=7"), "{text}");
+ }
+
+ #[test]
+ fn a_run_that_dies_still_closes_the_stream() {
+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_millis(20));
+ {
+ let _ticker = progress.spawn_ticker();
+ progress.phase("index", 10, 100);
+ }
+ let text = captured(&buffer);
+ assert!(
+ text.contains("xerj-done ok=false exit=1 reason=aborted"),
+ "{text}"
+ );
+ }
+
+ #[test]
+ fn silent_surface_writes_nothing_at_all() {
+ let (progress, buffer) = Progress::capture(Surface::Silent, Duration::from_millis(10));
+ let ticker = progress.spawn_ticker();
+ progress.phase("index", 10, 100);
+ progress.note("hello");
+ progress.item_done(10);
+ progress.tick();
+ progress.finish(true, 0, "completed", &[]);
+ drop(ticker);
+ assert!(captured(&buffer).is_empty());
+ }
+
+ #[test]
+ fn tty_surface_redraws_in_place_and_never_leaves_a_stale_tail() {
+ let (progress, buffer) = Progress::capture(Surface::Tty, Duration::from_secs(3600));
+ progress.phase("index", 1000, 1_000_000);
+ progress.item_done(500_000);
+ progress.tick();
+ progress.note("graph: 12 structural edges");
+ let text = captured(&buffer);
+ assert!(
+ text.contains('\r'),
+ "a tty surface redraws in place: {text:?}"
+ );
+ assert!(
+ text.ends_with("graph: 12 structural edges\n"),
+ "a note must land on its own line: {text:?}"
+ );
+ }
+
+ /// Caught on a real 185 s run, not in review: with 269 of 270 files done
+ /// and one 20 MB file still streaming, bytes stop being credited, so
+ /// folding those empty samples into the average sent the ETA from 10 s to
+ /// 471 s and climbing while roughly 50 s of work remained. An ETA that
+ /// runs away exactly when the user is watching is worse than no ETA, so
+ /// the estimate is withdrawn and the straggler named instead.
+ #[test]
+ fn a_stalled_phase_withdraws_its_eta_instead_of_inventing_one() {
+ let (progress, buffer) = Progress::capture(Surface::Plain, Duration::from_secs(1));
+ progress.phase("index", 270, 31_130_222);
+ let big = progress.file("bench/hdfs.jsonl", 20_881_790);
+ progress.bytes_done.store(10_248_432, Ordering::Relaxed);
+ progress.items_done.store(269, Ordering::Relaxed);
+ {
+ let mut state = progress.state.lock().unwrap();
+ state.phase_started = Instant::now() - Duration::from_secs(120);
+ state.last_sample_at = Instant::now() - Duration::from_secs(100);
+ state.last_advance_at = Instant::now() - Duration::from_secs(100);
+ state.last_sample_units = 10_248_432;
+ state.rate = Some(2_545_880.0);
+ state.shown_eta = Some(10.3);
+ }
+ progress.tick();
+ let line = captured(&buffer).lines().last().unwrap().to_string();
+ assert!(
+ line.contains("eta_s=unknown") && line.contains("eta_quality=stalled"),
+ "no completion in 100s is an absence of measurement, not a slow rate: {line}"
+ );
+ assert!(line.contains("since_progress_s=100."), "{line}");
+ assert!(
+ line.contains("waiting_on=bench/hdfs.jsonl(19.9MB)"),
+ "name what it is actually waiting on: {line}"
+ );
+ // ...and the percent stays honest: 99.6% of the FILES are done, but
+ // only a third of the bytes are.
+ assert!(line.contains("pct=32.9"), "{line}");
+ drop(big);
+ }
+
+ /// An empty sample must not drag the measured rate down either — the rate
+ /// is a statement about the work that actually completed.
+ #[test]
+ fn empty_samples_do_not_decay_the_measured_rate() {
+ let (progress, _buffer) = Progress::capture(Surface::Plain, Duration::from_secs(1));
+ progress.phase("index", 10, 1000);
+ progress.item_done(500);
+ {
+ let mut state = progress.state.lock().unwrap();
+ state.phase_started = Instant::now() - Duration::from_secs(10);
+ state.last_sample_at = Instant::now() - Duration::from_secs(1);
+ }
+ let first = progress.snapshot().rate.expect("a sample was taken");
+ for _ in 0..20 {
+ assert_eq!(progress.snapshot().rate, Some(first));
+ }
+ }
+
+ #[test]
+ fn displayed_eta_cannot_jump_more_than_the_clamp_per_tick() {
+ let (progress, _buffer) = Progress::capture(Surface::Plain, Duration::from_secs(3600));
+ {
+ let mut state = progress.state.lock().unwrap();
+ state.phase = "index";
+ state.items_total = 100;
+ state.bytes_total = 1000;
+ state.phase_started = Instant::now() - Duration::from_secs(30);
+ state.last_sample_at = Instant::now() - Duration::from_secs(1);
+ state.rate = Some(10.0);
+ state.shown_eta = Some(100.0);
+ }
+ progress.bytes_done.store(500, Ordering::Relaxed);
+ let snapshot = progress.snapshot();
+ let eta = snapshot.eta.seconds().expect("gate is satisfied");
+ assert!(
+ (80.0..=120.0).contains(&eta),
+ "a displayed ETA may move at most 20% per tick, got {eta}"
+ );
+ }
+}
diff --git a/engine/crates/xerj-server/src/brain.rs b/engine/crates/xerj-server/src/brain.rs
index 5296a227f..3ab287bf8 100644
--- a/engine/crates/xerj-server/src/brain.rs
+++ b/engine/crates/xerj-server/src/brain.rs
@@ -443,6 +443,10 @@ fn index_cfg(cfg: &BrainCfg, brain: &str, api_key: Option<String>) -> IndexCfg {
dry_run: false,
json: false,
quiet: false,
+ // `xerj brain` is a foreground command a human watches; it gets the
+ // same auto-resolved progress surface as `xerj autoindex` (#241).
+ progress: xerj_autoindex::progress::ProgressMode::Auto,
+ progress_interval: None,
}
}
diff --git a/landing/llms.txt b/landing/llms.txt
index 443ca92f8..4e98a31a4 100644
--- a/landing/llms.txt
+++ b/landing/llms.txt
@@ -76,7 +76,11 @@ If you are an AI coding agent driving XERJ on a person's machine, indexing is th
* **Fast** — `--no-semantic --no-graph`: skips embedding-backed body fields and relationship detection. You still get typed, queryable BM25 + keyword indices over the same files — a better-than-grep experience for a fraction of the time. (Note: XERJ's default embedder is lexical feature-hashing, not neural, so `--no-semantic` costs you less than the word "semantic" suggests unless the node runs `--embed-mode neural`.)
* **Narrower** — index the subdirectory that matters, or skip `node_modules`/`vendor`/`target`/build outputs. Usually the biggest win available.
* Also useful when their machine must stay usable: `--workers N` and `--pdf-workers N` (which accepts 1–4 only). They bound the **client-side extractor**, not the server — indexing itself runs in the node process, so do not promise a user these cap the machine's total CPU.
-- **Never run a long index as one silent blocking call.** Start it in the background with an explicit `--state-dir <dir>`, then poll `xerj autoindex status --state-dir <dir>` with the *same* directory — it prints `N files done, M records, in progress` (or `FINISHED`) plus live per-index doc counts — and relay that to your user periodically. Passing no `--state-dir` to `status` prints one line per journal under `~/.xerj/autoindex`, including unrelated runs from weeks ago, which is how an agent ends up reporting someone else's progress. `status` has **no** `--json` (only `map` does); parse its text, or poll `GET /_cat/indices` / `GET /ax-*/_count` for machine-readable counts.
+- **Read the progress stream — it is designed for you.** `autoindex` writes its RESULT to stdout and its LIVENESS to stderr, so `--json` stdout stays one clean parseable document while stderr narrates. When stderr is not a terminal (you), you get one line per interval, by default every 5 s: `xerj-progress phase=index basis=bytes pct=1.9 items=250/251 bytes=693420/37004502 rate=152775.9 eta_s=unknown eta_quality=unknown since_progress_s=5.0 phase_elapsed_s=9.5 elapsed_s=10.0 waiting_on=big-events.jsonl(34.6MB)` (a real line from the run that verified this). Use `--progress json` for one JSON object per line, `--progress-interval SECS` to change the cadence, `--progress none` (or `--quiet`) to silence it. Relay `pct`, `eta_s` and `waiting_on` to your user as they arrive.
+ * **`pct` and `eta_s` are honest or absent.** Both can be the literal string `unknown` (JSON `null`) — before a denominator exists, before the estimate has settled, or when nothing has completed recently (`eta_quality=stalled`, meaning one big file is still streaming). Never invent a number to fill the gap; report `waiting_on` instead, which names the file it is actually on.
+ * **Percent is bytes-based, not file-based.** `items=250/251` with `pct=1.9` is normal and correct: in that measured corpus one 34.6 MB file held 98% of the bytes, so 250 of 251 files done really was 1.9% of the work. A file-count percent would have claimed 99.6% and then sat there.
+ * **Every run that reaches an exit ends with one terminal line, in every mode** — `xerj-done ok=true exit=3 reason=completed-with-junk wall=57.6s files=1922 records=115139`, and `ok=false … reason=aborted` on the error path. Wait for it rather than inferring completion from silence. A run killed by a signal cannot print one, so a process that is gone with no terminal line died; it did not finish.
+- **You can still poll from outside.** Start the run in the background with an explicit `--state-dir <dir>`, then poll `xerj autoindex status --state-dir <dir>` with the *same* directory — it prints `N files done, M records, in progress` (or `FINISHED`) plus live per-index doc counts. Passing no `--state-dir` to `status` prints one line per journal under `~/.xerj/autoindex`, including unrelated runs from weeks ago, which is how an agent ends up reporting someone else's progress. `status` has **no** `--json` (only `map` does); parse its text, or poll `GET /_cat/indices` / `GET /ax-*/_count` for machine-readable counts.
- **Stopping is safe, so offer it.** The resume journal plus idempotent `_id`s mean an interrupted run resumes where it stopped and never duplicates. A user may abort a long index without losing the work already done.
- **Read the exit code correctly.** `0` complete · **`3` completed-with-junk — this is SUCCESS**, junk records are recorded and never fatal; do not report it as a failure · `2` usage error · `1` **any error at all** — it is the catch-all for every failure, so read the `error:` line on stderr before you act. Endpoint-unreachable is only the most common one; a bulk failure, an index-creation conflict, a journal-config mismatch, a state-dir lock conflict, corruption or a disk error all exit 1 too, and restarting the node "because 1 means it is down" will not fix any of them.
- **Close the loop.** When it finishes, tell the user the actual elapsed time, how many records landed, and what they can now ask — `xerj autoindex map` is the briefing to summarize from.