AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(autoindex): never wait on stdin for a gate question --quiet silenced
xerj-orge6a01beef4ae
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex 9d3e0c54f..e5217dbe1 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -157,7 +157,12 @@ pub fn help_text() -> String {progress cadence, 1..=3600 (default 1 on a terminal,\n\5 otherwise). This is the guaranteed upper bound on\n\silence between phases.\n\- --quiet errors only (implies --progress none)\n\+ --quiet errors only (implies --progress none). The decision gate\n\+ NEVER prompts under this flag, even at a terminal — the\n\+ question would be silenced with everything else. It emits\n\+ the JSON decision request on stdout and exits 4 instead,\n\+ exactly like an agent-driven run. See ESTIMATE + DECISION\n\+ GATE.\n\--dataset <SLUG> (map) show a single dataset\n\--help, -h this help\n\\n\@@ -237,8 +242,15 @@ pub fn help_text() -> String {given, nothing is indexed: a JSON decision request goes to stdout and the\n\process exits 4 (a code of its own — 1 is the catch-all for real failures).\n\Answer by re-running the same command with --approve proceed|fast|cancel.\n\- A person at a terminal is prompted instead; a piped or agent-driven run is\n\- never blocked on stdin.\n\+ A person at a terminal is prompted instead — but ONLY when the question can\n\+ actually be seen. All three must hold: stdin is a terminal, stderr is a\n\+ terminal, and the progress surface is on. --quiet / --progress none silences\n\+ the question, so those runs are never prompted and never wait on stdin; a\n\+ piped or agent-driven run is not prompted either. Every un-prompted run\n\+ behaves identically: the JSON decision request goes to stdout (which --quiet\n\+ does NOT silence) and the process exits 4. The payload's\n\+ `prompt_not_offered_because` says which of the three was missing.\n\+ autoindex never waits on stdin for a question it did not print.\n\\n\WORK ORDER:\n\Phase B drains source and documents first, then configuration, then\n\@@ -938,6 +950,11 @@ mod tests {"WORK ORDER:","vendored/generated/minified","CLIENT-SIDE EXTRACTION only",+ // A run that silently stops asking has to say so where the user+ // looks: --quiet is the flag, the gate section is the rule.+ "NEVER prompts under this flag",+ "never waits on stdin for a question it did not print",+ "prompt_not_offered_because",] {assert!(help.contains(expected), "help is missing {expected:?}");}diff --git a/engine/crates/xerj-autoindex/src/gate.rs b/engine/crates/xerj-autoindex/src/gate.rsindex a76c79d82..10136a4ec 100644--- a/engine/crates/xerj-autoindex/src/gate.rs+++ b/engine/crates/xerj-autoindex/src/gate.rs@@ -14,11 +14,16 @@//! only see 1 would have to parse prose to tell "your endpoint is down" from//! "I need you to choose". Anything that has to be distinguished//! programmatically gets its own code.-//! * **Never block on stdin.** The prompt is offered only when stdin is a-//! terminal — i.e. only when a person is actually there to type. A run-//! started by an agent, a CI job, or a pipe emits the payload and exits;-//! waiting for an answer that cannot arrive is how an agent deadlocks, and-//! it is a failure mode nobody can see from the outside.+//! * **Never wait on stdin for a question the user cannot see.** The prompt is+//! offered only when someone can answer it (stdin is a terminal) *and* the+//! question actually reached a screen (stderr is a terminal, and the progress+//! surface is not silenced). A run started by an agent, a CI job, a pipe, or+//! with `--quiet` emits the payload and exits; waiting for an answer that+//! cannot arrive is how an agent deadlocks, and an invisible prompt is+//! indistinguishable from a hang — the worst possible failure for a feature+//! whose whole purpose is to say "this is going to take a while".+//! [`prompt_blocked_by`] owns that decision and [`answer_from_terminal`]+//! never opens stdin when it says no.//! * **Options carry their measured cost, or say they were not measured.**//! `narrower` names the heaviest directories in this corpus with real byte//! counts and re-costs the run without them using the same measured rates.@@ -172,6 +177,11 @@ pub struct DecisionRequest<'a> {pub total_datasets: usize,/// Files relationship detection would run over under the current flags.pub graph_files: u64,+ /// Why this run did not stop to ask at a terminal, when it did not.+ /// Carried in the payload because stdout is the one channel `--quiet`+ /// leaves open: a user who silenced stderr still has to be able to find+ /// out that a question existed and why they were not asked it.+ pub prompt_blocked: Option<NoPrompt>,}impl DecisionRequest<'_> {@@ -189,6 +199,8 @@ impl DecisionRequest<'_> {"priority_order": self.bands,"heaviest_directories": self.heaviest,"options": self.options(),+ "prompt_offered": self.prompt_blocked.is_none(),+ "prompt_not_offered_because": self.prompt_blocked.map(NoPrompt::explain),"how_to_answer":"re-invoke the identical command with --approve <id> (--yes is an alias for \--approve proceed). Without an answer nothing has been indexed and nothing has \@@ -344,17 +356,120 @@ impl DecisionRequest<'_> {proceed). Nothing has been indexed.".to_string(),);+ // Say why nobody was asked, where that can still be read. A log that+ // shows the question but not the reason it went unasked is the same+ // guessing game one layer up.+ if let Some(blocked) = self.prompt_blocked {+ lines.push(format!(" {}", blocked.explain()));+ }lines}}-/// Is there a human who can answer right now?+/// Why no terminal prompt was offered. `None` from [`prompt_blocked_by`] means+/// one was.///-/// Both halves matter: stdin is where an answer would come from, stderr is-/// where the question is printed. A run with a redirected stdin has nobody to-/// ask, and blocking on it is the deadlock this gate exists to avoid.-pub fn can_prompt() -> bool {- std::io::stdin().is_terminal() && std::io::stderr().is_terminal()+/// This exists as a value rather than a bare `bool` because the reason is+/// printed: a run that stops without asking has to be able to say why it did+/// not ask, or the silence is just another thing for the user to guess at.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum NoPrompt {+ /// stdin is a pipe, a file, or closed — an agent, a CI job, a script.+ /// Nobody is at the keyboard, so there is no answer to wait for.+ StdinNotTerminal,+ /// stderr is redirected. The question would land in a log nobody is+ /// watching while the process sat waiting for a reply to it.+ StderrNotTerminal,+ /// `--quiet` / `--progress none` silenced the surface the question is+ /// printed on. Someone *is* at the keyboard, but they were never shown+ /// anything to answer — this is the case that made the run look hung.+ QuestionSilenced,+}++impl NoPrompt {+ /// One sentence, for the decision payload and the prose.+ pub fn explain(self) -> &'static str {+ match self {+ NoPrompt::StdinNotTerminal => {+ "no terminal prompt: stdin is not a terminal, so no answer could be typed. Re-run \+ with --approve proceed|fast|cancel"+ }+ NoPrompt::StderrNotTerminal => {+ "no terminal prompt: stderr is not a terminal, so the question would not have \+ reached a screen. Re-run with --approve proceed|fast|cancel"+ }+ NoPrompt::QuestionSilenced => {+ "no terminal prompt: --quiet / --progress none silences the question, and this run \+ will never wait on stdin for something it did not print. The decision request \+ below is on stdout, which --quiet does not silence. Re-run with --approve \+ proceed|fast|cancel, or without --quiet to be asked"+ }+ }+ }+}++/// Should this run stop to ask, and if not, why not?+///+/// Three conditions, and all three have to hold before autoindex touches+/// stdin:+///+/// * `stdin_tty` — someone can type an answer;+/// * `stderr_tty` — the question reaches a screen rather than a log file;+/// * `question_visible` — the progress surface actually printed it. This is+/// the one the first cut of the gate missed. `--quiet` / `--progress none`+/// routes [`crate::progress::Progress::note`] to nothing, so the prose and+/// the prompt vanished while the process still sat on `read_line`. Measured+/// before the fix: a pty run with `--progress none` printed **zero bytes**+/// and was still blocked at 220 s. An invisible prompt is a hang.+///+/// Pure, so the whole rule is testable without a terminal — the same shape as+/// [`crate::progress::resolve`].+pub fn prompt_blocked_by(+ stdin_tty: bool,+ stderr_tty: bool,+ question_visible: bool,+) -> Option<NoPrompt> {+ if !stdin_tty {+ return Some(NoPrompt::StdinNotTerminal);+ }+ if !stderr_tty {+ return Some(NoPrompt::StderrNotTerminal);+ }+ if !question_visible {+ return Some(NoPrompt::QuestionSilenced);+ }+ None+}++/// [`prompt_blocked_by`] against this process's real streams.+///+/// `question_visible` is the caller's progress surface+/// ([`crate::progress::Progress::enabled`]) — gate has no business inspecting+/// the surface itself, and passing the fact in keeps this testable.+pub fn detect_prompt_block(question_visible: bool) -> Option<NoPrompt> {+ prompt_blocked_by(+ std::io::stdin().is_terminal(),+ std::io::stderr().is_terminal(),+ question_visible,+ )+}++/// Ask, but only when [`prompt_blocked_by`] said it was safe to.+///+/// `open_stdin` is a closure and not a reader on purpose: when the prompt is+/// blocked it is **never called**, so there is no path on which this function+/// can end up holding — let alone reading — the process's stdin. That is the+/// invariant, and `a_question_that_cannot_be_seen_is_never_asked` asserts it by+/// handing in a reader that panics if anyone touches it.+pub fn answer_from_terminal<R: std::io::BufRead>(+ blocked: Option<NoPrompt>,+ open_stdin: impl FnOnce() -> R,+ echo: impl FnMut(&str),+) -> Option<Approval> {+ if blocked.is_some() {+ return None;+ }+ read_answer(&mut open_stdin(), echo)}/// Read one answer from a terminal. `None` means "no answer" — EOF, an@@ -525,6 +640,7 @@ mod tests {semantic_datasets: vec!["docs".into()],total_datasets: 3,graph_files: 10,+ prompt_blocked: None,};let payload = request.to_json();assert_eq!(payload["xerj"], "autoindex-decision-request");@@ -671,6 +787,129 @@ mod tests {);}+ /// A reader that fails the test if anything reads it. Standing in for the+ /// process's real stdin, which has no EOF and no answer coming.+ struct NeverRead;++ impl std::io::Read for NeverRead {+ fn read(&mut self, _: &mut [u8]) -> std::io::Result<usize> {+ panic!("stdin was read for a question the user was never shown");+ }+ }++ impl std::io::BufRead for NeverRead {+ fn fill_buf(&mut self) -> std::io::Result<&[u8]> {+ panic!("stdin was read for a question the user was never shown");+ }+ fn consume(&mut self, _: usize) {}+ }++ /// THE regression test.+ ///+ /// `--quiet` / `--progress none` silences `Progress::note`, which is where+ /// both the gate's prose and its prompt are printed — but the first cut of+ /// the gate decided whether to ask from the two tty checks alone. So a+ /// person at a terminal who passed `--quiet` was asked a question that was+ /// never printed, and the process sat on `read_line` forever. Measured on+ /// the pre-fix binary: a pty run with `--progress none` over the threshold+ /// emitted **0 bytes** and was still blocked when it was killed at 220 s.+ /// From the outside that is indistinguishable from a hang, in the one+ /// feature whose entire job is to say "this will take a while".+ ///+ /// The reader panics if it is touched, so the invariant asserted here is+ /// the real one — not "an answer was not returned" but "stdin was not+ /// read".+ #[test]+ fn a_question_that_cannot_be_seen_is_never_asked() {+ // Someone IS at the keyboard and both streams are terminals — the one+ // thing missing is that the surface printed nothing.+ let blocked = prompt_blocked_by(true, true, false);+ assert_eq!(+ blocked,+ Some(NoPrompt::QuestionSilenced),+ "a silenced surface must block the prompt on its own"+ );+ let mut echoed = Vec::new();+ // Panics unless the fix is present: pre-fix this opened and read stdin.+ let answer =+ answer_from_terminal(blocked, || NeverRead, |line| echoed.push(line.to_string()));+ assert_eq!(answer, None, "an unaskable question is never consent");+ assert!(+ echoed.is_empty(),+ "nothing may be echoed to a surface that prints nothing: {echoed:?}"+ );++ // …and the run still says so where it can be read. stdout is not+ // silenced by --quiet, so the payload has to carry the reason.+ let estimate = estimate(10, 1_000);+ let request = DecisionRequest {+ root: "/data".into(),+ estimate: &estimate,+ max_minutes: 1,+ bands: Vec::new(),+ heaviest: Vec::new(),+ without_generated: None,+ semantic_datasets: Vec::new(),+ total_datasets: 1,+ graph_files: 0,+ prompt_blocked: blocked,+ };+ let payload = request.to_json();+ assert_eq!(payload["prompt_offered"], false);+ let why = payload["prompt_not_offered_because"].as_str().unwrap();+ assert!(why.contains("--quiet"), "{why}");+ assert!(why.contains("stdout"), "{why}");+ assert!(why.contains("--approve"), "{why}");+ // The exit code is unchanged: a quiet run behaves exactly like the+ // agent-driven one it now matches.+ assert_eq!(payload["exit_code"], EXIT_NEEDS_DECISION);+ }++ /// The other three ways in, so the rule is pinned end to end rather than+ /// only at the case that was broken.+ #[test]+ fn the_prompt_is_offered_only_when_all_three_conditions_hold() {+ // A terminal, a visible surface: this is the one case that asks.+ assert_eq!(prompt_blocked_by(true, true, true), None);+ // Agent, CI, pipe: nobody to type. Checked first because it is the+ // reason that stays true however the surface is configured.+ assert_eq!(+ prompt_blocked_by(false, true, true),+ Some(NoPrompt::StdinNotTerminal)+ );+ assert_eq!(+ prompt_blocked_by(false, false, false),+ Some(NoPrompt::StdinNotTerminal)+ );+ // stderr redirected to a log: the question would not reach a screen.+ assert_eq!(+ prompt_blocked_by(true, false, true),+ Some(NoPrompt::StderrNotTerminal)+ );+ // Every reason names a way out, and none of them is "wait".+ for reason in [+ NoPrompt::StdinNotTerminal,+ NoPrompt::StderrNotTerminal,+ NoPrompt::QuestionSilenced,+ ] {+ let text = reason.explain();+ assert!(text.contains("--approve"), "{text}");+ // Every blocked path must leave stdin alone, not merely decline to+ // return an answer.+ assert_eq!(+ answer_from_terminal(Some(reason), || NeverRead, |_| {}),+ None+ );+ }++ // A visible terminal really does read the answer — the fix must not+ // have turned the prompt off for everyone.+ assert_eq!(+ answer_from_terminal(None, || std::io::Cursor::new(b"f\n"), |_| {}),+ Some(Approval::Fast)+ );+ }+#[test]fn every_spelling_of_an_answer_maps_to_one_decision() {for (input, expected) in [diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 9c8905b4b..f98d8f654 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -2947,6 +2947,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {.filter(|dataset| dataset.semantic_field.is_some()).map(|dataset| dataset.slug.clone()).collect();+ // Decided once, before anything is printed, and used for both halves:+ // whether to touch stdin at all, and what the payload says about why+ // nobody was asked. `pr.enabled()` is the missing third condition —+ // `--quiet` / `--progress none` routes every `pr.note` below to+ // nothing, so without it the gate printed no question and then waited+ // on stdin for the answer to it.+ let prompt_blocked = gate::detect_prompt_block(pr.enabled());let request = gate::DecisionRequest {root: root_str.clone(),estimate: &run_estimate,@@ -2966,6 +2973,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {} else {pending.len() as u64},+ prompt_blocked,};match cfg.approve {Some(answer) => pr.note(&format!(@@ -2982,16 +2990,18 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {for line in request.prose() {pr.note(&line);}- // Ask only where an answer can arrive. A pipe, a CI job or an- // agent has no one at the keyboard, and blocking on stdin- // there is an invisible deadlock — the failure this gate was- // added to prevent, not to introduce.- let answer = if gate::can_prompt() {- let stdin = std::io::stdin();- gate::read_answer(&mut stdin.lock(), |line| pr.note(line))- } else {- None- };+ // Ask only where an answer can arrive AND the question was+ // actually shown. A pipe, a CI job or an agent has no one at+ // the keyboard; a quiet run has someone at the keyboard who+ // was shown nothing. Blocking on stdin in either case is an+ // invisible deadlock — the failure this gate was added to+ // prevent, not to introduce. When it is blocked, stdin is not+ // opened at all.+ let answer = gate::answer_from_terminal(+ prompt_blocked,+ || std::io::stdin().lock(),+ |line| pr.note(line),+ );match answer {Some(gate::Approval::Proceed) => pr.note("gate: proceeding"),Some(gate::Approval::Fast) => {diff --git a/landing/llms.txt b/landing/llms.txtindex b7a44a813..63bd93d49 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -91,6 +91,7 @@ If you are an AI coding agent driving XERJ on a person's machine, indexing is th{"id":"fast","invoke":"--approve fast","effect":"adds --no-semantic --no-graph: …","cost":"speed-up NOT measured by this run. …"},{"id":"narrower","invoke":"re-run as `xerj autoindex <subdirectory>` …","effect":"index only the part that matters","cost":"…"},{"id":"cancel","invoke":"--approve cancel","effect":"index nothing, exit 0","cost":"none — nothing has been written yet"}],+ "prompt_offered":false,"prompt_not_offered_because":"no terminal prompt: stdin is not a terminal …","how_to_answer":"re-invoke the identical command with --approve <id> …"}```@@ -100,7 +101,7 @@ Two honesty notes the payload states about itself and you should pass on: the es- **Plan before you index.** `xerj autoindex <folder> --dry-run` walks, sniffs and samples every file, prints the plan, and indexes nothing. The line to report is the job size — `autoindex: 40 files (0 MB) under /path` — followed by the datasets it would create and the junk/skipped counts. Do not read it as the *first* line: a `bulk HTTP request timeout: Ns` line is printed before it, and both go to stderr. It needs a running node. Report the job-size line to your user before you commit their machine to anything. A dry run is read-only for the *index*, not for the disk: it creates the run's state dir, takes an exclusive lock on it and appends a journal record — so it fails if a real run of the same folder is already live, and a dry run before the first real run makes that run announce "resuming from journal".- **Estimate honestly, from two signals.** (a) The dry run performs the full phase-A pass **only when no plan is frozen yet**; if the journal already holds one — because an earlier run was interrupted — it reuses that plan, returns in under a second and prints the *old* numbers. Treat a suspiciously fast dry run as "no new estimate", and point `--state-dir` at a fresh directory if you need a real one. (b) Recorded reference throughput on one machine: 518 MB / 1,995 files → ~38–51 s; 33.7k records/s on 923 MB (server-bound). Hardware varies widely — Apple Silicon laptops, cold caches, and PDF-heavy or SQL-dump-heavy corpora are much slower per byte than source code. Give the user a range and say it is an estimate; never state a confident single number you have not measured on their machine.-- **The binary now enforces the ask — you no longer have to remember it.** Since v1.0.0-rc.15, `autoindex` measures per-format throughput **on the user's own machine** during phase A (it is already parsing every file to sniff it) and prints a range with its basis before phase B starts: `estimate: at least 64.2 s–64.2 s — a MEASURED FLOOR for client-side extraction…`, then one line per family (`code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s`). If the upper end exceeds `--max-minutes` (**default 10**) and you passed no `--approve`, it **indexes nothing**, writes a JSON decision request to stdout and exits **4** — a code of its own, so "I need a decision" is never confused with exit 1's catch-all failure. Answer by re-running the identical command with `--approve proceed|fast|cancel` (`--yes` = proceed); `--max-minutes 0` disables the gate. A person at a terminal is prompted instead, and a piped run is never blocked on stdin.+- **The binary now enforces the ask — you no longer have to remember it.** Since v1.0.0-rc.15, `autoindex` measures per-format throughput **on the user's own machine** during phase A (it is already parsing every file to sniff it) and prints a range with its basis before phase B starts: `estimate: at least 64.2 s–64.2 s — a MEASURED FLOOR for client-side extraction…`, then one line per family (`code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s`). If the upper end exceeds `--max-minutes` (**default 10**) and you passed no `--approve`, it **indexes nothing**, writes a JSON decision request to stdout and exits **4** — a code of its own, so "I need a decision" is never confused with exit 1's catch-all failure. Answer by re-running the identical command with `--approve proceed|fast|cancel` (`--yes` = proceed); `--max-minutes 0` disables the gate. A person at a terminal is prompted instead — but only when the question is actually visible: stdin a terminal, stderr a terminal, and the progress surface on. `--quiet`/`--progress none` silences the question, so those runs are never prompted; they take the same path you do (payload on stdout, exit `4`), and `prompt_not_offered_because` says which condition was missing. **autoindex never waits on stdin for a question it did not print** — an invisible prompt is indistinguishable from a hang.* **The number is a floor, not a prediction, and it says so.** It covers client-side extraction only: server indexing, embedding, network and edge-writing are not measured before the run starts, because measuring them would mean writing to the index the estimate exists to ask permission for. Measured here: on a 68 MB source tree the floor was 0.1 s against a real 8.9 s run; on a 793 MB one it was 64.2 s against a real ~350 s run. **So the gate under-asks and never over-asks** — when it fires the run is certainly long, but silence is not a promise the run is short. Say that to your user rather than repeating the floor as an ETA.* **Families it could not time are named, not guessed at.** A family only gets a rate from files phase A provably read end to end; `sqlite` (row-capped) and gzipped files never qualify. Unpriced families appear under `unmeasured_families` with their byte counts, and `coverage` tells you what share of planned bytes the number actually covers. If nothing could be measured, there is no number and no gate — the `basis` field says so.* **The payload is built for you.** `{"xerj":"autoindex-decision-request","exit_code":4,"reason":…,"estimate":{…},"priority_order":[…],"heaviest_directories":[…],"options":[…]}` — `heaviest_directories` names real byte counts and flags the ones matching the vendored/generated rule, and the `narrower` option re-costs the run without them using the same measured rates. The `fast` option deliberately states **no speed-up factor**: it reports which datasets and how many files it changes and says the factor was not measured.