Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(autoindex): never wait on stdin for a gate question --quiet silenced

xerj-orge6a01beef4ae
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index 9d3e0c54f..e5217dbe1 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -157,7 +157,12 @@ pub fn help_text() -> String {
progress cadence, 1..=3600 (default 1 on a terminal,\n\
5 otherwise). This is the guaranteed upper bound on\n\
silence between phases.\n\
- --quiet errors only (implies --progress none)\n\
+ --quiet errors only (implies --progress none). The decision gate\n\
+ NEVER prompts under this flag, even at a terminal — the\n\
+ question would be silenced with everything else. It emits\n\
+ the JSON decision request on stdout and exits 4 instead,\n\
+ exactly like an agent-driven run. See ESTIMATE + DECISION\n\
+ GATE.\n\
--dataset <SLUG> (map) show a single dataset\n\
--help, -h this help\n\
\n\
@@ -237,8 +242,15 @@ pub fn help_text() -> String {
given, nothing is indexed: a JSON decision request goes to stdout and the\n\
process exits 4 (a code of its own — 1 is the catch-all for real failures).\n\
Answer by re-running the same command with --approve proceed|fast|cancel.\n\
- A person at a terminal is prompted instead; a piped or agent-driven run is\n\
- never blocked on stdin.\n\
+ A person at a terminal is prompted instead — but ONLY when the question can\n\
+ actually be seen. All three must hold: stdin is a terminal, stderr is a\n\
+ terminal, and the progress surface is on. --quiet / --progress none silences\n\
+ the question, so those runs are never prompted and never wait on stdin; a\n\
+ piped or agent-driven run is not prompted either. Every un-prompted run\n\
+ behaves identically: the JSON decision request goes to stdout (which --quiet\n\
+ does NOT silence) and the process exits 4. The payload's\n\
+ `prompt_not_offered_because` says which of the three was missing.\n\
+ autoindex never waits on stdin for a question it did not print.\n\
\n\
WORK ORDER:\n\
Phase B drains source and documents first, then configuration, then\n\
@@ -938,6 +950,11 @@ mod tests {
"WORK ORDER:",
"vendored/generated/minified",
"CLIENT-SIDE EXTRACTION only",
+ // A run that silently stops asking has to say so where the user
+ // looks: --quiet is the flag, the gate section is the rule.
+ "NEVER prompts under this flag",
+ "never waits on stdin for a question it did not print",
+ "prompt_not_offered_because",
] {
assert!(help.contains(expected), "help is missing {expected:?}");
}
diff --git a/engine/crates/xerj-autoindex/src/gate.rs b/engine/crates/xerj-autoindex/src/gate.rs
index a76c79d82..10136a4ec 100644
--- a/engine/crates/xerj-autoindex/src/gate.rs
+++ b/engine/crates/xerj-autoindex/src/gate.rs
@@ -14,11 +14,16 @@
//! only see 1 would have to parse prose to tell "your endpoint is down" from
//! "I need you to choose". Anything that has to be distinguished
//! programmatically gets its own code.
-//! * **Never block on stdin.** The prompt is offered only when stdin is a
-//! terminal — i.e. only when a person is actually there to type. A run
-//! started by an agent, a CI job, or a pipe emits the payload and exits;
-//! waiting for an answer that cannot arrive is how an agent deadlocks, and
-//! it is a failure mode nobody can see from the outside.
+//! * **Never wait on stdin for a question the user cannot see.** The prompt is
+//! offered only when someone can answer it (stdin is a terminal) *and* the
+//! question actually reached a screen (stderr is a terminal, and the progress
+//! surface is not silenced). A run started by an agent, a CI job, a pipe, or
+//! with `--quiet` emits the payload and exits; waiting for an answer that
+//! cannot arrive is how an agent deadlocks, and an invisible prompt is
+//! indistinguishable from a hang — the worst possible failure for a feature
+//! whose whole purpose is to say "this is going to take a while".
+//! [`prompt_blocked_by`] owns that decision and [`answer_from_terminal`]
+//! never opens stdin when it says no.
//! * **Options carry their measured cost, or say they were not measured.**
//! `narrower` names the heaviest directories in this corpus with real byte
//! counts and re-costs the run without them using the same measured rates.
@@ -172,6 +177,11 @@ pub struct DecisionRequest<'a> {
pub total_datasets: usize,
/// Files relationship detection would run over under the current flags.
pub graph_files: u64,
+ /// Why this run did not stop to ask at a terminal, when it did not.
+ /// Carried in the payload because stdout is the one channel `--quiet`
+ /// leaves open: a user who silenced stderr still has to be able to find
+ /// out that a question existed and why they were not asked it.
+ pub prompt_blocked: Option<NoPrompt>,
}
impl DecisionRequest<'_> {
@@ -189,6 +199,8 @@ impl DecisionRequest<'_> {
"priority_order": self.bands,
"heaviest_directories": self.heaviest,
"options": self.options(),
+ "prompt_offered": self.prompt_blocked.is_none(),
+ "prompt_not_offered_because": self.prompt_blocked.map(NoPrompt::explain),
"how_to_answer":
"re-invoke the identical command with --approve <id> (--yes is an alias for \
--approve proceed). Without an answer nothing has been indexed and nothing has \
@@ -344,17 +356,120 @@ impl DecisionRequest<'_> {
proceed). Nothing has been indexed."
.to_string(),
);
+ // Say why nobody was asked, where that can still be read. A log that
+ // shows the question but not the reason it went unasked is the same
+ // guessing game one layer up.
+ if let Some(blocked) = self.prompt_blocked {
+ lines.push(format!(" {}", blocked.explain()));
+ }
lines
}
}
-/// Is there a human who can answer right now?
+/// Why no terminal prompt was offered. `None` from [`prompt_blocked_by`] means
+/// one was.
///
-/// Both halves matter: stdin is where an answer would come from, stderr is
-/// where the question is printed. A run with a redirected stdin has nobody to
-/// ask, and blocking on it is the deadlock this gate exists to avoid.
-pub fn can_prompt() -> bool {
- std::io::stdin().is_terminal() && std::io::stderr().is_terminal()
+/// This exists as a value rather than a bare `bool` because the reason is
+/// printed: a run that stops without asking has to be able to say why it did
+/// not ask, or the silence is just another thing for the user to guess at.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum NoPrompt {
+ /// stdin is a pipe, a file, or closed — an agent, a CI job, a script.
+ /// Nobody is at the keyboard, so there is no answer to wait for.
+ StdinNotTerminal,
+ /// stderr is redirected. The question would land in a log nobody is
+ /// watching while the process sat waiting for a reply to it.
+ StderrNotTerminal,
+ /// `--quiet` / `--progress none` silenced the surface the question is
+ /// printed on. Someone *is* at the keyboard, but they were never shown
+ /// anything to answer — this is the case that made the run look hung.
+ QuestionSilenced,
+}
+
+impl NoPrompt {
+ /// One sentence, for the decision payload and the prose.
+ pub fn explain(self) -> &'static str {
+ match self {
+ NoPrompt::StdinNotTerminal => {
+ "no terminal prompt: stdin is not a terminal, so no answer could be typed. Re-run \
+ with --approve proceed|fast|cancel"
+ }
+ NoPrompt::StderrNotTerminal => {
+ "no terminal prompt: stderr is not a terminal, so the question would not have \
+ reached a screen. Re-run with --approve proceed|fast|cancel"
+ }
+ NoPrompt::QuestionSilenced => {
+ "no terminal prompt: --quiet / --progress none silences the question, and this run \
+ will never wait on stdin for something it did not print. The decision request \
+ below is on stdout, which --quiet does not silence. Re-run with --approve \
+ proceed|fast|cancel, or without --quiet to be asked"
+ }
+ }
+ }
+}
+
+/// Should this run stop to ask, and if not, why not?
+///
+/// Three conditions, and all three have to hold before autoindex touches
+/// stdin:
+///
+/// * `stdin_tty` — someone can type an answer;
+/// * `stderr_tty` — the question reaches a screen rather than a log file;
+/// * `question_visible` — the progress surface actually printed it. This is
+/// the one the first cut of the gate missed. `--quiet` / `--progress none`
+/// routes [`crate::progress::Progress::note`] to nothing, so the prose and
+/// the prompt vanished while the process still sat on `read_line`. Measured
+/// before the fix: a pty run with `--progress none` printed **zero bytes**
+/// and was still blocked at 220 s. An invisible prompt is a hang.
+///
+/// Pure, so the whole rule is testable without a terminal — the same shape as
+/// [`crate::progress::resolve`].
+pub fn prompt_blocked_by(
+ stdin_tty: bool,
+ stderr_tty: bool,
+ question_visible: bool,
+) -> Option<NoPrompt> {
+ if !stdin_tty {
+ return Some(NoPrompt::StdinNotTerminal);
+ }
+ if !stderr_tty {
+ return Some(NoPrompt::StderrNotTerminal);
+ }
+ if !question_visible {
+ return Some(NoPrompt::QuestionSilenced);
+ }
+ None
+}
+
+/// [`prompt_blocked_by`] against this process's real streams.
+///
+/// `question_visible` is the caller's progress surface
+/// ([`crate::progress::Progress::enabled`]) — gate has no business inspecting
+/// the surface itself, and passing the fact in keeps this testable.
+pub fn detect_prompt_block(question_visible: bool) -> Option<NoPrompt> {
+ prompt_blocked_by(
+ std::io::stdin().is_terminal(),
+ std::io::stderr().is_terminal(),
+ question_visible,
+ )
+}
+
+/// Ask, but only when [`prompt_blocked_by`] said it was safe to.
+///
+/// `open_stdin` is a closure and not a reader on purpose: when the prompt is
+/// blocked it is **never called**, so there is no path on which this function
+/// can end up holding — let alone reading — the process's stdin. That is the
+/// invariant, and `a_question_that_cannot_be_seen_is_never_asked` asserts it by
+/// handing in a reader that panics if anyone touches it.
+pub fn answer_from_terminal<R: std::io::BufRead>(
+ blocked: Option<NoPrompt>,
+ open_stdin: impl FnOnce() -> R,
+ echo: impl FnMut(&str),
+) -> Option<Approval> {
+ if blocked.is_some() {
+ return None;
+ }
+ read_answer(&mut open_stdin(), echo)
}
/// Read one answer from a terminal. `None` means "no answer" — EOF, an
@@ -525,6 +640,7 @@ mod tests {
semantic_datasets: vec!["docs".into()],
total_datasets: 3,
graph_files: 10,
+ prompt_blocked: None,
};
let payload = request.to_json();
assert_eq!(payload["xerj"], "autoindex-decision-request");
@@ -671,6 +787,129 @@ mod tests {
);
}
+ /// A reader that fails the test if anything reads it. Standing in for the
+ /// process's real stdin, which has no EOF and no answer coming.
+ struct NeverRead;
+
+ impl std::io::Read for NeverRead {
+ fn read(&mut self, _: &mut [u8]) -> std::io::Result<usize> {
+ panic!("stdin was read for a question the user was never shown");
+ }
+ }
+
+ impl std::io::BufRead for NeverRead {
+ fn fill_buf(&mut self) -> std::io::Result<&[u8]> {
+ panic!("stdin was read for a question the user was never shown");
+ }
+ fn consume(&mut self, _: usize) {}
+ }
+
+ /// THE regression test.
+ ///
+ /// `--quiet` / `--progress none` silences `Progress::note`, which is where
+ /// both the gate's prose and its prompt are printed — but the first cut of
+ /// the gate decided whether to ask from the two tty checks alone. So a
+ /// person at a terminal who passed `--quiet` was asked a question that was
+ /// never printed, and the process sat on `read_line` forever. Measured on
+ /// the pre-fix binary: a pty run with `--progress none` over the threshold
+ /// emitted **0 bytes** and was still blocked when it was killed at 220 s.
+ /// From the outside that is indistinguishable from a hang, in the one
+ /// feature whose entire job is to say "this will take a while".
+ ///
+ /// The reader panics if it is touched, so the invariant asserted here is
+ /// the real one — not "an answer was not returned" but "stdin was not
+ /// read".
+ #[test]
+ fn a_question_that_cannot_be_seen_is_never_asked() {
+ // Someone IS at the keyboard and both streams are terminals — the one
+ // thing missing is that the surface printed nothing.
+ let blocked = prompt_blocked_by(true, true, false);
+ assert_eq!(
+ blocked,
+ Some(NoPrompt::QuestionSilenced),
+ "a silenced surface must block the prompt on its own"
+ );
+ let mut echoed = Vec::new();
+ // Panics unless the fix is present: pre-fix this opened and read stdin.
+ let answer =
+ answer_from_terminal(blocked, || NeverRead, |line| echoed.push(line.to_string()));
+ assert_eq!(answer, None, "an unaskable question is never consent");
+ assert!(
+ echoed.is_empty(),
+ "nothing may be echoed to a surface that prints nothing: {echoed:?}"
+ );
+
+ // …and the run still says so where it can be read. stdout is not
+ // silenced by --quiet, so the payload has to carry the reason.
+ let estimate = estimate(10, 1_000);
+ let request = DecisionRequest {
+ root: "/data".into(),
+ estimate: &estimate,
+ max_minutes: 1,
+ bands: Vec::new(),
+ heaviest: Vec::new(),
+ without_generated: None,
+ semantic_datasets: Vec::new(),
+ total_datasets: 1,
+ graph_files: 0,
+ prompt_blocked: blocked,
+ };
+ let payload = request.to_json();
+ assert_eq!(payload["prompt_offered"], false);
+ let why = payload["prompt_not_offered_because"].as_str().unwrap();
+ assert!(why.contains("--quiet"), "{why}");
+ assert!(why.contains("stdout"), "{why}");
+ assert!(why.contains("--approve"), "{why}");
+ // The exit code is unchanged: a quiet run behaves exactly like the
+ // agent-driven one it now matches.
+ assert_eq!(payload["exit_code"], EXIT_NEEDS_DECISION);
+ }
+
+ /// The other three ways in, so the rule is pinned end to end rather than
+ /// only at the case that was broken.
+ #[test]
+ fn the_prompt_is_offered_only_when_all_three_conditions_hold() {
+ // A terminal, a visible surface: this is the one case that asks.
+ assert_eq!(prompt_blocked_by(true, true, true), None);
+ // Agent, CI, pipe: nobody to type. Checked first because it is the
+ // reason that stays true however the surface is configured.
+ assert_eq!(
+ prompt_blocked_by(false, true, true),
+ Some(NoPrompt::StdinNotTerminal)
+ );
+ assert_eq!(
+ prompt_blocked_by(false, false, false),
+ Some(NoPrompt::StdinNotTerminal)
+ );
+ // stderr redirected to a log: the question would not reach a screen.
+ assert_eq!(
+ prompt_blocked_by(true, false, true),
+ Some(NoPrompt::StderrNotTerminal)
+ );
+ // Every reason names a way out, and none of them is "wait".
+ for reason in [
+ NoPrompt::StdinNotTerminal,
+ NoPrompt::StderrNotTerminal,
+ NoPrompt::QuestionSilenced,
+ ] {
+ let text = reason.explain();
+ assert!(text.contains("--approve"), "{text}");
+ // Every blocked path must leave stdin alone, not merely decline to
+ // return an answer.
+ assert_eq!(
+ answer_from_terminal(Some(reason), || NeverRead, |_| {}),
+ None
+ );
+ }
+
+ // A visible terminal really does read the answer — the fix must not
+ // have turned the prompt off for everyone.
+ assert_eq!(
+ answer_from_terminal(None, || std::io::Cursor::new(b"f\n"), |_| {}),
+ Some(Approval::Fast)
+ );
+ }
+
#[test]
fn every_spelling_of_an_answer_maps_to_one_decision() {
for (input, expected) in [
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index 9c8905b4b..f98d8f654 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -2947,6 +2947,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
.filter(|dataset| dataset.semantic_field.is_some())
.map(|dataset| dataset.slug.clone())
.collect();
+ // Decided once, before anything is printed, and used for both halves:
+ // whether to touch stdin at all, and what the payload says about why
+ // nobody was asked. `pr.enabled()` is the missing third condition —
+ // `--quiet` / `--progress none` routes every `pr.note` below to
+ // nothing, so without it the gate printed no question and then waited
+ // on stdin for the answer to it.
+ let prompt_blocked = gate::detect_prompt_block(pr.enabled());
let request = gate::DecisionRequest {
root: root_str.clone(),
estimate: &run_estimate,
@@ -2966,6 +2973,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
} else {
pending.len() as u64
},
+ prompt_blocked,
};
match cfg.approve {
Some(answer) => pr.note(&format!(
@@ -2982,16 +2990,18 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
for line in request.prose() {
pr.note(&line);
}
- // Ask only where an answer can arrive. A pipe, a CI job or an
- // agent has no one at the keyboard, and blocking on stdin
- // there is an invisible deadlock — the failure this gate was
- // added to prevent, not to introduce.
- let answer = if gate::can_prompt() {
- let stdin = std::io::stdin();
- gate::read_answer(&mut stdin.lock(), |line| pr.note(line))
- } else {
- None
- };
+ // Ask only where an answer can arrive AND the question was
+ // actually shown. A pipe, a CI job or an agent has no one at
+ // the keyboard; a quiet run has someone at the keyboard who
+ // was shown nothing. Blocking on stdin in either case is an
+ // invisible deadlock — the failure this gate was added to
+ // prevent, not to introduce. When it is blocked, stdin is not
+ // opened at all.
+ let answer = gate::answer_from_terminal(
+ prompt_blocked,
+ || std::io::stdin().lock(),
+ |line| pr.note(line),
+ );
match answer {
Some(gate::Approval::Proceed) => pr.note("gate: proceeding"),
Some(gate::Approval::Fast) => {
diff --git a/landing/llms.txt b/landing/llms.txt
index b7a44a813..63bd93d49 100644
--- a/landing/llms.txt
+++ b/landing/llms.txt
@@ -91,6 +91,7 @@ If you are an AI coding agent driving XERJ on a person's machine, indexing is th
{"id":"fast","invoke":"--approve fast","effect":"adds --no-semantic --no-graph: …","cost":"speed-up NOT measured by this run. …"},
{"id":"narrower","invoke":"re-run as `xerj autoindex <subdirectory>` …","effect":"index only the part that matters","cost":"…"},
{"id":"cancel","invoke":"--approve cancel","effect":"index nothing, exit 0","cost":"none — nothing has been written yet"}],
+ "prompt_offered":false,"prompt_not_offered_because":"no terminal prompt: stdin is not a terminal …",
"how_to_answer":"re-invoke the identical command with --approve <id> …"}
```
@@ -100,7 +101,7 @@ Two honesty notes the payload states about itself and you should pass on: the es
- **Plan before you index.** `xerj autoindex <folder> --dry-run` walks, sniffs and samples every file, prints the plan, and indexes nothing. The line to report is the job size — `autoindex: 40 files (0 MB) under /path` — followed by the datasets it would create and the junk/skipped counts. Do not read it as the *first* line: a `bulk HTTP request timeout: Ns` line is printed before it, and both go to stderr. It needs a running node. Report the job-size line to your user before you commit their machine to anything. A dry run is read-only for the *index*, not for the disk: it creates the run's state dir, takes an exclusive lock on it and appends a journal record — so it fails if a real run of the same folder is already live, and a dry run before the first real run makes that run announce "resuming from journal".
- **Estimate honestly, from two signals.** (a) The dry run performs the full phase-A pass **only when no plan is frozen yet**; if the journal already holds one — because an earlier run was interrupted — it reuses that plan, returns in under a second and prints the *old* numbers. Treat a suspiciously fast dry run as "no new estimate", and point `--state-dir` at a fresh directory if you need a real one. (b) Recorded reference throughput on one machine: 518 MB / 1,995 files → ~38–51 s; 33.7k records/s on 923 MB (server-bound). Hardware varies widely — Apple Silicon laptops, cold caches, and PDF-heavy or SQL-dump-heavy corpora are much slower per byte than source code. Give the user a range and say it is an estimate; never state a confident single number you have not measured on their machine.
-- **The binary now enforces the ask — you no longer have to remember it.** Since v1.0.0-rc.15, `autoindex` measures per-format throughput **on the user's own machine** during phase A (it is already parsing every file to sniff it) and prints a range with its basis before phase B starts: `estimate: at least 64.2 s–64.2 s — a MEASURED FLOOR for client-side extraction…`, then one line per family (`code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s`). If the upper end exceeds `--max-minutes` (**default 10**) and you passed no `--approve`, it **indexes nothing**, writes a JSON decision request to stdout and exits **4** — a code of its own, so "I need a decision" is never confused with exit 1's catch-all failure. Answer by re-running the identical command with `--approve proceed|fast|cancel` (`--yes` = proceed); `--max-minutes 0` disables the gate. A person at a terminal is prompted instead, and a piped run is never blocked on stdin.
+- **The binary now enforces the ask — you no longer have to remember it.** Since v1.0.0-rc.15, `autoindex` measures per-format throughput **on the user's own machine** during phase A (it is already parsing every file to sniff it) and prints a range with its basis before phase B starts: `estimate: at least 64.2 s–64.2 s — a MEASURED FLOOR for client-side extraction…`, then one line per family (`code 500 files 749.3 MB at 11.7 MB/s measured over 500 file(s) → 64.2 s`). If the upper end exceeds `--max-minutes` (**default 10**) and you passed no `--approve`, it **indexes nothing**, writes a JSON decision request to stdout and exits **4** — a code of its own, so "I need a decision" is never confused with exit 1's catch-all failure. Answer by re-running the identical command with `--approve proceed|fast|cancel` (`--yes` = proceed); `--max-minutes 0` disables the gate. A person at a terminal is prompted instead — but only when the question is actually visible: stdin a terminal, stderr a terminal, and the progress surface on. `--quiet`/`--progress none` silences the question, so those runs are never prompted; they take the same path you do (payload on stdout, exit `4`), and `prompt_not_offered_because` says which condition was missing. **autoindex never waits on stdin for a question it did not print** — an invisible prompt is indistinguishable from a hang.
* **The number is a floor, not a prediction, and it says so.** It covers client-side extraction only: server indexing, embedding, network and edge-writing are not measured before the run starts, because measuring them would mean writing to the index the estimate exists to ask permission for. Measured here: on a 68 MB source tree the floor was 0.1 s against a real 8.9 s run; on a 793 MB one it was 64.2 s against a real ~350 s run. **So the gate under-asks and never over-asks** — when it fires the run is certainly long, but silence is not a promise the run is short. Say that to your user rather than repeating the floor as an ETA.
* **Families it could not time are named, not guessed at.** A family only gets a rate from files phase A provably read end to end; `sqlite` (row-capped) and gzipped files never qualify. Unpriced families appear under `unmeasured_families` with their byte counts, and `coverage` tells you what share of planned bytes the number actually covers. If nothing could be measured, there is no number and no gate — the `basis` field says so.
* **The payload is built for you.** `{"xerj":"autoindex-decision-request","exit_code":4,"reason":…,"estimate":{…},"priority_order":[…],"heaviest_directories":[…],"options":[…]}` — `heaviest_directories` names real byte counts and flags the ones matching the vendored/generated rule, and the `narrower` option re-costs the run without them using the same measured rates. The `fast` option deliberately states **no speed-up factor**: it reports which datasets and how many files it changes and says the factor was not measured.