Skip to content

Worktrunk is a CLI for Git worktree management, designed for parallel AI agent workflows

test(remove): cover the reap guard's spared branch deterministically (#3805)

Maximilian Roos0cb3cc1ef7d7
diff --git a/tests/CLAUDE.md b/tests/CLAUDE.md
index ed2bc90c7..923c4d297 100644
--- a/tests/CLAUDE.md
+++ b/tests/CLAUDE.md
@@ -384,14 +384,17 @@ Each threshold has an env override pinning it, so the output is present or absen
## No Retries
-Tests run once. Worktrunk configures no nextest `retries` and writes no retry loops: a test that passes only on a second attempt is a bug report, and retrying it discards the report while leaving the bug. A green suite has to mean the code is green, not that the run's flakes stayed under a retry budget. Fix the flake at its root:
+Tests run once. Worktrunk configures no nextest `retries`, and no test re-runs its own failed body: a test that passes only on a second attempt is a bug report, and retrying it discards the report while leaving the bug. A green suite has to mean the code is green, not that the run's flakes stayed under a retry budget. Fix the flake at its root:
- A racy assertion is a timing bug. Make it deterministic, per Timing Tests above: poll for the event, or drive it causally through a callback.
- Resource pressure is a concurrency bug. Windows process creation intermittently fails with STATUS_DLL_INIT_FAILED (exit `-1073741502`) when many tests spawn git/wt children at once. Bound how many heavy tests run together; that removes the pressure instead of retrying past it.
+- A kill at the 180s slow-timeout is a duration symptom with two causes, told apart by the durations around it: many stretched durations alongside high machine load is CPU starvation, usually a sibling worktree's concurrent build; one test pinned at the timeout in an otherwise-normal run is a blocked call inside it, usually network. Starvation's only lever here would be nextest `threads-required` bounds on the heavy PTY tests, deliberately unset while these stay rare one-offs: the bound taxes every healthy run, and can't see the sibling build that caused the starvation.
- A shared channel with more than one producer is an attribution bug. Counting events drained from a channel between steps only measures the step that produced them if nothing *else* can produce them: a background task's event landing after its step's drain is charged to the next step, so the assertion that fails names the wrong step and the wrong cause. The events are usually indistinguishable (skim's `Event::RunPreview` carries no payload), so identity assertions aren't available — quiesce the other producers instead, before the step sequence arms whatever they'd match. `on_update_pokes_run_preview_only_when_the_visible_pane_changes` is the worked example: `PreviewOrchestrator::wait_for_idle()` before each `note_awaiting`, so the skeleton precompute lands while no key matches it.
- A spawn that fails `NotFound` is a concurrent-build bug. Cargo uplifts `target/debug/wt` by removing the path and recreating it, so a second `cargo` against the same target directory leaves the binary absent for a fraction of a millisecond per rebuild, failing whatever spawn is in flight with `Os { code: 2, kind: NotFound }` — anywhere: an `insta_cmd` snapshot, a PTY wrapper's `wt config shell init`, a hook whose marker file then never appears — and passing on re-run, which is the signature. `wt_bin()` closes the window by spawning a hardlink pinned under `target/debug/wt-test-bin/` instead of the uplifted path (`testing::pin_test_binary`), so route every `wt` spawn through it or a helper that does; `test_wt_spawns_are_pinned` makes a direct `CARGO_BIN_EXE_wt` spawn fail the suite.
- A shared namespace is a collision bug. **Never `NamedTempFile::new()`** for a file a test needs by name: `tempfile` retries a name collision only when it surfaces as `AlreadyExists`, and on Windows `create_new` against a name already held by a *directory* — or by a file in delete-pending state — comes back `PermissionDenied`, which it hands straight back to the caller. A full suite run leaves the temp directory full of `.tmpXXXXXX` entries (every `TestRepo` makes one), and under that load the call fails ~1% of the time with `Access is denied.` — a panic that has nothing to do with what the test asserts. Take a `TempDir` and give the files fixed names inside it (`worktrunk::testing::directive_files` is the pattern); a directory collision surfaces as `AlreadyExists`, which tempfile retries.
+A bounded poll that rides out one identified `ErrorKind` whose window is understood is itself a root fix, and the doctrine leaves it alone: `pin_test_binary` polls `NotFound` across cargo's uplift window, and `forward_with_etxtbsy_retry` (`src/completion.rs`) polls `ExecutableFileBusy` while a concurrently-forked child holds the just-written script's write fd open. Both fail immediately on any other error, which is what keeps a poll from drifting into a retry.
+
**Reproducing a Windows-only flake** means reproducing its *neighbours*, not starving it: run the suspect tests in a loop on a Windows runner with a full `cargo nextest run` going alongside. Pinning the CPU instead models the wrong thing whenever the failing run's own timing was normal (compare its duration against the same test passing — nextest prints it, and the `test` job uploads `junit.xml` with every test's time). Measured both ways on the same five tests: 80 iterations under a concurrent full suite reproduced a 1-in-100 Windows failure that no local run had ever shown, while pinning all four cores with busy loops instead just pushed nearly every iteration past the 30s waits — artificial failures that say nothing about the flake.
## Testing with --execute Commands
diff --git a/tests/integration_tests/remove.rs b/tests/integration_tests/remove.rs
index 6e13a0c62..d333a7283 100644
--- a/tests/integration_tests/remove.rs
+++ b/tests/integration_tests/remove.rs
@@ -52,9 +52,9 @@ fn test_remove_reap_no_processes(mut repo: TestRepo) {
#[cfg(unix)]
#[rstest]
fn test_remove_reap_kills_process(mut repo: TestRepo) {
+ use crate::common::{wait_for, wait_for_worktree_removed};
use std::os::unix::process::CommandExt;
use std::process::{Command, Stdio};
- use std::time::{Duration, Instant};
use worktrunk::git::reap;
let worktree_path = repo.add_worktree("feature-reapkill");
@@ -73,19 +73,16 @@ fn test_remove_reap_kills_process(mut repo: TestRepo) {
let pid = child.id();
// Wait until lsof reports the child's cwd — fast when idle, but under
- // suite load a single probe can burn its whole 5s in-process timeout, so
- // the deadline is the suite's generous 60s presence-poll convention.
- let deadline = Instant::now() + Duration::from_secs(60);
- while !reap::processes_under(&canonical)
- .iter()
- .any(|p| p.pid == pid)
- {
- assert!(
- Instant::now() < deadline,
- "child {pid} never discovered — is lsof installed and able to read process cwds?"
- );
- std::thread::sleep(Duration::from_millis(50));
- }
+ // suite load a single probe can burn its whole 5s in-process timeout;
+ // `wait_for` carries the suite's generous presence-poll deadline.
+ wait_for(
+ &format!("child {pid} in cwd discovery — is lsof installed and able to read process cwds?"),
+ || {
+ reap::processes_under(&canonical)
+ .iter()
+ .any(|p| p.pid == pid)
+ },
+ );
// The guard reaps the child iff it holds no controlling terminal — which
// it inherited from this process's session, so read the session directly:
@@ -126,6 +123,87 @@ fn test_remove_reap_kills_process(mut repo: TestRepo) {
let _ = child.kill();
let _ = child.wait();
}
+
+ // The reap phase prints before the removal runs, so neither branch's
+ // output proves the removal itself completed.
+ wait_for_worktree_removed(&worktree_path);
+}
+
+// `--reap` (experimental) with a process *holding* a controlling terminal
+// under the worktree: the guard spares it. Unlike
+// `test_remove_reap_kills_process` this doesn't branch on the suite's own
+// terminal — the child gets a fresh PTY as its controlling terminal at spawn
+// (portable_pty runs it as a session leader on the slave side), so the spared
+// branch runs deterministically in every environment, CI included.
+//
+// From the outside a spared process is indistinguishable from an undiscovered
+// one (both print "No processes to reap"), so the test first proves discovery
+// sees the child, then pins the guard's verdict in-process. The load-bearing
+// assertion is the child surviving removal — the data-safety contract — which
+// a guard regression fails whenever discovery succeeds.
+#[cfg(unix)]
+#[rstest]
+fn test_remove_reap_spares_terminal_process(mut repo: TestRepo) {
+ use crate::common::{open_pty_with_size, wait_for, wait_for_worktree_removed};
+ use portable_pty::CommandBuilder;
+ use worktrunk::git::reap;
+
+ let worktree_path = repo.add_worktree("feature-reapspare");
+ let canonical = std::fs::canonicalize(&worktree_path).unwrap();
+
+ // The "keep-me" shape: a terminal-holding process cwd'd in the worktree.
+ // `sleep` writes nothing, so the never-read master can't fill and block it;
+ // 600s bounds the leak if an assertion panics before cleanup.
+ let pty = open_pty_with_size(24, 80);
+ let mut cmd = CommandBuilder::new("sleep");
+ cmd.arg("600");
+ cmd.cwd(&canonical);
+ let mut child = pty.slave.spawn_command(cmd).unwrap();
+ let pid = child.process_id().unwrap();
+
+ // Same presence poll as the kill test: prove discovery sees the child
+ // before asking what the guard makes of it.
+ wait_for(
+ &format!("child {pid} in cwd discovery — is lsof installed and able to read process cwds?"),
+ || {
+ reap::processes_under(&canonical)
+ .iter()
+ .any(|p| p.pid == pid)
+ },
+ );
+
+ // The guard's verdict, pinned in-process: discovered but not reapable.
+ // (A transiently failed `ps` probe also yields "not reapable" — the
+ // fail-safe points the same way as the contract, so this can't flake.)
+ assert!(
+ !reap::collect_reapable(&canonical)
+ .iter()
+ .any(|p| p.pid == pid),
+ "terminal-holding child {pid} was classified reapable"
+ );
+
+ let output = repo
+ .wt_command()
+ .args(["remove", "--reap", "feature-reapspare"])
+ .output()
+ .unwrap();
+ let stderr = String::from_utf8_lossy(&output.stderr);
+ assert!(
+ stderr.contains("No processes to reap"),
+ "expected no-reap output, got:\n{stderr}"
+ );
+ // The reap phase prints before the removal runs, so confirm the removal
+ // actually happened — otherwise the survival assertion below would hold
+ // just as well for a `wt` that gave up before touching the worktree.
+ assert!(output.status.success(), "remove failed; stderr:\n{stderr}");
+ wait_for_worktree_removed(&worktree_path);
+ assert!(
+ child.try_wait().unwrap().is_none(),
+ "spared child {pid} was killed during removal"
+ );
+
+ let _ = child.kill();
+ let _ = child.wait();
}
#[rstest]