The fastest, cheapest, most efficient open-source agent harness. Run more than 500 agents on a $10 VPS.
Keep-running shell commands
Status: built (phase 1); phase 2 not started
Scope: crates/openhuman-core/src/tools/impl/system/shell*.rs,
crates/openhuman-core/src/sandbox/shell_exec.rs, a new per-turn process
registry, one new agent tool; TinyBox only for the optional phase 2
Related: tinyhumansai/openhuman#8226 (a timed-out command keeps its output),
tinyhumansai/tinybox#113 (a finished command returns even when a daemon holds
its pipes)
Problem
The shell tool blocks until the command ends or its deadline kills it.
- Killing loses output. A test, fuzzer or build that is still working at
its deadline is killed. #8226 returns what reached the tool before the kill,
but output held inside the process (C stdio, Ruby and Perl buffers) or
inside a pipeline (
… | tail -40) is lost. Across 85 timed-out shell calls in our Terminal-Bench 2 runs, 62 were piped totail,head,sortorgrep. No environment switch or buffering change recovers those. - Killing loses the work. A 40-minute test that is 30 minutes in is gone, and so is the time it took.
- Blocking loses the turn. While one call blocks, the model cannot look at
what the command has printed so far, decide it is going the wrong way, and
stop it. On
regex-chess(c14) a self-test with no timeout blocked for 43 minutes. Its first failure was printed in the first minute, and the model never saw it.
How other harnesses handle it
| Harness | Long command |
|---|---|
| OpenHands | After a stretch with no new output ("no-change timeout") the call returns what was printed so far with exit code -1 (still running). The model sends an empty command to keep waiting, or keys to interrupt. |
Codex CLI (unified_exec) |
exec_command returns after yield_time_ms (10 s by default) and the command keeps running; write_stdin polls it. |
| Terminus 2 (Terminal-Bench's agent) | A tmux terminal: keystrokes plus a wait, then the screen is read; a long command keeps running. |
| Claude Code | A hard timeout (2 min default, 10 max); for long work the model sets run_in_background and is notified on completion. |
None of these kills a long command at a fixed deadline by default. The cost they report is runaway polling: a Codex session polled a finished command for about three hours, and another spent about 55M tokens re-sending context on polls (openai/codex #53659, #38495).
What TinyBox already provides
Checked against the vendored TinyBox (c2e34c5):
- A native shell runs as a process the host reads by cursor
(
read_output(process, cursor)returns chunks andnext_sequence). Reading does not consume anything, and the process keeps running whether or not the host reads (tinybox-module/src/tinybus_module/output.rs,OutputJournal). - The process stops on its own runtime budget (
timeout_msin the request, up to 24 h; it ends asFailed { TimedOut }), oncancel, or when it exits.is_running(process)answers without reading. release_outputis accepted only once cleanup is acknowledged, that is after the process has stopped; the host then closes the command's resource.- Limit: the journal keeps every chunk until release, and the total is
capped at
MAX_OUTPUT_BYTES= 1 MiB per process. Past it, the process is stopped withFailed { OutputLimit }. A chatty command kept alive for many minutes can reach it. - Up to 32 stream executions per module instance (
MAX_STREAM_EXECUTIONS).
Today the host (sandbox/shell_exec.rs) reserves a resource, creates it,
starts the shell, reads in a loop until exit or its own deadline, cancels on
the deadline, releases the output and closes the resource, all inside one
tool call. A dropped wait (cancelled turn) cancels the command.
So the capability exists. What is missing is a host that lets the process outlive one tool call.
Design
Tool surface
In a turn, shell's timeout_secs is a yield point, not a kill.
If the command is still running
timeout_secsafter it started (default 120 s, at most 600 s;0waits for the end), the call returns without cancelling:Still running as process p3 after 2m 0s; it keeps running (it stops at its 10m 0s timeout). Output so far: [stdout] … Follow it with shell_process {"process": "p3"} to wait for more output or the end, or {"process": "p3", "action": "kill"} to stop it. …The result is not an error. A command that finishes before the yield point returns exactly as today.
The yield point comes no later than an eighth of the command's deadline (not earlier than 30 s for that cap) and always before the deadline itself, so a command started late in a turn still yields before the turn kills it.
A deliberate wait,
sleepfollowed only by commands that look at the result (sleep 60; tail -20 build.log), runs to its end instead of yielding with nothing to show. Claude Code makes a similar exception.sleep 2; make -j8yields as usual.A call that cannot yield is still bounded by the model's time. When the turn already follows four processes, when the command is a deliberate wait, or on a backend that cannot leave a command running (Docker), the model's
timeout_secsstays the command's deadline, as before.Only where the model can follow. The host middleware moves
timeout_secsaside only in a turn that registersshell_process. An agent whose tool belt namesshellalone keepstimeout_secsas a kill and never yields.The kill is the turn's. TinyAgents'
ShellTurnBudgetclamp writes the turn's remaining budget intotimeout_secs, and the shell passes that to TinyBox as the command'stimeout_ms. So a host middleware (agent/tinyagents/middleware/shell_yield.rs), registered just before the clamp, moves the model'stimeout_secsto the shell's internal yield argument (yield_after_secs); the clamp then sees a call that asked for no deadline and sets it as before. When the deadline fires, the result is the #8226 timeout report.For a kill of its own, the model writes coreutils
timeout Ninto the command, as it already does.Outside a turn (CLI, JSON-RPC) nothing moves
timeout_secs: it is the caller's deadline, and nothing yields.
Why one knob: on the Terminal-Bench baseline the model set timeout_secs on
16% of shell calls, mostly 300, 600 or 900 s, as an estimate of how long the
work takes, and wrote timeout N into 95 commands when it wanted a kill. Of
27 killed calls, 15 asked for no deadline and died at the turn's clamp. With
two knobs, a short timeout_secs silently meant "never yield, kill at N".
A new tool, shell_process:
| Argument | Meaning |
|---|---|
process |
The id a yield returned. |
action |
wait (default), kill or status. |
wait_secs |
For wait: how long to block before returning, default 120, clamped to 1–600. The process's own deadline still stops it. |
waitblocks host-side until the process exits orwait_secspasses, then returns only the output written since the last read, plus "exited with code N" or "still running (Nm Ns)".killcancels it and returns its remaining output.statussays whether it is still running, without output. It refreshes the state with a read that returns at once and keeps that output for the nextwait, so a process that has ended says so and frees its place.- A
killthat did not take within 10 s is reported as such; the process keeps its place until it stops. - A
waiton a process that has already exited and been read returns "already exited with code N" at once, so a loop cannot spin on it.
Process registry (OpenHuman host)
A per-turn registry, e.g. sandbox/shell_processes.rs:
- An entry holds the live handle (TinyBox resource, process and read cursor),
the start time, the command line (for display) and the hard deadline.
Output read after the yield goes to the model in
shell_processreplies, not through the shell's progress meter. - Ids are short and turn-local (
p1,p2, …). - At most 4 live processes per turn. A call reserves its place before the
command starts, so parallel calls cannot overshoot; a command that finishes
first frees it. A fifth command waits for its end, bounded by the model's
timeout_secs. - Audit: a command left running is audited when it ends (or is stopped at turn end), with its real outcome and duration, not as a success at the yield.
- When a process has stopped and its output has been read, the registry releases the output and closes the resource.
- Turn end: the registry lives on the turn's run context
(
OpenHumanRunContext::shell_processes; a child run gets its own). Dropping it at the end of the turn drops every handle, and a handle dropped while its process runs cancels it, waits up to 10 s for it to stop, releases its output and closes its resource in the background. Processes do not outlive the turn; long-lived services are started detached, as today (tinybox#113 makes that return promptly). - Direct dispatch (CLI, JSON-RPC: no run context) has no registry, so
shellwaits for the end there as before.
Output
- Each reply carries the new output since the last read, under the existing tool-output caps. Longer output is kept by the harness and previewed, as for any tool result.
- The yield and
waitreplies label streams with[stdout]and[stderr], like #8226. - The 1 MiB journal cap is a hard stop. The reply names it ("TinyBox stopped the command: its output exceeded the execution output limit"). See phase 2.
Model guidance
The shell tool's description says that a command still running after
timeout_secs comes back with its output so far and keeps running as a
process, to follow it with shell_process rather than start a second copy,
that it is stopped when the turn's time runs out, that timeout_secs is when
the call comes back rather than a kill (put timeout N in the command for
that), and not to pipe a long
command through tail or head (a pipeline holds its output until the
command ends, so a yield shows nothing).
Failure classification and recovery
- A yield is not a failure; the repeat and no-progress guards do not count it.
- A
killby the model is not a failure. - A process that reaches its hard deadline reports exactly as a timed-out command does today (#8226), and keeps the shell timeout class.
- The no-progress guard counts consecutive
waitreplies that carried no new output on the same process. From the third, the reply suggests stopping it, and the tool-loop limits still apply.
Paths
| Path | Keep-running |
|---|---|
| Native TinyBox shell (default) | yes |
| Native jail | yes (same process API) |
Docker module (run_docker) |
no: it is one-shot and has no process handle; the model's timeout_secs is its deadline, as before |
| Sandbox-policy path (sandboxed agents, SaaS) | yes on the no-sandbox and native-jail backends; Docker as above |
| Cron jobs | unchanged; they keep blocking semantics |
| Windows | yes, nothing here is Unix-specific |
Phase 2 (TinyBox, only if the 1 MiB cap bites)
Let the host trim what it has read: release_output_until(process, cursor)
drops chunks below the cursor, and the cap applies to bytes still held rather
than to everything the process ever wrote. A long-running chatty command then
runs to its deadline. This is a contract change in TinyBox, so it waits for
evidence that the cap is reached in practice.
What this replaces
- "Half the remaining turn for an unbounded command" (held, not published): with a yield point, an unbounded command no longer blocks the turn, so it needs no special cap.
- A pseudo-terminal for every command: rejected as too invasive (pagers, colours, password prompts, merged streams). Keeping the process alive recovers buffered and piped output without it.
stdbufpreloading: Linux-only and fragile; not needed once nothing is killed early.- A "don't pipe to
tail" line in the description: kept, since a pipeline shows nothing at a yield either.
Risks
- Polling loops. Mitigated by host-side blocking waits, the already-exited answer, the no-new-output note and the turn's tool-loop limits. The Codex reports are the cases to test against.
- Forgotten processes. Bounded by the per-turn cap, the hard deadline and turn-end cleanup.
- The model starts a duplicate. The guidance line, plus the yield reply naming the running process.
- Token cost. Each wait re-sends the conversation; a long wait (120 s default) keeps that to a few calls per command.
Tests
- Unit, through the test stand-in (
sandbox/shell_processes_tests.rs,tools/impl/system/shell_process_tests.rs,tools/impl/system/shell_tests_keep_running_tests.rs):- a command finishing before the yield point returns as today;
- a command past it yields with its output and is followed to its end;
waitreturns only new output, then the exit code; a non-zero end reads like a failedshellcall;killstops it and returns the rest;- asking about an ended process again says how it ended;
- waits with nothing new are counted, and the reply suggests stopping it;
- at most four processes are left running;
- dropping the registry stops its processes;
- the runtime budget still stops a process, as a timeout;
timeout_secs: 0, a command that starts withsleep, and direct dispatch wait for the end;- the yield point's default, ceiling and deadline-relative cap;
- the turn moves the model's
timeout_secsto the yield argument, and the turn-budget clamp then sets the deadline.
- Not covered by a unit test: the 1 MiB cap (the message comes from the existing TinyBox failure mapping).
- Live, on the released TinyBox through the Terminal-Bench rig (no automated
test): processes survive between tool calls, are followed, killed and
stopped at turn end (
mailman,gpt2-codegolf,regex-chess).
Work
- OpenHuman (done):
run_yieldingand the live handle (sandbox/shell_exec.rs,sandbox/shell_live.rs), the registry (sandbox/shell_processes.rs), theshellyield,shell_process, turn-end cleanup, description, tests. - Bench validation: single-task runs so far; a full-suite A/B against the no-yield baseline is still to do.
- TinyBox phase 2 only with evidence.
Open questions
- The default yield point: 120 s total, or "no new output for 30 s" as in OpenHands? A total is predictable and doesn't cut short a computation that prints nothing; a silence rule returns sooner on a daemon that printed its startup and went quiet. Built: the total. A silence rule waits for an A/B.
- Other harnesses, for reference: Codex CLI yields after 10 s (0.25–30 s) and has no hard kill in interactive mode; Terminus 2 waits a model-chosen duration of at most 60 s; OpenHands returns after 30 s without new output; Claude Code moves a command to the background at its timeout (120 s default, 600 s max) instead of killing it.
- Should processes survive into the thread's next turn (a dev server started by one message, inspected by the next)? This spec says no: turn-scoped only.