Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

fix(openai-codex): evict a retired sandbox module's suspended scripts

Jack Rudenkob366f12a81de
diff --git a/ai-docs/architecture/README.md b/ai-docs/architecture/README.md
index 54780dde..3ecf8859 100644
--- a/ai-docs/architecture/README.md
+++ b/ai-docs/architecture/README.md
@@ -28,7 +28,7 @@ three earlier write-ups this material cites were lost.
| [`keychain.md`](keychain.md) | The macOS Keychain backend: the 17ms-vs-28ms measurement that split presence from values, the two `security` behaviours that are traps (hex reads, a "secure" write that stores nothing and exits 0), and why no new `CredentialSource` member | touch `providers/keychain.ts`, `keychain-source.ts`, or the Providers tab's key write/delete |
| [`wire.md`](wire.md) | Every request from the connection's catalog `wire` and the route registry: the four-step pipeline, Jack's decisions where they diverge from the backend spec (clamp, borrow-and-drop, try in turn), the first-run refusal that stuck for a session, `+` doubling as the chain separator, name gates one layer down, custom endpoints' declared wire, the advisor and discovery on registry hosts | touch `connection-*.ts`, `wire-*.ts`, `declared-wire.ts`, `ComposedHandler` step 5w, an adapter's limit or reasoning fields, the advisor's request, or `+`/`#` model-spec markers |
| [`adapters.md`](adapters.md) | Layers 1–3 (FormatConverter / ModelTranslator / ProviderTransport), stream-parser selection, errors that ride an HTTP 200, and the `upstream_status` remap with its downstream readers | add a provider, model translator, or stream parser; or change error classification and retry |
-| [`code-mode.md`](code-mode.md) | `--code-mode` for `openai-codex`: one freeform `exec` tool running JavaScript in a QuickJS sandbox, nested calls relayed as `tool_use` blocks; the decision and why its gates run in that order, the lite and classic shapes and the lite → classic fallback, the request-scoped binding and the account/session keys, step 5r resume and its tombstones, the input builder (turn records, classic rounds transliterated), the store byte ceilings, and the sandbox measurements (handle ownership, wasm growing to 2 GiB under a 32 MiB heap limit, hence the 1 GiB ceiling) | touch `code-mode/`, step 5d or 5r, the Responses leg parser's exec handling, the turn driver's code-mode hooks, or the sandbox limits |
+| [`code-mode.md`](code-mode.md) | `--code-mode` for `openai-codex`: one freeform `exec` tool running JavaScript in a QuickJS sandbox, nested calls relayed as `tool_use` blocks; the decision and why its gates run in that order, the lite and classic shapes and the lite → classic fallback, the request-scoped binding and the account/session keys, step 5r resume and its tombstones, the input builder (turn records, classic rounds transliterated), the store byte ceilings, and the sandbox measurements (handle ownership, wasm growing to 2 GiB under a 32 MiB heap limit, hence the 256 MiB ceiling per module and 512 MiB per process) | touch `code-mode/`, step 5d or 5r, the Responses leg parser's exec handling, the turn driver's code-mode hooks, or the sandbox limits |
| [`network-recovery.md`](network-recovery.md) | The two-tier connection hold: why the deadline is DERIVED from `API_TIMEOUT_MS` (measured 359.607 s, a default and not a watchdog), why the tier-2 status is 503 and never 429, why the banner is magmux's `overlay` and never a pane of ours (closing a pane wedged Claude Code), why its lease is renewed on its own tick and not by ladder events, the five sites where a network failure wore an auth status code, and why a pane child gets no watchdog | touch the retry ladder, `recovery/`, the tier-2 status, the recovery banner, or any `x-claudish-recovery` reader |
| [`behavior-layer.md`](behavior-layer.md) | Layer 4, the harness-convention SUPERVISOR: the rule engine, the four hooks, the 115-call plan-mode measurement it was built on, and opt-in telemetry | write a `behavior/` rule, or change harness-conformance handling |
| [`advisor.md`](advisor.md) | `--advisor` for any main model: why it is independent of `--monitor` yet keeps four of its launch bits, the decorator around the ROUTED handler, id capture by tool name (the `toolu_` prefix that made it a no-op on every foreign provider), the tee and retained session-keyed state, the ten stub paths and advice origin, Claude Code's advisor gates, metered panel billing, and the inherited-placeholder 401 | touch `--advisor`, `advisor-decorator.ts`, `native-handler-advisor.ts`, `advisor-startup.ts`, monitor-gated launch code, or native auth in `claude-runner.ts` |
diff --git a/ai-docs/architecture/code-mode.md b/ai-docs/architecture/code-mode.md
index 1e1c82ee..90155638 100644
--- a/ai-docs/architecture/code-mode.md
+++ b/ai-docs/architecture/code-mode.md
@@ -243,7 +243,7 @@ raise:
| Store | Ceiling | Eviction |
|---|---|---|
-| suspended runtimes | 32 per module (one running + ≤ 31 suspended) | LRU → record `lost` |
+| suspended runtimes | 32 (one running + ≤ 31 suspended), all in the current module | LRU → record `lost`; a retired module's at once |
| suspended idle | 6 h | `expire` → `lost` |
| exec records (settled) | ≤ 2,000 AND ≤ 32 MiB (source + output text + reasoning JSON; images are references) | LRU; id kept in `evictedExecs` |
| turn records | ≤ 2,000 AND ≤ 32 MiB; one record > 8 MiB is never stored | LRU by last render; anchor kept in `evictedTurns` |
@@ -282,10 +282,16 @@ a fresh module.
`setMemoryLimit`, `a.push("x".repeat(1024) + a.length)` grew the module's linear memory to the
2048 MiB wasm maximum in 3.8 s before QuickJS reported out of memory; `a.push({i})` reached
143 MiB. Linear memory never shrinks. So the interrupt handler also checks the module's
-`HEAPU8.byteLength` (private, guarded): a slice that grows it past `WASM_CEILING_BYTES = 1 GiB` is
-stopped (`out of memory: the script grew the sandbox's memory past claudish's limit`) and the module
-is RETIRED (its scripts finish, no new ones are admitted). An idle module over 256 MiB is dropped
-for reload when its last runtime ends.
+`HEAPU8.byteLength` (private, guarded; a module where it reads 0 logs once that the ceilings are
+off): a slice that grows it past `WASM_CEILING_BYTES = 256 MiB` is stopped (`out of memory: the
+script grew the sandbox's memory past claudish's limit`) and the module is RETIRED: no new script
+is admitted, and every SUSPENDED runtime in it is evicted at once (its record goes `lost`; replay
+re-runs it in the next module), because a suspended runtime holds the whole module and may wait on
+Claude Code for hours. An idle module over 256 MiB is dropped for reload when its last runtime
+ends. Admission is also gated process-wide: a new runtime is refused while the wasm memory summed
+over every module still alive exceeds `PROCESS_WASM_CEILING_BYTES = 512 MiB`, so the per-module
+ceiling is a per-process one too. 256 MiB, not the 1 GiB first shipped: linear memory never
+shrinks, so whatever one runaway script reaches stays resident in the proxy.
**Settlement after every slice.** The top-level promise is read FIRST: fulfilled, or rejected with
the `exit()` sentinel → completed, and every outstanding call is DISCARDED (in claudish an unawaited
@@ -297,7 +303,7 @@ promise that can never settle`). `blocked([])` is unrepresentable.
|---|---|---|
| QuickJS heap per runtime | 32 MiB | `failed` (out of memory) |
| CPU per script, cumulative across slices | 2 s | `failed` (`interrupted: exceeded the 2 s CPU budget`) |
-| wasm linear memory per module | 1 GiB | slice stopped, module retired |
+| wasm linear memory per module / summed over live modules | 256 MiB / 512 MiB | slice stopped, module retired, its suspended scripts lost / new script refused |
| source | 256 KB | `failed` before running |
| calls per batch / per script | 32 / 256 | the excess call rejects with a string, so `Promise.allSettled` still finishes |
| one call input / one batch | 2 MiB / 8 MiB | that call rejects |
@@ -305,8 +311,8 @@ promise that can never settle`). `blocked([])` is unrepresentable.
| `text()` output | 1 MiB per exec, enforced by the HOST on what it receives (a tampered prelude can waste one bounded copy, never raise the total) | dropped, with a warning line |
| output tokens | `max_output_tokens` pragma clamped to [1, 50,000]; default discovery's `truncation_policy.limit` in tokens, else 10,000 | Codex's truncation header, at 4 B/token |
-Why 32 MiB and 2 s: 32 runtimes × 32 MiB keeps the live set under the 1 GiB wasm ceiling, half the
-2 GiB wasm maximum. The interrupt handler can only TERMINATE a slice, never yield it (R2 §2:
+Why 32 MiB and 2 s: the heap limit stops one runtime's ordinary allocations; it bounds nothing
+process-wide (above), which is the wasm ceilings' job. The interrupt handler can only TERMINATE a slice, never yield it (R2 §2:
`InternalError: interrupted`, 202 ms against a 200 ms budget), so the cumulative CPU budget is also
the longest one script can block the event loop, and 2 s keeps one script from pausing every
stream of a `serve` process for longer. Orchestration scripts need milliseconds. Neither value was
diff --git a/packages/cli/src/code-mode/sandbox/quickjs-loader.ts b/packages/cli/src/code-mode/sandbox/quickjs-loader.ts
index 062ba83e..dbfb5ff0 100644
--- a/packages/cli/src/code-mode/sandbox/quickjs-loader.ts
+++ b/packages/cli/src/code-mode/sandbox/quickjs-loader.ts
@@ -27,7 +27,7 @@
import type { QuickJSWASMModule } from "quickjs-emscripten-core";
import { log } from "../../logger.js";
-/** One running + up to 31 suspended runtimes: ≤ 1 GiB of the 2 GiB wasm ceiling at 32 MiB each. */
+/** One running + up to 31 suspended runtimes, process-wide (every runtime lives in the current lease). */
export const MAX_LIVE_RUNTIMES = 32;
/** Wasm memory above which an idle module is dropped. */
export const RECYCLE_WASM_BYTES = 256 * 1024 * 1024;
@@ -38,9 +38,18 @@ export const RECYCLE_WASM_BYTES = 256 * 1024 * 1024;
* grew the module to 2048 MiB (the wasm maximum) in 3.8 s before QuickJS
* reported out of memory, and an object-push loop reached 143 MiB. A slice
* that grows the module past this ceiling is interrupted as out of memory, and
- * the module is retired so the next script starts in a fresh one.
+ * the module is retired so the next script starts in a fresh one. 256 MiB, not
+ * more: linear memory never shrinks, so whatever one runaway script reaches
+ * stays resident in the proxy until the module is dropped, and orchestration
+ * scripts need kilobytes.
*/
-export const WASM_CEILING_BYTES = 1024 * 1024 * 1024;
+export const WASM_CEILING_BYTES = 256 * 1024 * 1024;
+/**
+ * Wasm memory summed over EVERY module still alive in the process (the
+ * current one plus retired ones a running script still holds). A new runtime
+ * is refused while the sum is over this, so retired modules can never pile up.
+ */
+export const PROCESS_WASM_CEILING_BYTES = 2 * WASM_CEILING_BYTES;
/** A runtime the lease tracks for the live cap and LRU eviction. */
export interface LeasedRuntime {
@@ -63,6 +72,7 @@ export class QuickJSModuleLease {
constructor(module: QuickJSWASMModule) {
this.module = module;
+ liveLeases.add(this);
}
/** True once the module failed internally; no new runtime may be created in it. */
@@ -81,17 +91,26 @@ export class QuickJSModuleLease {
/**
* Register a new runtime. At the cap, the least recently active suspended
- * runtime is evicted first. Returns false when no slot can be freed (every
- * live runtime is running), in which case the caller must not create one.
+ * runtime is evicted first. Returns why the runtime is refused — the wasm
+ * memory of every live module is over `PROCESS_WASM_CEILING_BYTES`, or no
+ * slot can be freed (every live runtime is running) — in which case the
+ * caller must not create one; undefined when admitted.
*/
- admit(runtime: LeasedRuntime): boolean {
+ admit(runtime: LeasedRuntime): string | undefined {
+ const processBytes = processWasmBytes();
+ if (processBytes > PROCESS_WASM_CEILING_BYTES) {
+ log(
+ `[CodeMode] sandbox: ${processBytes} bytes of wasm memory across ${liveLeases.size} module(s); new script refused`
+ );
+ return "claudish: the sandbox is over its memory limit while other scripts finish; try again";
+ }
while (this.live.size >= MAX_LIVE_RUNTIMES) {
let oldest: LeasedRuntime | undefined;
for (const candidate of this.live) {
if (candidate.running) continue;
if (!oldest || candidate.lastActive < oldest.lastActive) oldest = candidate;
}
- if (!oldest) return false;
+ if (!oldest) return SLOTS_BUSY;
this.live.delete(oldest);
log(
`[CodeMode] sandbox: ${MAX_LIVE_RUNTIMES} live scripts; evicting the least recently used`
@@ -100,11 +119,11 @@ export class QuickJSModuleLease {
oldest.evict();
} catch (err) {
this.poison(`evicting a runtime threw: ${err}`);
- return false;
+ return SLOTS_BUSY;
}
}
this.live.add(runtime);
- return true;
+ return undefined;
}
/** A runtime was disposed by its owner. May recycle an idle, oversized module. */
@@ -115,17 +134,33 @@ export class QuickJSModuleLease {
dropCurrent(this);
log("[CodeMode] sandbox: idle module over 256 MiB of wasm memory; dropped for reload");
}
+ if (this.live.size === 0 && !this.usable) liveLeases.delete(this);
}
/**
* Stop admitting scripts into this module (its wasm memory is over the
- * ceiling and never shrinks); the scripts already in it finish normally.
+ * ceiling and never shrinks). Every SUSPENDED runtime in it is evicted at
+ * once (its record goes lost; replay re-runs it in the next module): a
+ * suspended runtime holds the whole module, and a script can wait on Claude
+ * Code for hours. Only the running script, the one that crossed the ceiling,
+ * finishes here.
*/
retire(reason: string): void {
if (this.retired) return;
this.retired = true;
dropCurrent(this);
log(`[CodeMode] sandbox module retired: ${reason}`);
+ for (const runtime of [...this.live]) {
+ if (runtime.running) continue;
+ this.live.delete(runtime);
+ try {
+ runtime.evict();
+ } catch (err) {
+ this.poison(`evicting a runtime of a retired module threw: ${err}`);
+ return;
+ }
+ }
+ if (this.live.size === 0) liveLeases.delete(this);
}
/**
@@ -146,6 +181,7 @@ export class QuickJSModuleLease {
}
}
dropCurrent(this);
+ liveLeases.delete(this);
}
/** Current wasm linear memory size; 0 when the library does not expose it. */
@@ -159,10 +195,25 @@ export class QuickJSModuleLease {
}
}
+const SLOTS_BUSY = "claudish: every sandbox slot is running a script; try again";
+
+/**
+ * Every lease still alive: the current one, plus a retired one until its last
+ * runtime ends. Their wasm memory is summed for admission.
+ */
+const liveLeases = new Set<QuickJSModuleLease>();
+
+function processWasmBytes(): number {
+ let bytes = 0;
+ for (const lease of liveLeases) bytes += lease.wasmBytes();
+ return bytes;
+}
+
let current: QuickJSModuleLease | undefined;
let loading: Promise<QuickJSModuleLease | undefined> | undefined;
/** Set when the packages cannot be imported at all: no retry in this process. */
let unavailable: string | undefined;
+let wasmBytesUnreadableLogged = false;
function dropCurrent(lease: QuickJSModuleLease): void {
if (current === lease) current = undefined;
@@ -175,7 +226,16 @@ async function loadModule(): Promise<QuickJSModuleLease | undefined> {
const variant = await import("@jitl/quickjs-singlefile-mjs-release-sync");
const module = await core.newQuickJSWASMModuleFromVariant(variant.default);
log(`[CodeMode] sandbox module loaded in ${(performance.now() - started).toFixed(1)} ms`);
- return new QuickJSModuleLease(module);
+ const lease = new QuickJSModuleLease(module);
+ if (lease.wasmBytes() === 0 && !wasmBytesUnreadableLogged) {
+ // `wasmBytes()` reads a private field; a library upgrade that removes it
+ // would switch the wasm ceilings off with no other sign.
+ wasmBytesUnreadableLogged = true;
+ log(
+ "[CodeMode] sandbox: the module's wasm memory size is unreadable; the wasm memory ceilings are OFF"
+ );
+ }
+ return lease;
} catch (err) {
unavailable = String(err);
log(`[CodeMode] sandbox unavailable: ${unavailable}`);
diff --git a/packages/cli/src/code-mode/sandbox/script-runtime.ts b/packages/cli/src/code-mode/sandbox/script-runtime.ts
index 141f3c87..f4640d39 100644
--- a/packages/cli/src/code-mode/sandbox/script-runtime.ts
+++ b/packages/cli/src/code-mode/sandbox/script-runtime.ts
@@ -338,9 +338,8 @@ class QuickJSScriptRuntime implements ScriptRuntime, LeasedRuntime {
// ─── context setup ──────────────────────────────────────────────────
private createContext(): void {
- if (!this.lease.admit(this)) {
- throw new GuestFault("claudish: every sandbox slot is running a script; try again");
- }
+ const refused = this.lease.admit(this);
+ if (refused !== undefined) throw new GuestFault(refused);
const rt = this.lease.module.newRuntime();
this.rt = rt;
rt.setMemoryLimit(this.limits.memoryBytes);