Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
fix(openai-codex): evict a retired sandbox module's suspended scripts
Jack Rudenkob366f12a81de
diff --git a/ai-docs/architecture/README.md b/ai-docs/architecture/README.mdindex 54780dde..3ecf8859 100644--- a/ai-docs/architecture/README.md+++ b/ai-docs/architecture/README.md@@ -28,7 +28,7 @@ three earlier write-ups this material cites were lost.| [`keychain.md`](keychain.md) | The macOS Keychain backend: the 17ms-vs-28ms measurement that split presence from values, the two `security` behaviours that are traps (hex reads, a "secure" write that stores nothing and exits 0), and why no new `CredentialSource` member | touch `providers/keychain.ts`, `keychain-source.ts`, or the Providers tab's key write/delete || [`wire.md`](wire.md) | Every request from the connection's catalog `wire` and the route registry: the four-step pipeline, Jack's decisions where they diverge from the backend spec (clamp, borrow-and-drop, try in turn), the first-run refusal that stuck for a session, `+` doubling as the chain separator, name gates one layer down, custom endpoints' declared wire, the advisor and discovery on registry hosts | touch `connection-*.ts`, `wire-*.ts`, `declared-wire.ts`, `ComposedHandler` step 5w, an adapter's limit or reasoning fields, the advisor's request, or `+`/`#` model-spec markers || [`adapters.md`](adapters.md) | Layers 1–3 (FormatConverter / ModelTranslator / ProviderTransport), stream-parser selection, errors that ride an HTTP 200, and the `upstream_status` remap with its downstream readers | add a provider, model translator, or stream parser; or change error classification and retry |-| [`code-mode.md`](code-mode.md) | `--code-mode` for `openai-codex`: one freeform `exec` tool running JavaScript in a QuickJS sandbox, nested calls relayed as `tool_use` blocks; the decision and why its gates run in that order, the lite and classic shapes and the lite → classic fallback, the request-scoped binding and the account/session keys, step 5r resume and its tombstones, the input builder (turn records, classic rounds transliterated), the store byte ceilings, and the sandbox measurements (handle ownership, wasm growing to 2 GiB under a 32 MiB heap limit, hence the 1 GiB ceiling) | touch `code-mode/`, step 5d or 5r, the Responses leg parser's exec handling, the turn driver's code-mode hooks, or the sandbox limits |+| [`code-mode.md`](code-mode.md) | `--code-mode` for `openai-codex`: one freeform `exec` tool running JavaScript in a QuickJS sandbox, nested calls relayed as `tool_use` blocks; the decision and why its gates run in that order, the lite and classic shapes and the lite → classic fallback, the request-scoped binding and the account/session keys, step 5r resume and its tombstones, the input builder (turn records, classic rounds transliterated), the store byte ceilings, and the sandbox measurements (handle ownership, wasm growing to 2 GiB under a 32 MiB heap limit, hence the 256 MiB ceiling per module and 512 MiB per process) | touch `code-mode/`, step 5d or 5r, the Responses leg parser's exec handling, the turn driver's code-mode hooks, or the sandbox limits || [`network-recovery.md`](network-recovery.md) | The two-tier connection hold: why the deadline is DERIVED from `API_TIMEOUT_MS` (measured 359.607 s, a default and not a watchdog), why the tier-2 status is 503 and never 429, why the banner is magmux's `overlay` and never a pane of ours (closing a pane wedged Claude Code), why its lease is renewed on its own tick and not by ladder events, the five sites where a network failure wore an auth status code, and why a pane child gets no watchdog | touch the retry ladder, `recovery/`, the tier-2 status, the recovery banner, or any `x-claudish-recovery` reader || [`behavior-layer.md`](behavior-layer.md) | Layer 4, the harness-convention SUPERVISOR: the rule engine, the four hooks, the 115-call plan-mode measurement it was built on, and opt-in telemetry | write a `behavior/` rule, or change harness-conformance handling || [`advisor.md`](advisor.md) | `--advisor` for any main model: why it is independent of `--monitor` yet keeps four of its launch bits, the decorator around the ROUTED handler, id capture by tool name (the `toolu_` prefix that made it a no-op on every foreign provider), the tee and retained session-keyed state, the ten stub paths and advice origin, Claude Code's advisor gates, metered panel billing, and the inherited-placeholder 401 | touch `--advisor`, `advisor-decorator.ts`, `native-handler-advisor.ts`, `advisor-startup.ts`, monitor-gated launch code, or native auth in `claude-runner.ts` |diff --git a/ai-docs/architecture/code-mode.md b/ai-docs/architecture/code-mode.mdindex 1e1c82ee..90155638 100644--- a/ai-docs/architecture/code-mode.md+++ b/ai-docs/architecture/code-mode.md@@ -243,7 +243,7 @@ raise:| Store | Ceiling | Eviction ||---|---|---|-| suspended runtimes | 32 per module (one running + ≤ 31 suspended) | LRU → record `lost` |+| suspended runtimes | 32 (one running + ≤ 31 suspended), all in the current module | LRU → record `lost`; a retired module's at once || suspended idle | 6 h | `expire` → `lost` || exec records (settled) | ≤ 2,000 AND ≤ 32 MiB (source + output text + reasoning JSON; images are references) | LRU; id kept in `evictedExecs` || turn records | ≤ 2,000 AND ≤ 32 MiB; one record > 8 MiB is never stored | LRU by last render; anchor kept in `evictedTurns` |@@ -282,10 +282,16 @@ a fresh module.`setMemoryLimit`, `a.push("x".repeat(1024) + a.length)` grew the module's linear memory to the2048 MiB wasm maximum in 3.8 s before QuickJS reported out of memory; `a.push({i})` reached143 MiB. Linear memory never shrinks. So the interrupt handler also checks the module's-`HEAPU8.byteLength` (private, guarded): a slice that grows it past `WASM_CEILING_BYTES = 1 GiB` is-stopped (`out of memory: the script grew the sandbox's memory past claudish's limit`) and the module-is RETIRED (its scripts finish, no new ones are admitted). An idle module over 256 MiB is dropped-for reload when its last runtime ends.+`HEAPU8.byteLength` (private, guarded; a module where it reads 0 logs once that the ceilings are+off): a slice that grows it past `WASM_CEILING_BYTES = 256 MiB` is stopped (`out of memory: the+script grew the sandbox's memory past claudish's limit`) and the module is RETIRED: no new script+is admitted, and every SUSPENDED runtime in it is evicted at once (its record goes `lost`; replay+re-runs it in the next module), because a suspended runtime holds the whole module and may wait on+Claude Code for hours. An idle module over 256 MiB is dropped for reload when its last runtime+ends. Admission is also gated process-wide: a new runtime is refused while the wasm memory summed+over every module still alive exceeds `PROCESS_WASM_CEILING_BYTES = 512 MiB`, so the per-module+ceiling is a per-process one too. 256 MiB, not the 1 GiB first shipped: linear memory never+shrinks, so whatever one runaway script reaches stays resident in the proxy.**Settlement after every slice.** The top-level promise is read FIRST: fulfilled, or rejected withthe `exit()` sentinel → completed, and every outstanding call is DISCARDED (in claudish an unawaited@@ -297,7 +303,7 @@ promise that can never settle`). `blocked([])` is unrepresentable.|---|---|---|| QuickJS heap per runtime | 32 MiB | `failed` (out of memory) || CPU per script, cumulative across slices | 2 s | `failed` (`interrupted: exceeded the 2 s CPU budget`) |-| wasm linear memory per module | 1 GiB | slice stopped, module retired |+| wasm linear memory per module / summed over live modules | 256 MiB / 512 MiB | slice stopped, module retired, its suspended scripts lost / new script refused || source | 256 KB | `failed` before running || calls per batch / per script | 32 / 256 | the excess call rejects with a string, so `Promise.allSettled` still finishes || one call input / one batch | 2 MiB / 8 MiB | that call rejects |@@ -305,8 +311,8 @@ promise that can never settle`). `blocked([])` is unrepresentable.| `text()` output | 1 MiB per exec, enforced by the HOST on what it receives (a tampered prelude can waste one bounded copy, never raise the total) | dropped, with a warning line || output tokens | `max_output_tokens` pragma clamped to [1, 50,000]; default discovery's `truncation_policy.limit` in tokens, else 10,000 | Codex's truncation header, at 4 B/token |-Why 32 MiB and 2 s: 32 runtimes × 32 MiB keeps the live set under the 1 GiB wasm ceiling, half the-2 GiB wasm maximum. The interrupt handler can only TERMINATE a slice, never yield it (R2 §2:+Why 32 MiB and 2 s: the heap limit stops one runtime's ordinary allocations; it bounds nothing+process-wide (above), which is the wasm ceilings' job. The interrupt handler can only TERMINATE a slice, never yield it (R2 §2:`InternalError: interrupted`, 202 ms against a 200 ms budget), so the cumulative CPU budget is alsothe longest one script can block the event loop, and 2 s keeps one script from pausing everystream of a `serve` process for longer. Orchestration scripts need milliseconds. Neither value wasdiff --git a/packages/cli/src/code-mode/sandbox/quickjs-loader.ts b/packages/cli/src/code-mode/sandbox/quickjs-loader.tsindex 062ba83e..dbfb5ff0 100644--- a/packages/cli/src/code-mode/sandbox/quickjs-loader.ts+++ b/packages/cli/src/code-mode/sandbox/quickjs-loader.ts@@ -27,7 +27,7 @@import type { QuickJSWASMModule } from "quickjs-emscripten-core";import { log } from "../../logger.js";-/** One running + up to 31 suspended runtimes: ≤ 1 GiB of the 2 GiB wasm ceiling at 32 MiB each. */+/** One running + up to 31 suspended runtimes, process-wide (every runtime lives in the current lease). */export const MAX_LIVE_RUNTIMES = 32;/** Wasm memory above which an idle module is dropped. */export const RECYCLE_WASM_BYTES = 256 * 1024 * 1024;@@ -38,9 +38,18 @@ export const RECYCLE_WASM_BYTES = 256 * 1024 * 1024;* grew the module to 2048 MiB (the wasm maximum) in 3.8 s before QuickJS* reported out of memory, and an object-push loop reached 143 MiB. A slice* that grows the module past this ceiling is interrupted as out of memory, and- * the module is retired so the next script starts in a fresh one.+ * the module is retired so the next script starts in a fresh one. 256 MiB, not+ * more: linear memory never shrinks, so whatever one runaway script reaches+ * stays resident in the proxy until the module is dropped, and orchestration+ * scripts need kilobytes.*/-export const WASM_CEILING_BYTES = 1024 * 1024 * 1024;+export const WASM_CEILING_BYTES = 256 * 1024 * 1024;+/**+ * Wasm memory summed over EVERY module still alive in the process (the+ * current one plus retired ones a running script still holds). A new runtime+ * is refused while the sum is over this, so retired modules can never pile up.+ */+export const PROCESS_WASM_CEILING_BYTES = 2 * WASM_CEILING_BYTES;/** A runtime the lease tracks for the live cap and LRU eviction. */export interface LeasedRuntime {@@ -63,6 +72,7 @@ export class QuickJSModuleLease {constructor(module: QuickJSWASMModule) {this.module = module;+ liveLeases.add(this);}/** True once the module failed internally; no new runtime may be created in it. */@@ -81,17 +91,26 @@ export class QuickJSModuleLease {/*** Register a new runtime. At the cap, the least recently active suspended- * runtime is evicted first. Returns false when no slot can be freed (every- * live runtime is running), in which case the caller must not create one.+ * runtime is evicted first. Returns why the runtime is refused — the wasm+ * memory of every live module is over `PROCESS_WASM_CEILING_BYTES`, or no+ * slot can be freed (every live runtime is running) — in which case the+ * caller must not create one; undefined when admitted.*/- admit(runtime: LeasedRuntime): boolean {+ admit(runtime: LeasedRuntime): string | undefined {+ const processBytes = processWasmBytes();+ if (processBytes > PROCESS_WASM_CEILING_BYTES) {+ log(+ `[CodeMode] sandbox: ${processBytes} bytes of wasm memory across ${liveLeases.size} module(s); new script refused`+ );+ return "claudish: the sandbox is over its memory limit while other scripts finish; try again";+ }while (this.live.size >= MAX_LIVE_RUNTIMES) {let oldest: LeasedRuntime | undefined;for (const candidate of this.live) {if (candidate.running) continue;if (!oldest || candidate.lastActive < oldest.lastActive) oldest = candidate;}- if (!oldest) return false;+ if (!oldest) return SLOTS_BUSY;this.live.delete(oldest);log(`[CodeMode] sandbox: ${MAX_LIVE_RUNTIMES} live scripts; evicting the least recently used`@@ -100,11 +119,11 @@ export class QuickJSModuleLease {oldest.evict();} catch (err) {this.poison(`evicting a runtime threw: ${err}`);- return false;+ return SLOTS_BUSY;}}this.live.add(runtime);- return true;+ return undefined;}/** A runtime was disposed by its owner. May recycle an idle, oversized module. */@@ -115,17 +134,33 @@ export class QuickJSModuleLease {dropCurrent(this);log("[CodeMode] sandbox: idle module over 256 MiB of wasm memory; dropped for reload");}+ if (this.live.size === 0 && !this.usable) liveLeases.delete(this);}/*** Stop admitting scripts into this module (its wasm memory is over the- * ceiling and never shrinks); the scripts already in it finish normally.+ * ceiling and never shrinks). Every SUSPENDED runtime in it is evicted at+ * once (its record goes lost; replay re-runs it in the next module): a+ * suspended runtime holds the whole module, and a script can wait on Claude+ * Code for hours. Only the running script, the one that crossed the ceiling,+ * finishes here.*/retire(reason: string): void {if (this.retired) return;this.retired = true;dropCurrent(this);log(`[CodeMode] sandbox module retired: ${reason}`);+ for (const runtime of [...this.live]) {+ if (runtime.running) continue;+ this.live.delete(runtime);+ try {+ runtime.evict();+ } catch (err) {+ this.poison(`evicting a runtime of a retired module threw: ${err}`);+ return;+ }+ }+ if (this.live.size === 0) liveLeases.delete(this);}/**@@ -146,6 +181,7 @@ export class QuickJSModuleLease {}}dropCurrent(this);+ liveLeases.delete(this);}/** Current wasm linear memory size; 0 when the library does not expose it. */@@ -159,10 +195,25 @@ export class QuickJSModuleLease {}}+const SLOTS_BUSY = "claudish: every sandbox slot is running a script; try again";++/**+ * Every lease still alive: the current one, plus a retired one until its last+ * runtime ends. Their wasm memory is summed for admission.+ */+const liveLeases = new Set<QuickJSModuleLease>();++function processWasmBytes(): number {+ let bytes = 0;+ for (const lease of liveLeases) bytes += lease.wasmBytes();+ return bytes;+}+let current: QuickJSModuleLease | undefined;let loading: Promise<QuickJSModuleLease | undefined> | undefined;/** Set when the packages cannot be imported at all: no retry in this process. */let unavailable: string | undefined;+let wasmBytesUnreadableLogged = false;function dropCurrent(lease: QuickJSModuleLease): void {if (current === lease) current = undefined;@@ -175,7 +226,16 @@ async function loadModule(): Promise<QuickJSModuleLease | undefined> {const variant = await import("@jitl/quickjs-singlefile-mjs-release-sync");const module = await core.newQuickJSWASMModuleFromVariant(variant.default);log(`[CodeMode] sandbox module loaded in ${(performance.now() - started).toFixed(1)} ms`);- return new QuickJSModuleLease(module);+ const lease = new QuickJSModuleLease(module);+ if (lease.wasmBytes() === 0 && !wasmBytesUnreadableLogged) {+ // `wasmBytes()` reads a private field; a library upgrade that removes it+ // would switch the wasm ceilings off with no other sign.+ wasmBytesUnreadableLogged = true;+ log(+ "[CodeMode] sandbox: the module's wasm memory size is unreadable; the wasm memory ceilings are OFF"+ );+ }+ return lease;} catch (err) {unavailable = String(err);log(`[CodeMode] sandbox unavailable: ${unavailable}`);diff --git a/packages/cli/src/code-mode/sandbox/script-runtime.ts b/packages/cli/src/code-mode/sandbox/script-runtime.tsindex 141f3c87..f4640d39 100644--- a/packages/cli/src/code-mode/sandbox/script-runtime.ts+++ b/packages/cli/src/code-mode/sandbox/script-runtime.ts@@ -338,9 +338,8 @@ class QuickJSScriptRuntime implements ScriptRuntime, LeasedRuntime {// ─── context setup ──────────────────────────────────────────────────private createContext(): void {- if (!this.lease.admit(this)) {- throw new GuestFault("claudish: every sandbox slot is running a script; try again");- }+ const refused = this.lease.admit(this);+ if (refused !== undefined) throw new GuestFault(refused);const rt = this.lease.module.newRuntime();this.rt = rt;rt.setMemoryLimit(this.limits.memoryBytes);