Multiplayer agent harness for work.
Align the E2B backend with E2B lifetime, snapshot and network docs (#1421)
Regan Bell90221513b8dc
diff --git a/deploy/e2b/e2b.Dockerfile b/deploy/e2b/e2b.Dockerfileindex f2b8332ff..e605b5c86 100644--- a/deploy/e2b/e2b.Dockerfile+++ b/deploy/e2b/e2b.Dockerfile@@ -8,5 +8,5 @@ RUN apt-get update && apt-get install -y --no-install-recommends \&& rm -rf /var/lib/apt/lists/* \&& node --version && npm --version-RUN usermod -l user -d /home/user -m node && groupmod -n user node+RUN id -u user >/dev/null 2>&1 || useradd --create-home --user-group --shell /bin/bash userRUN mkdir -p /home/user/workspace && chown -R user:user /home/userdiff --git a/docs/e2b-template.md b/docs/e2b-template.mdnew file mode 100644index 000000000..f8b3595eb--- /dev/null+++ b/docs/e2b-template.md@@ -0,0 +1,47 @@+# E2B sandbox template++E2B stopped accepting V1 template builds (`e2b template build` against a bare+Dockerfile with an access token) on 2026-08-01. `deploy/e2b/e2b.Dockerfile` is still+the single source for the sandbox image, but it is now fed to the Template SDK, which+E2B's migration guide names as the replacement for Dockerfile-only builds.++From the matching source checkout, install dependencies with `npm ci`, export+`E2B_API_KEY`, then run:++```bash+npm run build:e2b-template+```++The command builds the Dockerfile as the template `qm-base` and prints+`E2B_TEMPLATE_ID=qm-base` with the template and build identifiers. Set+`E2B_TEMPLATE_ID` in the core deployment to that name. Sandboxes created from the+template run as the `user` account with `/home/user/workspace` as the working+directory, which is what `src/sandbox/e2b-sandbox.ts` assumes.++`E2B_TEMPLATE_NAME` renames the template. `E2B_TEMPLATE_CPUS` and+`E2B_TEMPLATE_MEMORY_MB` set the sandbox shape recorded on the template; E2B's plan+limits cap them, and the running shape is reported back through `Sandbox.getInfo`, so+the profile advertises whatever the sandbox actually received rather than a guessed+value. Leaving them unset uses E2B's default of 2 vCPU and 512 MiB.++## Layer tools baked into a child template++Core installs the deployment layer's tool files into every new sandbox on first+provision, after checking their checksums. To skip that step at runtime, point+`DEPLOYMENT_LAYER` at the layer directory when building:++```bash+DEPLOYMENT_LAYER=deploy/layers/<org> npm run build:e2b-template+```++This builds a second template, `qm-base-layer` (override with+`E2B_LAYER_TEMPLATE_NAME`), derived from `qm-base` with `fromTemplate` and containing+each declared install file at its destination path with its declared mode. Set+`E2B_TEMPLATE_ID` to the layer template instead. The runtime checksum probe finds the+files already present and installs nothing; when the layer changes, rebuild the child+template or let the runtime installer replace the stale copies on the next provision.++Rebuilding from unchanged inputs uses E2B's build cache. Existing sandboxes keep the+template they were created from; a paused scope sandbox only picks up a new template+after it is destroyed or restored from a recovery snapshot, which is itself a template+derived from the old image.diff --git a/docs/sandbox-preservation.md b/docs/sandbox-preservation.mdindex 94f032c49..c8517560d 100644--- a/docs/sandbox-preservation.md+++ b/docs/sandbox-preservation.md@@ -13,7 +13,15 @@ When enabled, Modal uses native directory checkpoints of `/root`, mounted into a`MODAL_SNAPSHOT_RETENTION_SEC` defaults to 2592000 (30 days) and must be positive. Checkpoint IDs, capture timestamps, errors and expiration times are stored with sandbox records. `DATABASE_URL` must point to durable Postgres to retain these references across core restarts; without it the sandbox record map is memory-only. Status reports `recovery.checkpointExpiresAtMs`; using a checkpoint does not renew its expiry, because Modal retains a directory snapshot for a hard cutoff measured from creation. The maintenance sweep therefore renews the checkpoint of a reaped idle scope once less than seven days (or half the checkpoint's lifetime, whichever is shorter) remain: it restores the checkpoint into a fresh machine, captures a new checkpoint, and terminates that machine again, so an idle user never loses a home to retention alone. Renewal holds the same lifecycle lock as provisioning and leaves the scope's idle history untouched. Expired or failed native restores stop with an explicit recovery error instead of replacing the home with an empty directory or silently using an older portable backup. A core crash during hydration leaves a durable pending marker; another core refuses to expose that potentially incomplete home until an operator confirms the original core is no longer hydrating and invokes `sandbox` with `action: "restart"`. Restart terminates only the incomplete replacement, retains checkpoint references, and clears the pending marker; the next provision retries that retained checkpoint. It does not archive or promote the incomplete home. Handled restore failures can retry after successful cleanup of the failed replacement.-E2B creates persistent machines with an explicit pause-on-timeout lifecycle and pauses them at teardown unless `keepWarm` retains running background work. The provider timeout still applies; `keepWarm` does not extend that timeout. Routine native pause does not create a second tar backup. Provider pause retains disk and normally memory; provider fallback may retain disk only, so reconnectable processes must not depend on RAM surviving. Status can inspect a paused machine without waking it. Pause failures are recorded and returned, with the running machine retained for retry. A lost native machine requires explicit recovery rather than an automatic blank replacement. Existing E2B machines whose lifecycle does not confirm pause-on-timeout retain legacy portable checkpointing.+E2B creates persistent machines with an explicit pause-on-timeout lifecycle and pauses them at teardown unless `keepWarm` retains running background work. See [E2B sandbox template](e2b-template.md) for building the sandbox image.++The provider timeout is managed, not left to expire under work. Every command first extends the sandbox timeout to cover the command plus a one-minute margin when the remaining lifetime is shorter, `E2B_SANDBOX_TTL_SEC` defaults to the largest command timeout plus that margin instead of fifteen minutes, and `keepWarm` teardown extends the timeout to `BACKGROUND_JOB_TTL_MAX_SEC` so background processes survive as long as the background-work contract promises. Every extension is capped by `E2B_MAX_LIFETIME_SEC` (default 86400, E2B's Pro-plan maximum; Hobby plans must set 3600). If a sandbox is nevertheless lost while a command is running, the command is not re-run: it may have partially executed, so the turn receives an explicit error and the next command reconnects. A sandbox found gone before a command starts is reconnected or replaced transparently, as before.++Routine native pause does not create a tar backup. Instead, used-turn teardown captures a provider-native persistent snapshot (E2B `createSnapshot`, an image that survives deletion of the sandbox) when one is due under `E2B_NATIVE_SNAPSHOT_INTERVAL_SEC` (default 300), then pauses. Only the newest snapshot per scope is kept; the superseded one is deleted after the new capture succeeds, and destroying the scope deletes the snapshot too. The capture briefly pauses the sandbox and drops any open connections, which is why it runs only at teardown and never while a command is in flight. Status reports the snapshot as `recovery.checkpointId` with no provider expiry. When the paused sandbox itself is gone, the next provision creates a replacement from that snapshot; a scope with no snapshot still requires explicit recovery rather than an automatic blank replacement. Snapshot capture failures are recorded as the recovery error and reported, and the pause still proceeds.++Provider pause retains disk and normally memory; provider fallback may retain disk only, so reconnectable processes must not depend on RAM surviving. Status can inspect a paused machine without waking it and, for a running one, reports CPU, memory and disk usage from the provider's metrics. Pause failures are recorded and returned, with the running machine retained for retry. Existing E2B machines whose lifecycle does not confirm pause-on-timeout retain legacy portable checkpointing under `E2B_SNAPSHOT_INTERVAL_SEC`.++With `E2B_EGRESS_PROXY_URL` set, sandboxes are created with E2B network rules that deny all outbound traffic except the proxy host, and existing sandboxes receive the same rules when core reconnects, so the `HTTPS_PROXY` environment cannot be bypassed from inside the sandbox. E2B matches allowed hostnames by SNI on port 443 and by Host header on port 80 only, so the proxy must listen on one of those ports or be given by IP; core refuses to start with a hostname on any other port rather than silently blocking all egress. Without the setting, sandboxes have unrestricted egress and the profile reports no enforcement.Sprites keep the whole disk across sleeps, so the sprite itself is the durable home. Every used-turn teardown that may have changed the home takes a provider checkpoint (copy-on-write, throttled to one per five minutes per core); `computerStatus` reports the newest one as `recovery.strategy: "provider_snapshot"`. Checkpoints are the rollback path: when a computer restart is refused and the health check positively reports an unhealthy status, restart restores the newest checkpoint and records `checkpoint_restored`, which discards changes made since that checkpoint. Deleting a sprite is irreversible and deletes its checkpoints with it, so with `SPRITES_SNAPSHOT_S3_BUCKET` set a scope's home is exported as a portable tar before its sprite is destroyed, a failed export refuses the destroy, and a replacement sprite for that scope hydrates from the export. Initialization is recorded as pending before provider creation in a durable map, and provision, capture, restore and destruction share a cross-core lifecycle lock. A failed initialization remains pending across core restarts and cannot overwrite a saved home; a later provision retries hydration. Failed or unknown health observations never authorize checkpoint rollback. Background processes started with process sessions hold the sprite awake through the Tasks API while they run; they still do not survive a cold wake, a restart or a checkpoint restore. `SPRITES_MEMORY_MB` sets an explicit memory resources policy per sprite; without it memory is provider-managed. Commands run over the WebSocket exec channel and files move through the filesystem API.diff --git a/package-lock.json b/package-lock.jsonindex f77ef7788..f103220c4 100644--- a/package-lock.json+++ b/package-lock.json@@ -31,7 +31,7 @@"@smithy/signature-v4": "^5.6.6","@superserve/sdk": "^0.8.3","croner": "^10.0.1",- "e2b": "^2.39.0",+ "e2b": "^2.51.0","emoji-datasource": "^16.0.0","fastify": "^5.12.1","jose": "^6.2.3",@@ -827,9 +827,9 @@}},"node_modules/@bufbuild/protobuf": {- "version": "2.14.1",- "resolved": "https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.14.1.tgz",- "integrity": "sha512-agRJn3+EJDUe8AvxTx/LnHA/GErvLE62pSaSk7+MwFOtOv8eWBu/qCq2qoZjBjVZ3C2aiFJCveuSs17KMkYGOw==",+ "version": "2.15.0",+ "resolved": "https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.15.0.tgz",+ "integrity": "sha512-DAheWUkVr/SJTWCc+lg9dhY0eN4SaWlf4+bG1KzHeXbnqt0AfB/NX0Z+VunGlM1ki1B4zVvye27MpKh/svySUA==","license": "(Apache-2.0 AND BSD-3-Clause)"},"node_modules/@cbor-extract/cbor-extract-darwin-arm64": {@@ -911,22 +911,22 @@]},"node_modules/@connectrpc/connect": {- "version": "2.1.2",- "resolved": "https://registry.npmjs.org/@connectrpc/connect/-/connect-2.1.2.tgz",- "integrity": "sha512-MXkBijtcX09R10Eb6sFeIetc6w6746eio6xtfuyVOH7oQAacT1X0GzMIQFux6Qy8cq3W/T5qX5Bei8YbFtmRGA==",+ "version": "2.2.0",+ "resolved": "https://registry.npmjs.org/@connectrpc/connect/-/connect-2.2.0.tgz",+ "integrity": "sha512-M63OQ1ZFcPUm85CuewJ9G9CMV8LjzGR9935xOnXNQGSazAbwMb3x2W7vXBYM1zqrP7XFD8uxp/dUAhirclgwnQ==","license": "Apache-2.0","peerDependencies": {"@bufbuild/protobuf": "^2.7.0"}},"node_modules/@connectrpc/connect-web": {- "version": "2.1.2",- "resolved": "https://registry.npmjs.org/@connectrpc/connect-web/-/connect-web-2.1.2.tgz",- "integrity": "sha512-1tfaK85MU+gJjwwmL31d2rzdf0XCYX99chZf63uG89SGBUd4XuZ4ZzhGo2u79TPXOE6nLIZQ2okrpyey42PYdg==",+ "version": "2.2.0",+ "resolved": "https://registry.npmjs.org/@connectrpc/connect-web/-/connect-web-2.2.0.tgz",+ "integrity": "sha512-uuBUcBTYfb/r1o98LzCCZk1H90yx0vmZ5m21dpw/q4EUwsKwkCDGI8H0aEdjcZld+Q2jJ5A5ADvPqfb6lzQ/sg==","license": "Apache-2.0","peerDependencies": {"@bufbuild/protobuf": "^2.7.0",- "@connectrpc/connect": "2.1.2"+ "@connectrpc/connect": "2.2.0"}},"node_modules/@earendil-works/pi-ai": {@@ -7153,14 +7153,14 @@}},"node_modules/e2b": {- "version": "2.46.1",- "resolved": "https://registry.npmjs.org/e2b/-/e2b-2.46.1.tgz",- "integrity": "sha512-OqYovS2oFrt4mk737CgfW/RoMadBYK84l5qjKpvbEoOB9KKxaZIXm7YUwOKSRTlijrrwDRX7oZlyPoVXiCpyTw==",+ "version": "2.51.0",+ "resolved": "https://registry.npmjs.org/e2b/-/e2b-2.51.0.tgz",+ "integrity": "sha512-P+50lx1nOYpgCFyE1xuz6JVN/TqJivMUiH7v+RW0CUSfb4+NCu8IkC5KruT2sNPgp6aexddQQ77+OZKGpZZJ+Q==","license": "MIT","dependencies": {- "@bufbuild/protobuf": "^2.14.0",- "@connectrpc/connect": "^2.1.2",- "@connectrpc/connect-web": "^2.1.2",+ "@bufbuild/protobuf": "^2.14.1",+ "@connectrpc/connect": "^2.2.0",+ "@connectrpc/connect-web": "^2.2.0","chalk": "^5.3.0","compare-versions": "^6.1.0","dockerfile-ast": "^0.7.1",@@ -7168,13 +7168,13 @@"openapi-fetch": "^0.14.1","platform": "^1.3.6","tar": "^7.5.22",- "undici": "^7.29.0"+ "undici": "^7.29.1"},"engines": {"node": ">=20.18.1 <21 || >=22"},"optionalDependencies": {- "undici8": "npm:[email protected]"+ "undici8": "npm:[email protected]"}},"node_modules/ecdsa-sig-formatter": {@@ -11318,9 +11318,9 @@},"node_modules/undici8": {"name": "undici",- "version": "8.10.0",- "resolved": "https://registry.npmjs.org/undici/-/undici-8.10.0.tgz",- "integrity": "sha512-HvltHd7avK13QIw/oLe4qoOLyoVSoafqJ2jYOrtMRBkbYT31eiBQ8O0ehRKZiEZCMEyLFQNIADpgCWC5fALvYQ==",+ "version": "8.10.2",+ "resolved": "https://registry.npmjs.org/undici/-/undici-8.10.2.tgz",+ "integrity": "sha512-/y4/bH9YNU5hi9NIrpOuvGXFcxrj3CMrV+/AYpowAYTpHn8gX/XPFjNy766FPoYY0miQhdW977JFWKGNhBdwyQ==","license": "MIT","optional": true,"engines": {diff --git a/package.json b/package.jsonindex 492d0c168..07dec9908 100644--- a/package.json+++ b/package.json@@ -41,6 +41,7 @@"smoke:aws-sandbox": "node scripts/aws-sandbox-smoke.ts","smoke:local-sandbox": "node scripts/local-sandbox-smoke.ts","build:modal-image": "node scripts/modal-build-image.ts",+ "build:e2b-template": "node scripts/e2b-build-template.ts","build:aws-image": "node scripts/aws-build-sandbox-image.ts","sandbox:local:build": "bash scripts/local-sandbox-build.sh","sandbox:porter:build": "docker build --platform linux/amd64 -t \"${PORTER_SANDBOX_IMAGE:?Set PORTER_SANDBOX_IMAGE to the registry tag to publish}\" porter && docker push \"$PORTER_SANDBOX_IMAGE\"",@@ -85,7 +86,7 @@"@smithy/signature-v4": "^5.6.6","@superserve/sdk": "^0.8.3","croner": "^10.0.1",- "e2b": "^2.39.0",+ "e2b": "^2.51.0","emoji-datasource": "^16.0.0","fastify": "^5.12.1","jose": "^6.2.3",diff --git a/scripts/e2b-build-template.ts b/scripts/e2b-build-template.tsnew file mode 100644index 000000000..19a415f6c--- /dev/null+++ b/scripts/e2b-build-template.ts@@ -0,0 +1,46 @@+import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { Template, defaultBuildLogger, type CopyItem } from "e2b";+import { loadDeploymentLayer } from "../src/deployment/load-layer.ts";++const positiveInt = (name: string): number | undefined => {+ const raw = process.env[name];+ if (!raw) return undefined;+ const n = Number(raw);+ if (!Number.isInteger(n) || n <= 0) throw new Error(`${name} must be a positive integer, got ${JSON.stringify(raw)}`);+ return n;+};++const baseName = process.env.E2B_TEMPLATE_NAME ?? "qm-base";+const cpuCount = positiveInt("E2B_TEMPLATE_CPUS");+const memoryMB = positiveInt("E2B_TEMPLATE_MEMORY_MB");+const resources = { ...(cpuCount ? { cpuCount } : {}), ...(memoryMB ? { memoryMB } : {}) };+const dockerfile = readFileSync(new URL("../deploy/e2b/e2b.Dockerfile", import.meta.url), "utf8");++const base = await Template.build(Template().fromDockerfile(dockerfile), baseName, {+ ...resources,+ onBuildLogs: defaultBuildLogger(),+});+console.log(`E2B_TEMPLATE_ID=${base.name} (templateId ${base.templateId}, build ${base.buildId})`);++const layerDir = process.env.DEPLOYMENT_LAYER;+if (layerDir) {+ const files = loadDeploymentLayer(layerDir).installFiles;+ if (files.length === 0) {+ console.log(`DEPLOYMENT_LAYER=${layerDir} declares no install files; no layer template built`);+ } else {+ const staging = mkdtempSync(join(tmpdir(), "e2b-layer-"));+ const items: CopyItem[] = files.map((file, index) => {+ writeFileSync(join(staging, String(index)), file.content);+ return { src: String(index), dest: file.to, mode: parseInt(file.mode, 8), user: "user" };+ });+ const layerName = process.env.E2B_LAYER_TEMPLATE_NAME ?? `${baseName}-layer`;+ const layered = await Template.build(+ Template({ fileContextPath: staging }).fromTemplate(base.name).copyItems(items),+ layerName,+ { ...resources, onBuildLogs: defaultBuildLogger() },+ );+ console.log(`E2B_TEMPLATE_ID=${layered.name} (templateId ${layered.templateId}, build ${layered.buildId})`);+ }+}diff --git a/src/config.ts b/src/config.tsindex 5a1d5440f..6c8caaa44 100644--- a/src/config.ts+++ b/src/config.ts@@ -380,6 +380,8 @@ interface E2bSandboxEnv {egressProxyUrl?: string;snapshotS3Bucket?: string;snapshotIntervalSec?: number;+ nativeSnapshotIntervalSec?: number;+ maxLifetimeSec?: number;defaultTimeoutSec?: number;}@@ -391,12 +393,23 @@ function e2bSandboxEnv(env: NodeJS.ProcessEnv): E2bSandboxEnv {...(numEnvStrict("E2B_SANDBOX_TTL_SEC", env.E2B_SANDBOX_TTL_SEC) !== undefined? { sandboxTtlSec: numEnvStrict("E2B_SANDBOX_TTL_SEC", env.E2B_SANDBOX_TTL_SEC) }: {}),+ ...(numEnvStrict("E2B_MAX_LIFETIME_SEC", env.E2B_MAX_LIFETIME_SEC) !== undefined+ ? { maxLifetimeSec: numEnvStrict("E2B_MAX_LIFETIME_SEC", env.E2B_MAX_LIFETIME_SEC) }+ : {}),...(env.E2B_PROXY ? { proxy: env.E2B_PROXY } : {}),...(env.E2B_EGRESS_PROXY_URL ? { egressProxyUrl: env.E2B_EGRESS_PROXY_URL } : {}),...(env.E2B_SNAPSHOT_S3_BUCKET ? { snapshotS3Bucket: env.E2B_SNAPSHOT_S3_BUCKET } : {}),...(numEnvStrict("E2B_SNAPSHOT_INTERVAL_SEC", env.E2B_SNAPSHOT_INTERVAL_SEC) !== undefined? { snapshotIntervalSec: numEnvStrict("E2B_SNAPSHOT_INTERVAL_SEC", env.E2B_SNAPSHOT_INTERVAL_SEC) }: {}),+ ...(numEnvStrict("E2B_NATIVE_SNAPSHOT_INTERVAL_SEC", env.E2B_NATIVE_SNAPSHOT_INTERVAL_SEC) !== undefined+ ? {+ nativeSnapshotIntervalSec: numEnvStrict(+ "E2B_NATIVE_SNAPSHOT_INTERVAL_SEC",+ env.E2B_NATIVE_SNAPSHOT_INTERVAL_SEC,+ ),+ }+ : {}),...(numEnvStrict("SANDBOX_TIMEOUT_SEC", env.SANDBOX_TIMEOUT_SEC) !== undefined? { defaultTimeoutSec: numEnvStrict("SANDBOX_TIMEOUT_SEC", env.SANDBOX_TIMEOUT_SEC) }: {}),@@ -1181,6 +1194,11 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {"[config] SANDBOX_BACKEND=sprites without SPRITES_EGRESS_PROXY_URL — sandboxes run with NO egress enforcement (fail-open); set SPRITES_EGRESS_PROXY_URL to the public egress proxy to force sandbox traffic through it.",);}+ if (env.SANDBOX_BACKEND === "e2b" && !env.E2B_EGRESS_PROXY_URL) {+ console.warn(+ "[config] SANDBOX_BACKEND=e2b without E2B_EGRESS_PROXY_URL — sandboxes run with NO egress enforcement (fail-open); set E2B_EGRESS_PROXY_URL to the public egress proxy so E2B's network rules admit only that host.",+ );+ }if (env.MODAL_TOKEN_ID && env.MODAL_TOKEN_SECRET && !env.MODAL_EGRESS_PROXY_URL) {console.warn("[config] modal sandbox backend enabled without MODAL_EGRESS_PROXY_URL — sandboxes run with NO egress enforcement (fail-open); set MODAL_EGRESS_PROXY_URL to the public https egress proxy so Modal's outbound allowlist admits only that host.",diff --git a/src/harness/agent-tools.ts b/src/harness/agent-tools.tsindex a53496f88..bf1533c6d 100644--- a/src/harness/agent-tools.ts+++ b/src/harness/agent-tools.ts@@ -936,6 +936,9 @@ export function createAgentTools(ref: ToolContextRef, opts?: AgentToolsOptions):const verdict = computerVerdict(s);const machineLine = s.listed && s.listed !== s.machine ? `${s.machine} (listed: ${s.listed})` : s.machine;const pressureLine = s.pressure ? `; io pressure: ${s.pressure.ioFull60}% (load ${s.pressure.load1})` : "";+ const resourcesLine = s.resources+ ? `; cpu ${s.resources.cpuUsedPct}%, memory ${s.resources.memUsedMb}/${s.resources.memTotalMb} MiB, disk ${s.resources.diskUsedGb}/${s.resources.diskTotalGb} GiB`+ : "";let shellLine = s.guestResponsive ? "answering" : `NOT answering${s.probeError ? ` (${s.probeError})` : ""}`;if (s.lifecycleState === "paused") shellLine = "paused (not probed)";const recoveryLines: string[] = [];@@ -962,7 +965,10 @@ export function createAgentTools(ref: ToolContextRef, opts?: AgentToolsOptions):callId,{ tool: "sandbox", action: "status", verdict, ...s },text(- [`machine: ${machineLine}; shell: ${shellLine}${pressureLine}${verdictLine}`, ...recoveryLines].join("\n"),+ [+ `machine: ${machineLine}; shell: ${shellLine}${pressureLine}${resourcesLine}${verdictLine}`,+ ...recoveryLines,+ ].join("\n"),),);} catch (e) {diff --git a/src/sandbox/e2b-client.ts b/src/sandbox/e2b-client.tsindex 51a2fb04d..23f026b77 100644--- a/src/sandbox/e2b-client.ts+++ b/src/sandbox/e2b-client.ts@@ -8,13 +8,25 @@ interface E2bRunOpts {timeoutMs?: number;}+export interface E2bMetrics {+ cpuUsedPct: number;+ cpuCount: number;+ memUsedBytes: number;+ memTotalBytes: number;+ diskUsedBytes: number;+ diskTotalBytes: number;+}+export interface E2bSession {readonly sandboxId: string;runCommand(command: string, opts?: E2bRunOpts): Promise<E2bCommandResult>;readFileBytes(absPath: string): Promise<Uint8Array | null>;writeFileBytes(absPath: string, data: Uint8Array): Promise<void>;+ keepAlive(ms: number): Promise<void>;pause(): Promise<void>;+ createSnapshot(): Promise<{ snapshotId: string }>;+ metrics(): Promise<E2bMetrics | null>;kill(): Promise<void>;}@@ -24,6 +36,14 @@ interface E2bSandboxSummary {metadata?: Record<string, string>;}+export interface E2bSandboxInfo {+ state: string;+ expiresAtMs: number;+ onTimeout?: string;+ cpuCount?: number;+ memoryMb?: number;+}+export class E2bSandboxGoneError extends Error {constructor(sandboxId: string, detail: string) {super(`e2b sandbox ${sandboxId} is gone: ${detail}`);@@ -31,15 +51,25 @@ export class E2bSandboxGoneError extends Error {}}+export class E2bCommandLostError extends Error {+ constructor(sandboxId: string, detail: string) {+ super(+ `e2b sandbox ${sandboxId} was lost while a command was running (${detail}); the command may have partially executed and was not retried`,+ );+ this.name = "E2bCommandLostError";+ }+}+interface E2bCreateOpts {metadata: Record<string, string>;timeoutMs?: number;autoPause?: boolean;+ fromSnapshot?: string;}export interface E2bClient {readonly nativePause?: boolean;- info?(sandboxId: string): Promise<{ state: string; expiresAtMs: number; onTimeout?: string }>;+ info?(sandboxId: string): Promise<E2bSandboxInfo>;create(opts: E2bCreateOpts): Promise<E2bSession>;@@ -48,6 +78,8 @@ export interface E2bClient {list(metadata: Record<string, string>): Promise<E2bSandboxSummary[]>;kill(sandboxId: string): Promise<void>;++ deleteSnapshot(snapshotId: string): Promise<void>;}export interface SdkE2bClientOptions {@@ -56,9 +88,13 @@ export interface SdkE2bClientOptions {sandboxTtlMs?: number;+ maxLifetimeMs?: number;+proxy?: string;maxCommandMs?: number;++ egressProxyUrl?: string;}interface SdkCommandResultLike {@@ -67,9 +103,44 @@ interface SdkCommandResultLike {exitCode?: number;}+export interface E2bEgressNetwork {+ allowOut: string[];+ denyOut: string[];+}+const DEFAULT_TEMPLATE = "base";const DEFAULT_TTL_MS = 15 * 60_000;const DEFAULT_MAX_COMMAND_MS = 3600_000;+const E2B_PRO_MAX_LIFETIME_MS = 24 * 3600_000;+export const E2B_EXEC_MARGIN_MS = 60_000;+const ALL_TRAFFIC = "0.0.0.0/0";++const HOSTNAME_MATCHED_PORTS: Record<string, string> = { "http:": "80", "https:": "443" };++export function e2bEgressNetwork(egressProxyUrl: string): E2bEgressNetwork {+ const url = new URL(egressProxyUrl);+ const host = url.hostname.replace(/^\[|\]$/g, "");+ const port = url.port || HOSTNAME_MATCHED_PORTS[url.protocol] || "";+ const ipLiteral = /^[\d.]+$/.test(host) || host.includes(":");+ if (!ipLiteral && port !== "80" && port !== "443")+ throw new Error(+ `E2B egress proxy ${egressProxyUrl}: E2B matches allowed hostnames only on ports 80 and 443; give the proxy on one of those ports or by IP address`,+ );+ return { allowOut: [host], denyOut: [ALL_TRAFFIC] };+}++function e2bSandboxTtlMs(opts: Pick<SdkE2bClientOptions, "sandboxTtlMs" | "maxLifetimeMs" | "maxCommandMs">): {+ sandboxTtlMs: number;+ maxLifetimeMs: number;+ maxCommandMs: number;+} {+ const maxLifetimeMs = opts.maxLifetimeMs ?? E2B_PRO_MAX_LIFETIME_MS;+ if (!Number.isFinite(maxLifetimeMs) || maxLifetimeMs <= E2B_EXEC_MARGIN_MS)+ throw new Error(`e2b maxLifetimeMs must be a finite number of milliseconds above ${E2B_EXEC_MARGIN_MS}`);+ const maxCommandMs = Math.min(opts.maxCommandMs ?? DEFAULT_MAX_COMMAND_MS, maxLifetimeMs - E2B_EXEC_MARGIN_MS);+ const requested = opts.sandboxTtlMs ?? Math.max(DEFAULT_TTL_MS, maxCommandMs + E2B_EXEC_MARGIN_MS);+ return { sandboxTtlMs: Math.min(maxLifetimeMs, requested), maxLifetimeMs, maxCommandMs };+}function commandResultFrom(value: unknown): E2bCommandResult | null {const r = value as SdkCommandResultLike & { result?: SdkCommandResultLike };@@ -80,32 +151,24 @@ function commandResultFrom(value: unknown): E2bCommandResult | null {return { stdout: source.stdout ?? "", stderr: source.stderr ?? "", exitCode: source.exitCode ?? -1 };}-function isGoneError(err: unknown): boolean {- const msg = String((err as Error)?.message ?? err);- const name = String((err as Error)?.name ?? "");- return (- name.endsWith("NotFoundError") ||- /not found|not running anymore|does not exist|invalid sandbox id|sandbox was not found|410|expired/i.test(msg)- );-}+type E2bSdk = typeof import("e2b");++const SANDBOX_TIMEOUT_MESSAGE = /sandbox timeout|end of life|not running anymore/i;-function isSandboxGoneError(err: unknown): boolean {- const msg = String((err as Error)?.message ?? err);- const name = String((err as Error)?.name ?? "");- if (name === "SandboxNotFoundError") return true;- return /not running anymore|sandbox (was )?not found|invalid sandbox id|sandbox.*(does not exist|expired)/i.test(msg);+function isSandboxGone(sdk: E2bSdk, err: unknown): boolean {+ if (err instanceof sdk.SandboxNotFoundError) return true;+ return err instanceof sdk.TimeoutError && SANDBOX_TIMEOUT_MESSAGE.test(err.message);}export function createSdkE2bClient(opts: SdkE2bClientOptions): E2bClient {const templateId = opts.templateId ?? DEFAULT_TEMPLATE;- const sandboxTtlMs = opts.sandboxTtlMs ?? DEFAULT_TTL_MS;- const maxCommandMs = opts.maxCommandMs ?? DEFAULT_MAX_COMMAND_MS;+ const { sandboxTtlMs, maxLifetimeMs, maxCommandMs } = e2bSandboxTtlMs(opts);+ const network = opts.egressProxyUrl ? e2bEgressNetwork(opts.egressProxyUrl) : undefined;const common = {apiKey: opts.apiKey,...(opts.proxy ? { proxy: opts.proxy } : {}),};- type E2bSdk = typeof import("e2b");let sdk: Promise<E2bSdk> | null = null;const loadSdk = (): Promise<E2bSdk> => {const loaded = (sdk ??= import("e2b").then(@@ -120,87 +183,139 @@ export function createSdkE2bClient(opts: SdkE2bClientOptions): E2bClient {type SdkSandbox = Awaited<ReturnType<E2bSdk["Sandbox"]["create"]>>;- const wrap = (sbx: SdkSandbox): E2bSession => ({- sandboxId: sbx.sandboxId,- async runCommand(command, runOpts): Promise<E2bCommandResult> {- const timeoutMs = runOpts?.timeoutMs ?? maxCommandMs;+ const wrap = (lib: E2bSdk, sbx: SdkSandbox, ttlMs: number): E2bSession => {+ let expiresAtMs = Date.now() + ttlMs;+ const gone = (err: unknown): unknown =>+ isSandboxGone(lib, err) ? new E2bSandboxGoneError(sbx.sandboxId, String((err as Error).message)) : err;+ const extend = async (ms: number): Promise<void> => {+ const bounded = Math.min(maxLifetimeMs, Math.max(ms, expiresAtMs - Date.now()));+ const at = Date.now() + bounded;try {- const r = await sbx.commands.run(command, {- timeoutMs,-- requestTimeoutMs: timeoutMs + 30_000,- });- return commandResultFrom(r) ?? { stdout: "", stderr: "", exitCode: -1 };+ await sbx.setTimeout(bounded);} catch (err) {- const asResult = commandResultFrom(err);- if (asResult) return asResult;- if (isGoneError(err)) throw new E2bSandboxGoneError(sbx.sandboxId, String((err as Error).message));- throw err;+ throw gone(err);}- },- async readFileBytes(absPath): Promise<Uint8Array | null> {- try {- const data = await sbx.files.read(absPath, { format: "bytes" });- return data instanceof Uint8Array ? data : new Uint8Array(data as ArrayBuffer);- } catch (err) {- if (isSandboxGoneError(err)) throw new E2bSandboxGoneError(sbx.sandboxId, String((err as Error).message));- if (- String((err as Error)?.name) === "NotFoundError" ||- /not found|no such file/i.test(String((err as Error)?.message))- )- return null;- throw err;- }- },- async writeFileBytes(absPath, data): Promise<void> {- const buf = new ArrayBuffer(data.byteLength);- new Uint8Array(buf).set(data);- try {- await sbx.files.write(absPath, buf);- } catch (err) {- if (isSandboxGoneError(err)) throw new E2bSandboxGoneError(sbx.sandboxId, String((err as Error).message));- throw err;- }- },- async pause(): Promise<void> {- await sbx.pause({ keepMemory: true });- },- async kill(): Promise<void> {- await sbx.kill().catch((err) => {- if (!isGoneError(err)) throw err;- });- },- });+ expiresAtMs = at;+ };+ const cover = async (commandMs: number): Promise<void> => {+ const needed = Math.min(commandMs, maxCommandMs) + E2B_EXEC_MARGIN_MS;+ if (Date.now() + needed <= expiresAtMs) return;+ await extend(Math.max(ttlMs, needed));+ };+ return {+ sandboxId: sbx.sandboxId,+ async runCommand(command, runOpts): Promise<E2bCommandResult> {+ const timeoutMs = runOpts?.timeoutMs ?? maxCommandMs;+ await cover(timeoutMs);+ let handle: Awaited<ReturnType<SdkSandbox["commands"]["run"]>>;+ try {+ handle = await sbx.commands.run(command, { background: true, timeoutMs });+ } catch (err) {+ if (isSandboxGone(lib, err)) throw new E2bCommandLostError(sbx.sandboxId, String((err as Error).message));+ throw err;+ }+ try {+ return commandResultFrom(await handle.wait()) ?? { stdout: "", stderr: "", exitCode: -1 };+ } catch (err) {+ const asResult = commandResultFrom(err);+ if (asResult) return asResult;+ if (isSandboxGone(lib, err)) throw new E2bCommandLostError(sbx.sandboxId, String((err as Error).message));+ throw err;+ }+ },+ async readFileBytes(absPath): Promise<Uint8Array | null> {+ try {+ const data = await sbx.files.read(absPath, { format: "bytes" });+ return data instanceof Uint8Array ? data : new Uint8Array(data as ArrayBuffer);+ } catch (err) {+ if (err instanceof lib.FileNotFoundError) return null;+ throw gone(err);+ }+ },+ async writeFileBytes(absPath, data): Promise<void> {+ const buf = new ArrayBuffer(data.byteLength);+ new Uint8Array(buf).set(data);+ try {+ await sbx.files.write(absPath, buf);+ } catch (err) {+ throw gone(err);+ }+ },+ keepAlive: (ms) => extend(ms),+ async pause(): Promise<void> {+ try {+ await sbx.pause({ keepMemory: true });+ } catch (err) {+ throw gone(err);+ }+ },+ async createSnapshot(): Promise<{ snapshotId: string }> {+ try {+ const info = await sbx.createSnapshot();+ return { snapshotId: info.snapshotId };+ } catch (err) {+ throw gone(err);+ }+ },+ async metrics(): Promise<E2bMetrics | null> {+ const rows = await sbx.getMetrics();+ const last = rows.at(-1);+ if (!last) return null;+ return {+ cpuUsedPct: last.cpuUsedPct,+ cpuCount: last.cpuCount,+ memUsedBytes: last.memUsed,+ memTotalBytes: last.memTotal,+ diskUsedBytes: last.diskUsed,+ diskTotalBytes: last.diskTotal,+ };+ },+ async kill(): Promise<void> {+ await sbx.kill().catch((err) => {+ if (!isSandboxGone(lib, err)) throw err;+ });+ },+ };+ };return {nativePause: true,async info(sandboxId) {- const { Sandbox } = await loadSdk();+ const lib = await loadSdk();try {- const info = await Sandbox.getInfo(sandboxId, common);- return { state: info.state, expiresAtMs: info.endAt.getTime(), onTimeout: info.lifecycle?.onTimeout };+ const info = await lib.Sandbox.getInfo(sandboxId, common);+ return {+ state: info.state,+ expiresAtMs: info.endAt.getTime(),+ onTimeout: info.lifecycle?.onTimeout,+ cpuCount: info.cpuCount,+ memoryMb: info.memoryMB,+ };} catch (err) {- if (isGoneError(err)) throw new E2bSandboxGoneError(sandboxId, String((err as Error).message));+ if (isSandboxGone(lib, err)) throw new E2bSandboxGoneError(sandboxId, String((err as Error).message));throw err;}},async create(createOpts): Promise<E2bSession> {- const { Sandbox } = await loadSdk();- const sbx = await Sandbox.create(templateId, {+ const lib = await loadSdk();+ const timeoutMs = Math.min(maxLifetimeMs, createOpts.timeoutMs ?? sandboxTtlMs);+ const sbx = await lib.Sandbox.create(createOpts.fromSnapshot ?? templateId, {...common,- timeoutMs: createOpts.timeoutMs ?? sandboxTtlMs,+ timeoutMs,metadata: createOpts.metadata,lifecycle: { onTimeout: createOpts.autoPause ? "pause" : "kill", autoResume: false },+ ...(network ? { network } : {}),});- return wrap(sbx);+ return wrap(lib, sbx, timeoutMs);},async connect(sandboxId): Promise<E2bSession> {- const { Sandbox } = await loadSdk();+ const lib = await loadSdk();try {- const sbx = await Sandbox.connect(sandboxId, { ...common, timeoutMs: sandboxTtlMs });- return wrap(sbx);+ const sbx = await lib.Sandbox.connect(sandboxId, { ...common, timeoutMs: sandboxTtlMs });+ if (network) await sbx.updateNetwork(network);+ return wrap(lib, sbx, sandboxTtlMs);} catch (err) {- if (isGoneError(err)) throw new E2bSandboxGoneError(sandboxId, String((err as Error).message));+ if (isSandboxGone(lib, err)) throw new E2bSandboxGoneError(sandboxId, String((err as Error).message));throw err;}},@@ -220,10 +335,14 @@ export function createSdkE2bClient(opts: SdkE2bClientOptions): E2bClient {return out;},async kill(sandboxId): Promise<void> {- const { Sandbox } = await loadSdk();- await Sandbox.kill(sandboxId, common).catch((err) => {- if (!isGoneError(err)) throw err;+ const lib = await loadSdk();+ await lib.Sandbox.kill(sandboxId, common).catch((err) => {+ if (!isSandboxGone(lib, err)) throw err;});},+ async deleteSnapshot(snapshotId): Promise<void> {+ const { Sandbox } = await loadSdk();+ await Sandbox.deleteSnapshot(snapshotId, common);+ },};}diff --git a/src/sandbox/e2b-sandbox.ts b/src/sandbox/e2b-sandbox.tsindex 8c1f55b79..bfdbe4c57 100644--- a/src/sandbox/e2b-sandbox.ts+++ b/src/sandbox/e2b-sandbox.ts@@ -27,7 +27,13 @@ import {import { killableScript, killScript } from "./exec-kill.ts";import { visibleNotInstalled, visibleTools } from "./sandbox.ts";import { sandboxScopeName } from "./exec-sandbox-base.ts";-import { E2bSandboxGoneError, type E2bClient, type E2bSession } from "./e2b-client.ts";+import {+ E2bCommandLostError,+ E2bSandboxGoneError,+ type E2bClient,+ type E2bSandboxInfo,+ type E2bSession,+} from "./e2b-client.ts";import {createHomeSnapshotOps,createMemorySnapshotStore,@@ -54,12 +60,16 @@ const HOME_TAR = `${HOME_DIR}/.qm-home.tar`;const IN_MEMORY_ADOPT_MAX_BYTES = 256 * 1024 * 1024;const SNAPSHOT_PRUNE = HOME_SNAPSHOT_PRUNE;+const DEFAULT_KEEP_WARM_SEC = 3600;+const DEFAULT_NATIVE_SNAPSHOT_INTERVAL_MS = 5 * 60_000;export interface StoredE2bSandbox {sandboxId: string;nativePause?: boolean;preservationState?: "running" | "paused" | "pause_failed";preservationError?: string;+ recoverySnapshotId?: string;+ recoveryError?: string;createdAtMs: number;lastSnapshotMs?: number;homeDirty?: boolean;@@ -69,8 +79,10 @@ export interface E2bSandboxOptions extends BlobStagingOptions {client: E2bClient;namePrefix?: string;defaultTimeoutSec?: number;+ keepWarmSec?: number;snapshotIntervalMs?: number;+ nativeSnapshotIntervalMs?: number;egressProxyUrl?: string;extraTools?: string[];credentialPaths?: CredentialPathSpec[];@@ -84,7 +96,14 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpticonst client = opts.client;const prefix = opts.namePrefix ?? "qm";const defaultTimeoutSec = opts.defaultTimeoutSec ?? 600;+ const keepWarmMs = (opts.keepWarmSec ?? DEFAULT_KEEP_WARM_SEC) * 1000;const snapshotIntervalMs = opts.snapshotIntervalMs ?? 0;+ const nativeSnapshotIntervalMs = opts.nativeSnapshotIntervalMs ?? DEFAULT_NATIVE_SNAPSHOT_INTERVAL_MS;+ const observed: { cpus?: number; memoryMb?: number; diskGb?: number } = {};+ const noteInfo = (info: E2bSandboxInfo | undefined): void => {+ if (info?.cpuCount) observed.cpus = info.cpuCount;+ if (info?.memoryMb) observed.memoryMb = info.memoryMb;+ };const workspaceDir = `${HOME_DIR}/${WORKSPACE_BASENAME}`;const store = opts.store ?? createMemoryMap<StoredE2bSandbox>();const snapshots = opts.snapshots ?? createMemorySnapshotStore();@@ -139,6 +158,7 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptitry {const session = await client.connect(stored.sandboxId);const info = await client.info?.(session.sandboxId);+ noteInfo(info);await store.merge(scope, {preservationState: "running",...(info ? { nativePause: info.onTimeout === "pause" } : {}),@@ -154,7 +174,9 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptitry {const session = await client.connect(summary.sandboxId);const info = await client.info?.(session.sandboxId);+ noteInfo(info);await store.put(scope, {+ ...stored,sandboxId: session.sandboxId,createdAtMs: Date.now(),nativePause: info?.onTimeout === "pause",@@ -166,7 +188,35 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti}if (stored?.nativePause) {- throw new Error("e2b sandbox is gone; explicitly import a recovery snapshot before replacing its home");+ if (!stored.recoverySnapshotId)+ throw new Error("e2b sandbox is gone; explicitly import a recovery snapshot before replacing its home");+ try {+ onStatus?.("Restoring the sandbox from its recovery snapshot…");+ } catch (error) {+ void error;+ }+ let session: E2bSession;+ try {+ session = await client.create({+ metadata: { name },+ autoPause: true,+ fromSnapshot: stored.recoverySnapshotId,+ });+ } catch (e) {+ await store.merge(scope, { recoveryError: errMessage(e) });+ reportError("sandbox_hydrate", "snapshot_restore_failed", errMessage(e), scope);+ throw new Error(`e2b provision: restore from recovery snapshot ${stored.recoverySnapshotId} failed`, {+ cause: e,+ });+ }+ await store.merge(scope, {+ sandboxId: session.sandboxId,+ createdAtMs: Date.now(),+ preservationState: "running",+ preservationError: undefined,+ recoveryError: undefined,+ });+ return adopt(session);}try {onStatus?.("Creating the sandbox…");@@ -193,6 +243,7 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti});}await store.merge(scope, { nativePause: client.nativePause });+ noteInfo(await client.info?.(session.sandboxId).catch(() => undefined));return { session, coldStart: !hydrated };});}@@ -226,6 +277,7 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptitry {return await action(first.session);} catch (err) {+ if (err instanceof E2bCommandLostError) sessionByName.delete(name);if (!(err instanceof E2bSandboxGoneError)) throw err;sessionByName.delete(name);const second =@@ -249,7 +301,7 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptibackend: "e2b",writablePersistence: client.nativePause ? "provider_managed" : "snapshot_to_workspace",processSessions: true,- egressEnforcement: "none",+ egressEnforcement: opts.egressProxyUrl ? "domain" : "none",spec: {os: "Linux — E2B Firecracker sandbox (provider pause preserves state; publish durable work to git or Files)",runtimes: ["Node", "Python 3"],@@ -259,6 +311,15 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptiget notInstalled() {return visibleNotInstalled(["gh", "aws", "gcloud", "kubectl", "flyctl", "glab"], opts.extraTools ?? []);},+ get cpus() {+ return observed.cpus;+ },+ get memoryMb() {+ return observed.memoryMb;+ },+ get diskGb() {+ return observed.diskGb;+ },homeDir: HOME_DIR,workdir: workspaceDir,},@@ -309,6 +370,28 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptiawait store.delete(scope);sessionByName.delete(name);scopeByName.delete(name);+ await forgetSnapshot(stored?.recoverySnapshotId);+ }++ const forgetSnapshot = (snapshotId: string | undefined): Promise<void> =>+ snapshotId+ ? client.deleteSnapshot(snapshotId).catch(swallowAs("e2b-sandbox: recovery snapshot delete", undefined))+ : Promise.resolve();++ async function captureRecoverySnapshot(scope: string, session: E2bSession, previous?: string): Promise<void> {+ try {+ const { snapshotId } = await session.createSnapshot();+ await store.merge(scope, {+ recoverySnapshotId: snapshotId,+ recoveryError: undefined,+ lastSnapshotMs: Date.now(),+ homeDirty: false,+ });+ if (previous !== snapshotId) await forgetSnapshot(previous);+ } catch (e) {+ await store.merge(scope, { recoveryError: errMessage(e) });+ reportError("sandbox_snapshot", "recovery_snapshot_failed", errMessage(e), scope);+ }}const sandbox: Sandbox = {@@ -451,6 +534,7 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptiif (session) await session.kill().catch(killGone);else if (stored) await client.kill(stored.sandboxId).catch(killGone);await store.delete(scopeId);+ await forgetSnapshot(stored?.recoverySnapshotId);});},@@ -468,10 +552,12 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptiif (!stored) return { machine: "no sandbox provisioned yet", provisioned: false, guestResponsive: false };const machine = `e2b sandbox ${stored.sandboxId}`;let expiresAtMs: number | undefined;+ const recoveryError = stored.recoveryError ?? stored.preservationError;const recovery = {strategy: stored.nativePause ? ("provider_pause" as const) : ("workspace_snapshot" as const),state: stored.preservationState,- ...(stored.preservationError ? { error: stored.preservationError } : {}),+ ...(recoveryError ? { error: recoveryError } : {}),+ ...(stored.recoverySnapshotId ? { checkpointId: stored.recoverySnapshotId, checkpointExpiresAtMs: null } : {}),...(stored.lastSnapshotMs ? { checkpointAtMs: stored.lastSnapshotMs } : {}),};try {@@ -495,12 +581,25 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptisessionByName.set(name, session);}const r = await session.runCommand("echo responsive", { timeoutMs: 30_000 });+ const metrics = await session.metrics().catch(swallowAs("e2b-sandbox: metrics", null));+ if (metrics?.diskTotalBytes) observed.diskGb = Math.round(metrics.diskTotalBytes / 2 ** 30);return {machine,expiresAtMs,recovery,provisioned: true,guestResponsive: r.exitCode === 0 && /responsive/.test(r.stdout),+ ...(metrics+ ? {+ resources: {+ cpuUsedPct: metrics.cpuUsedPct,+ memUsedMb: Math.round(metrics.memUsedBytes / 2 ** 20),+ memTotalMb: Math.round(metrics.memTotalBytes / 2 ** 20),+ diskUsedGb: Math.round((metrics.diskUsedBytes / 2 ** 30) * 10) / 10,+ diskTotalGb: Math.round((metrics.diskTotalBytes / 2 ** 30) * 10) / 10,+ },+ }+ : {}),};} catch (e) {return {@@ -547,14 +646,19 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOptireportError("sandbox_snapshot", "teardown_snapshot_failed", errMessage(e), scope);}}- if (tdOpts?.keepWarm) return;+ if (tdOpts?.keepWarm) {+ try {+ await session.keepAlive(keepWarmMs);+ } catch (e) {+ reportError("sandbox_preservation", "keep_warm_failed", errMessage(e), scope);+ }+ return;+ }+ if (stored?.nativePause && snapshotDue(stored, tdOpts, nativeSnapshotIntervalMs))+ await captureRecoverySnapshot(scope, session, stored.recoverySnapshotId);try {await session.pause();- await store.merge(scope, {- preservationState: "paused",- preservationError: undefined,- ...(stored?.nativePause ? { homeDirty: false } : {}),- });+ await store.merge(scope, { preservationState: "paused", preservationError: undefined });sessionByName.delete(handle.id);} catch (error) {await store.merge(scope, { preservationState: "pause_failed", preservationError: errMessage(error) });diff --git a/src/sandbox/sandbox.ts b/src/sandbox/sandbox.tsindex c85e7f728..0229a3786 100644--- a/src/sandbox/sandbox.ts+++ b/src/sandbox/sandbox.ts@@ -156,6 +156,15 @@ export interface ComputerStatus {guestResponsive: boolean;probeError?: string;pressure?: ExecPressure;+ resources?: ComputerResources;+}++interface ComputerResources {+ cpuUsedPct: number;+ memUsedMb: number;+ memTotalMb: number;+ diskUsedGb: number;+ diskTotalGb: number;}export type ComputerVerdict = "ok" | "wedged" | "down";diff --git a/src/wiring.ts b/src/wiring.tsindex 3f2d68900..bbc54a362 100644--- a/src/wiring.ts+++ b/src/wiring.ts@@ -891,11 +891,17 @@ export function buildApp(apiKey: e2b.apiKey,...(e2b.templateId ? { templateId: e2b.templateId } : {}),...(e2b.sandboxTtlSec ? { sandboxTtlMs: e2b.sandboxTtlSec * 1000 } : {}),+ ...(e2b.maxLifetimeSec ? { maxLifetimeMs: e2b.maxLifetimeSec * 1000 } : {}),...(e2b.proxy ? { proxy: e2b.proxy } : {}),+ ...(e2b.egressProxyUrl ? { egressProxyUrl: e2b.egressProxyUrl } : {}),}),...(e2b.namePrefix ? { namePrefix: e2b.namePrefix } : {}),...(e2b.defaultTimeoutSec ? { defaultTimeoutSec: e2b.defaultTimeoutSec } : {}),+ keepWarmSec: Math.ceil(config.backgroundJobTtlMaxMs / 1000),...(e2b.snapshotIntervalSec !== undefined ? { snapshotIntervalMs: e2b.snapshotIntervalSec * 1000 } : {}),+ ...(e2b.nativeSnapshotIntervalSec !== undefined+ ? { nativeSnapshotIntervalMs: e2b.nativeSnapshotIntervalSec * 1000 }+ : {}),...(e2b.egressProxyUrl ? { egressProxyUrl: e2b.egressProxyUrl } : {}),extraTools: deploymentLayer.advertisedTools,credentialPaths: deploymentLayer.credentialPaths,diff --git a/test/e2b-sandbox.test.ts b/test/e2b-sandbox.test.tsindex fdcf78f3d..f78ff6b31 100644--- a/test/e2b-sandbox.test.ts+++ b/test/e2b-sandbox.test.ts@@ -36,6 +36,16 @@ beforeEach(() => {fake = installFakeE2b();sandbox = make();});++const nativeClient = (): typeof fake.client => ({+ ...fake.client,+ nativePause: true,+ async info() {+ const current = fake.current(scopeName());+ if (!current) throw new E2bSandboxGoneError("unknown", "sandbox was not found");+ return { state: current.state, expiresAtMs: Date.now() + 60_000, onTimeout: "pause" };+ },+});after(() => fake?.cleanup());test("provision runs commands with env and cwd", async () => {@@ -440,18 +450,161 @@ test("pause failures are durable and visible and leave the source available forassert.equal((await store.get(scope))?.preservationError, undefined);});-test("a lost native E2B sandbox requires explicit recovery instead of a blank replacement", async () => {+test("a lost native E2B sandbox without a recovery snapshot requires explicit recovery instead of a blank replacement", async () => {const store = createMemoryMap<StoredE2bSandbox>();- const client = { ...fake.client, nativePause: true };- const first = make({ client, store });+ const errors: string[] = [];+ const client = {+ ...nativeClient(),+ async create(options: Parameters<typeof fake.client.create>[0]) {+ const session = await fake.client.create(options);+ return {+ ...session,+ async createSnapshot(): Promise<{ snapshotId: string }> {+ throw new Error("snapshot quota exhausted");+ },+ };+ },+ };+ const first = make({ client, store, onError: (e: { code: string }) => errors.push(e.code) });const handle = await first.provision(layers);await first.teardown(handle);+ assert.equal(fake.current(scopeName())?.state, "paused", "a failed snapshot still lets the pause proceed");+ assert.equal((await store.get(scope))?.recoveryError, "snapshot quota exhausted");+ assert.deepEqual(errors, ["recovery_snapshot_failed"]);+ const status = await first.computerStatus!(scope);+ assert.equal(status.recovery?.error, "snapshot quota exhausted");fake.expirePaused();const restarted = make({ client, store });await assert.rejects(restarted.provision(layers), /explicitly import a recovery snapshot/);assert.equal(fake.createdCount(scopeName()), 1);});+test("a lost native E2B sandbox is replaced from its persistent recovery snapshot", async () => {+ const store = createMemoryMap<StoredE2bSandbox>();+ const client = nativeClient();+ const first = make({ client, store });+ const handle = await first.provision(layers);+ await first.writeFile(handle, "work.txt", "captured before the pause\n");+ await first.teardown(handle);+ const captured = (await store.get(scope))!;+ assert.ok(captured.recoverySnapshotId, "used-turn teardown captures a provider snapshot");+ assert.deepEqual(fake.snapshots(), [captured.recoverySnapshotId]);+ const paused = await first.computerStatus!(scope);+ assert.equal(paused.recovery?.checkpointId, captured.recoverySnapshotId);+ assert.equal(paused.recovery?.checkpointExpiresAtMs, null);+ fake.expirePaused();+ const restarted = make({ client, store });+ const revived = await restarted.provision(layers);+ assert.equal(revived.coldStart, false);+ assert.equal(fake.createdCount(scopeName()), 2);+ assert.equal(fake.restoredFrom(scopeName()), captured.recoverySnapshotId);+ assert.equal(await restarted.readFile(revived, "work.txt"), "captured before the pause\n");+ assert.equal((await store.get(scope))?.sandboxId, fake.current(scopeName())?.sandboxId);+ assert.equal((await store.get(scope))?.preservationState, "running");+});++test("recovery snapshots follow the native interval and supersede the previous capture", async () => {+ const store = createMemoryMap<StoredE2bSandbox>();+ const client = { ...fake.client, nativePause: true };+ const throttled = make({ client, store, nativeSnapshotIntervalMs: 60 * 60_000 });+ const a = await throttled.provision(layers);+ await throttled.teardown(a);+ const b = await throttled.provision(layers);+ await throttled.teardown(b);+ assert.equal(fake.snapshots().length, 1, "second teardown inside the interval skips the capture");+ const firstId = (await store.get(scope))!.recoverySnapshotId!;++ const eager = make({ client, store, nativeSnapshotIntervalMs: 0 });+ const c = await eager.provision(layers);+ await eager.teardown(c);+ const secondId = (await store.get(scope))!.recoverySnapshotId!;+ assert.notEqual(secondId, firstId);+ assert.deepEqual(fake.snapshots(), [secondId], "only the newest snapshot is retained");++ await eager.destroyScope!(scope);+ assert.deepEqual(fake.snapshots(), [], "destroying the scope deletes its snapshot");+});++test("legacy keepWarm teardown still takes its portable checkpoint", async () => {+ const counting = instrumentedSnapshotStore();+ const s = make({ snapshots: counting.store });+ const h = await s.provision(layers);+ await s.teardown(h, { keepWarm: true });+ assert.equal(counting.puts(), 1);+ assert.equal(fake.current(h.id)?.state, "running");+});++test("a failing snapshot delete never wedges destroy", async () => {+ const store = createMemoryMap<StoredE2bSandbox>();+ const client = {+ ...nativeClient(),+ async deleteSnapshot(): Promise<void> {+ throw new Error("snapshot API unavailable");+ },+ };+ const s = make({ client, store });+ const h = await s.provision(layers);+ await s.teardown(h);+ assert.ok((await store.get(scope))?.recoverySnapshotId);+ await s.destroyScope!(scope);+ assert.equal(await store.get(scope), null);+ assert.equal(fake.current(scopeName()), null);+});++test("keepWarm teardown extends the sandbox timeout to the keep-warm horizon", async () => {+ const s = make({ keepWarmSec: 7200 });+ const h = await s.provision(layers);+ await s.teardown(h, { keepWarm: true });+ assert.equal(fake.current(h.id)?.state, "running");+ assert.ok(fake.timeouts(h.id).includes(7200_000), `timeouts: ${fake.timeouts(h.id).join(",")}`);+});++test("a command whose sandbox is lost mid-flight fails loudly instead of running twice", async () => {+ const h = await sandbox.provision(layers);+ fake.loseNextCommand(h.id);+ const before = fake.execScripts().length;+ await assert.rejects(sandbox.run(h, "echo must-not-repeat"), /was lost while a command was running/);+ assert.equal(fake.execScripts().length, before, "the lost command was not re-run");+ const r = await sandbox.run(h, "echo back");+ assert.equal(r.stdout.trim(), "back", "the next command reconnects");+ assert.equal(fake.createdCount(scopeName()), 1);+});++test("an egress proxy is advertised as domain enforcement", () => {+ assert.equal(make({ egressProxyUrl: "https://proxy.example.com" }).profile.egressEnforcement, "domain");+ assert.equal(make().profile.egressEnforcement, "none");+});++test("computerStatus reports provider metrics and the profile learns the sandbox shape", async () => {+ const client = {+ ...fake.client,+ async info() {+ return { state: "running", expiresAtMs: Date.now() + 60_000, onTimeout: "pause", cpuCount: 2, memoryMb: 512 };+ },+ };+ const s = make({ client });+ fake.metrics = {+ cpuUsedPct: 12.5,+ cpuCount: 2,+ memUsedBytes: 256 * 2 ** 20,+ memTotalBytes: 512 * 2 ** 20,+ diskUsedBytes: 1.5 * 2 ** 30,+ diskTotalBytes: 20 * 2 ** 30,+ };+ await s.provision(layers);+ const status = await s.computerStatus!(scope);+ assert.deepEqual(status.resources, {+ cpuUsedPct: 12.5,+ memUsedMb: 256,+ memTotalMb: 512,+ diskUsedGb: 1.5,+ diskTotalGb: 20,+ });+ assert.equal(s.profile.spec?.cpus, 2);+ assert.equal(s.profile.spec?.memoryMb, 512);+ assert.equal(s.profile.spec?.diskGb, 20);+});+test("legacy pause preserves dirty home after failed portable checkpoint and retries on an unused turn", async () => {const store = createMemoryMap<StoredE2bSandbox>();const portable = instrumentedSnapshotStore();diff --git a/test/native-provider-clients.test.ts b/test/native-provider-clients.test.tsindex cfe1a5ea1..4e32a32c3 100644--- a/test/native-provider-clients.test.ts+++ b/test/native-provider-clients.test.ts@@ -2,6 +2,7 @@ import { spawnSync } from "node:child_process";import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";import { tmpdir } from "node:os";import { dirname, join } from "node:path";+import { createE2bSandbox } from "../src/sandbox/e2b-sandbox.ts";import { createModalSandbox } from "../src/sandbox/modal-sandbox.ts";import { createLocalWorkspaceStore } from "../src/workspace/workspace-store.ts";import { test, mock } from "node:test";@@ -12,7 +13,13 @@ import {MODAL_EXEC_GRACE_MS,MODAL_MAX_EXEC_ARG_BYTES,} from "../src/sandbox/modal-client.ts";-import { createSdkE2bClient } from "../src/sandbox/e2b-client.ts";+import {+ createSdkE2bClient,+ e2bEgressNetwork,+ E2B_EXEC_MARGIN_MS,+ E2bCommandLostError,+ E2bSandboxGoneError,+} from "../src/sandbox/e2b-client.ts";const modalCalls: unknown[][] = [];const modalCreateParams: Record<string, unknown>[] = [];@@ -68,23 +75,111 @@ mock.module("modal", {const e2bCalls: unknown[][] = [];let pauseError: Error | undefined;+let startError: Error | undefined;+let onDispatch: ((command: string) => void) | undefined;+let waitError: Error | undefined;+class FakeSandboxError extends Error {}+class FakeNotFoundError extends FakeSandboxError {}+class FakeSandboxNotFoundError extends FakeNotFoundError {}+class FakeFileNotFoundError extends FakeNotFoundError {}+class FakeTimeoutError extends FakeSandboxError {}+class FakeCommandExitError extends FakeSandboxError {+ readonly exitCode: number;+ readonly stdout: string;+ readonly stderr: string;+ constructor(exitCode: number, stdout: string, stderr: string) {+ super(`exit ${exitCode}`);+ this.exitCode = exitCode;+ this.stdout = stdout;+ this.stderr = stderr;+ }+}+const fakeE2bSandbox = (sandboxId: string) => ({+ sandboxId,+ async setTimeout(ms: number) {+ e2bCalls.push(["setTimeout", ms]);+ },+ async updateNetwork(network: unknown) {+ e2bCalls.push(["updateNetwork", network]);+ },+ commands: {+ async run(cmd: string, options: { background?: boolean }) {+ e2bCalls.push(["run", cmd, options]);+ if (options?.background !== true) throw new Error("the client must start commands in the background");+ onDispatch?.(cmd);+ if (startError) throw startError;+ return {+ pid: 7,+ async wait() {+ if (waitError) throw waitError;+ return { exitCode: 0, stdout: "ran", stderr: "" };+ },+ };+ },+ },+ files: {+ async read(path: string) {+ e2bCalls.push(["read", path]);+ throw new FakeFileNotFoundError(`${path} not found`);+ },+ },+ async pause(options: unknown) {+ e2bCalls.push(["pause", options]);+ if (pauseError) throw pauseError;+ return false;+ },+ async createSnapshot(options: unknown) {+ e2bCalls.push(["createSnapshot", options]);+ return { snapshotId: "snap-native", names: [] };+ },+ async getMetrics() {+ return [+ { timestamp: new Date(0), cpuUsedPct: 1, cpuCount: 2, memUsed: 1, memTotal: 2, diskUsed: 3, diskTotal: 4 },+ {+ timestamp: new Date(1),+ cpuUsedPct: 12.5,+ cpuCount: 2,+ memUsed: 2 ** 30,+ memTotal: 2 ** 31,+ diskUsed: 2 ** 30,+ diskTotal: 20 * 2 ** 30,+ },+ ];+ },+});mock.module("e2b", {namedExports: {+ SandboxError: FakeSandboxError,+ NotFoundError: FakeNotFoundError,+ SandboxNotFoundError: FakeSandboxNotFoundError,+ FileNotFoundError: FakeFileNotFoundError,+ TimeoutError: FakeTimeoutError,+ CommandExitError: FakeCommandExitError,Sandbox: class {+ static list() {+ return { hasNext: false, nextItems: async () => [] };+ }static async create(template: string, options: unknown) {e2bCalls.push(["create", template, options]);- return {- sandboxId: "e2b-native",- async pause(options: unknown) {- e2bCalls.push(["pause", options]);- if (pauseError) throw pauseError;- return false;- },- };+ return fakeE2bSandbox("e2b-native");+ }+ static async connect(sandboxId: string, options: unknown) {+ e2bCalls.push(["connect", sandboxId, options]);+ return fakeE2bSandbox(sandboxId);}static async getInfo() {e2bCalls.push(["info"]);- return { state: "paused", endAt: new Date(1000), lifecycle: { onTimeout: "pause" } };+ return {+ state: "paused",+ endAt: new Date(1000),+ lifecycle: { onTimeout: "pause" },+ cpuCount: 2,+ memoryMB: 512,+ };+ }+ static async deleteSnapshot(snapshotId: string) {+ e2bCalls.push(["deleteSnapshot", snapshotId]);+ return true;}},},@@ -213,10 +308,17 @@ test("Modal rejects invalid checkpoint retention", () => {});test("E2B sends explicit pause lifecycle, reads state without connect, and surfaces failed pause", async () => {+ e2bCalls.length = 0;const client = createSdkE2bClient({ apiKey: "test" });const session = await client.create({ metadata: { name: "test" }, autoPause: true });- assert.deepEqual((e2bCalls[0]![2] as { lifecycle: unknown }).lifecycle, { onTimeout: "pause", autoResume: false });- assert.equal(await client.info!(session.sandboxId).then((info) => info.state), "paused");+ const createOpts = e2bCalls[0]![2] as { lifecycle: unknown; timeoutMs: number; network?: unknown };+ assert.deepEqual(createOpts.lifecycle, { onTimeout: "pause", autoResume: false });+ assert.equal(createOpts.timeoutMs, 3600_000 + E2B_EXEC_MARGIN_MS, "default TTL covers the longest command");+ assert.equal(createOpts.network, undefined, "no egress proxy means no network rules");+ const info = await client.info!(session.sandboxId);+ assert.equal(info.state, "paused");+ assert.equal(info.cpuCount, 2);+ assert.equal(info.memoryMb, 512);await session.pause();assert.deepEqual(e2bCalls.at(-1), ["pause", { keepMemory: true }]);pauseError = new Error("snapshot backlog");@@ -226,6 +328,120 @@ test("E2B sends explicit pause lifecycle, reads state without connect, and surfaassert.deepEqual((e2bCalls.at(-1)![2] as { lifecycle: unknown }).lifecycle, { onTimeout: "kill", autoResume: false });});+test("E2B extends the sandbox timeout only when a command outlives the remaining lifetime, capped by the plan", async () => {+ e2bCalls.length = 0;+ const client = createSdkE2bClient({ apiKey: "test", sandboxTtlMs: 120_000 });+ const session = await client.create({ metadata: {}, autoPause: true });+ await session.runCommand("echo short", { timeoutMs: 1_000 });+ assert.equal(e2bCalls.filter((c) => c[0] === "setTimeout").length, 0, "a short command fits the TTL");+ await session.runCommand("sleep long", { timeoutMs: 600_000 });+ assert.deepEqual(+ e2bCalls.filter((c) => c[0] === "setTimeout"),+ [["setTimeout", 600_000 + E2B_EXEC_MARGIN_MS]],+ );+ await session.runCommand("echo again", { timeoutMs: 1_000 });+ assert.equal(e2bCalls.filter((c) => c[0] === "setTimeout").length, 1, "the extension covers later short commands");+ await session.keepAlive(30_000);+ const kept = e2bCalls.filter((c) => c[0] === "setTimeout").at(-1)![1] as number;+ assert.ok(kept > 600_000, `keepAlive never shortens a lifetime already covering a command (${kept})`);++ e2bCalls.length = 0;+ const hobby = createSdkE2bClient({ apiKey: "test", maxLifetimeMs: 3600_000 });+ const capped = await hobby.create({ metadata: {}, autoPause: true });+ assert.equal((e2bCalls[0]![2] as { timeoutMs: number }).timeoutMs, 3600_000, "TTL never exceeds the plan cap");+ await capped.runCommand("sleep", { timeoutMs: 3600_000 });+ await capped.keepAlive(7200_000);+ const caps = e2bCalls.filter((c) => c[0] === "setTimeout").map((c) => c[1]);+ assert.ok(caps.length >= 1, "the keep-warm request beyond the cap still extends to the cap");+ assert.ok(+ caps.every((ms) => ms === 3600_000),+ `every extension is capped at the plan maximum (${caps.join(",")})`,+ );+ assert.throws(() => createSdkE2bClient({ apiKey: "test", maxLifetimeMs: 0 }), /above 60000/);+});++test("E2B refuses to re-run a command lost mid-flight and classifies gone sandboxes by SDK error class", async () => {+ const client = createSdkE2bClient({ apiKey: "test" });+ const session = await client.create({ metadata: {}, autoPause: true });+ startError = new FakeSandboxNotFoundError("Sandbox is probably not running anymore");+ const beforeDispatch = e2bCalls.filter(([kind]) => kind === "run").length;+ await assert.rejects(session.runCommand("echo"), E2bCommandLostError);+ assert.equal(e2bCalls.filter(([kind]) => kind === "run").length, beforeDispatch + 1);+ startError = new Error("upstream returned 410: resource not found");+ await assert.rejects(+ session.runCommand("echo"),+ (e: Error) => !(e instanceof E2bSandboxGoneError) && /410/.test(e.message),+ );+ startError = undefined;+ waitError = new FakeTimeoutError(+ "connection terminated: The sandbox was killed or reached its end of life while the request was in flight.",+ );+ await assert.rejects(session.runCommand("echo"), E2bCommandLostError);+ waitError = new FakeSandboxNotFoundError("Sandbox is probably not running anymore");+ await assert.rejects(session.runCommand("echo"), E2bCommandLostError);+ waitError = new FakeTimeoutError("deadline exceeded: This error is likely due to exceeding 'timeoutMs'");+ await assert.rejects(+ session.runCommand("echo"),+ (e: Error) => e instanceof FakeTimeoutError && !(e instanceof E2bCommandLostError),+ "a command deadline is neither a lost sandbox nor a gone one",+ );+ waitError = new FakeCommandExitError(3, "out", "err");+ assert.deepEqual(await session.runCommand("exit 3"), { stdout: "out", stderr: "err", exitCode: 3 });+ waitError = undefined;+ assert.deepEqual(await session.runCommand("echo"), { stdout: "ran", stderr: "", exitCode: 0 });+ assert.equal(await session.readFileBytes("/home/user/missing"), null, "a missing file is null, not a gone sandbox");+});++test("E2B turns the egress proxy into host-level network rules on create and on reconnect", async () => {+ e2bCalls.length = 0;+ const client = createSdkE2bClient({ apiKey: "test", egressProxyUrl: "https://egress.example.com" });+ const rules = { allowOut: ["egress.example.com"], denyOut: ["0.0.0.0/0"] };+ await client.create({ metadata: {}, autoPause: true });+ assert.deepEqual((e2bCalls.at(-1)![2] as { network: unknown }).network, rules);+ await client.connect("e2b-native");+ assert.deepEqual(e2bCalls.at(-1), ["updateNetwork", rules]);+ assert.deepEqual(e2bEgressNetwork("http://10.0.0.5:3128"), { allowOut: ["10.0.0.5"], denyOut: ["0.0.0.0/0"] });+ assert.deepEqual(e2bEgressNetwork("http://egress.example.com"), rules);+ assert.throws(() => e2bEgressNetwork("http://proxy.internal:3128"), /ports 80 and 443/);+ assert.throws(() => e2bEgressNetwork("https://egress.example.com:8443"), /ports 80 and 443/);+});++test("E2B persistent snapshots and metrics map onto the client contract", async () => {+ e2bCalls.length = 0;+ const client = createSdkE2bClient({ apiKey: "test" });+ const session = await client.create({ metadata: {}, autoPause: true, fromSnapshot: "snap-old" });+ assert.equal(e2bCalls[0]![1], "snap-old", "a restore creates from the snapshot instead of the template");+ assert.deepEqual(await session.createSnapshot(), { snapshotId: "snap-native" });+ assert.deepEqual(await session.metrics(), {+ cpuUsedPct: 12.5,+ cpuCount: 2,+ memUsedBytes: 2 ** 30,+ memTotalBytes: 2 ** 31,+ diskUsedBytes: 2 ** 30,+ diskTotalBytes: 20 * 2 ** 30,+ });+ await client.deleteSnapshot("snap-old");+ assert.deepEqual(e2bCalls.at(-1), ["deleteSnapshot", "snap-old"]);+});++test("E2B does not dispatch twice when execution occurs before its start acknowledgment is lost", async (t) => {+ const root = mkdtempSync(join(tmpdir(), "e2b-lost-start-"));+ t.after(() => {+ startError = undefined;+ onDispatch = undefined;+ rmSync(root, { recursive: true, force: true });+ });+ const sandbox = createE2bSandbox(createLocalWorkspaceStore(root), { client: createSdkE2bClient({ apiKey: "test" }) });+ const handle = await sandbox.provision([{ scopeId: "lost-start", mountPath: "", mode: "rw" }]);+ let effects = 0;+ onDispatch = (command) => {+ if (command.includes("perform-side-effect")) effects++;+ };+ startError = new FakeSandboxNotFoundError("start stream unavailable after dispatch");+ await assert.rejects(sandbox.run(handle, "perform-side-effect"), /may have partially executed and was not retried/);+ assert.equal(effects, 1);+});+test("Modal executes a one-megabyte payload through the sandbox and SDK without nested oversized arguments", async (t) => {const root = mkdtempSync(join(tmpdir(), "modal-large-command-"));const home = join(root, "home");diff --git a/test/support/fake-e2b.ts b/test/support/fake-e2b.tsindex 795581df3..7667b0bab 100644--- a/test/support/fake-e2b.ts+++ b/test/support/fake-e2b.ts@@ -1,11 +1,13 @@import { spawnSync } from "node:child_process";-import { existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { cpSync, existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";import { dirname, join } from "node:path";import {+ E2bCommandLostError,E2bSandboxGoneError,type E2bClient,type E2bCommandResult,+ type E2bMetrics,type E2bSession,} from "../../src/sandbox/e2b-client.ts";@@ -17,6 +19,9 @@ interface FakeRecord {metadata: Record<string, string>;home: string;createdAt: number;+ fromSnapshot?: string;+ timeoutMs: number[];+ loseNextCommand: boolean;}export interface FakeE2b {@@ -27,6 +32,11 @@ export interface FakeE2b {pause(name: string): void;expirePaused(): void;+ loseNextCommand(name: string): void;+ timeouts(name: string): number[];+ snapshots(): string[];+ restoredFrom(name: string): string | undefined;+ metrics: E2bMetrics | null;execScripts(): string[];cleanup(): void;}@@ -34,8 +44,10 @@ export interface FakeE2b {export function installFakeE2b(): FakeE2b {const root = mkdtempSync(join(tmpdir(), "fake-e2b-"));const records = new Map<string, FakeRecord>();+ const snapshots = new Map<string, string>();const execScripts: string[] = [];let nextId = 1;+ let nextSnapshot = 1;let clock = 0;const byName = (name: string): FakeRecord | undefined => {@@ -64,10 +76,52 @@ export function installFakeE2b(): FakeE2b {}};+ const fake: FakeE2b = {+ client: null as unknown as E2bClient,+ metrics: null,+ current: (name) => {+ const r = byName(name);+ return r && !r.expired ? { sandboxId: r.sandboxId, state: r.state, metadata: r.metadata } : null;+ },+ createdCount: (name) => [...records.values()].filter((r) => r.metadata.name === name).length,+ homeDir: (name) => {+ const r = byName(name);+ if (!r) throw new Error(`fake-e2b: no sandbox named ${name}`);+ return r.home;+ },+ pause: (name) => {+ const r = byName(name);+ if (r) r.state = "paused";+ },+ expirePaused: () => {+ for (const r of records.values()) {+ if (r.state === "paused" && !r.expired) {+ r.expired = true;+ rmSync(r.home, { recursive: true, force: true });+ }+ }+ },+ loseNextCommand: (name) => {+ const r = byName(name);+ if (r) r.loseNextCommand = true;+ },+ timeouts: (name) => [...(byName(name)?.timeoutMs ?? [])],+ snapshots: () => [...snapshots.keys()],+ restoredFrom: (name) => byName(name)?.fromSnapshot,+ execScripts: () => [...execScripts],+ cleanup: () => rmSync(root, { recursive: true, force: true }),+ };+const session = (r: FakeRecord): E2bSession => ({sandboxId: r.sandboxId,- async runCommand(command): Promise<E2bCommandResult> {+ async runCommand(command, opts): Promise<E2bCommandResult> {alive(r);+ if (opts?.timeoutMs) r.timeoutMs.push(opts.timeoutMs);+ if (r.loseNextCommand) {+ r.loseNextCommand = false;+ r.state = "paused";+ throw new E2bCommandLostError(r.sandboxId, "sandbox timeout");+ }execScripts.push(command);mkdirSync(join(r.home, "tmp"), { recursive: true });const spawned = spawnSync("sh", ["-c", remap(r, command)], {@@ -93,17 +147,34 @@ export function installFakeE2b(): FakeE2b {mkdirSync(dirname(hostPath), { recursive: true });writeFileSync(hostPath, Buffer.from(data));},+ async keepAlive(ms): Promise<void> {+ alive(r);+ r.timeoutMs.push(ms);+ },async pause(): Promise<void> {if (r.expired) throw new E2bSandboxGoneError(r.sandboxId, "sandbox was not found");r.state = "paused";},+ async createSnapshot(): Promise<{ snapshotId: string }> {+ alive(r);+ const snapshotId = `snap-${nextSnapshot++}`;+ const dir = join(root, "snapshots", snapshotId);+ mkdirSync(dirname(dir), { recursive: true });+ cpSync(r.home, dir, { recursive: true });+ snapshots.set(snapshotId, dir);+ return { snapshotId };+ },+ async metrics(): Promise<E2bMetrics | null> {+ alive(r);+ return fake.metrics;+ },async kill(): Promise<void> {r.expired = true;rmSync(r.home, { recursive: true, force: true });},});- const client: E2bClient = {+ fake.client = {async create(opts): Promise<E2bSession> {const id = `sbx-${nextId++}`;const r: FakeRecord = {@@ -114,8 +185,17 @@ export function installFakeE2b(): FakeE2b {metadata: opts.metadata,home: join(root, id),createdAt: ++clock,+ timeoutMs: [],+ loseNextCommand: false,+ ...(opts.fromSnapshot ? { fromSnapshot: opts.fromSnapshot } : {}),};- mkdirSync(r.home, { recursive: true });+ if (opts.fromSnapshot) {+ const dir = snapshots.get(opts.fromSnapshot);+ if (!dir) throw new Error(`fake-e2b: snapshot ${opts.fromSnapshot} not found`);+ cpSync(dir, r.home, { recursive: true });+ } else {+ mkdirSync(r.home, { recursive: true });+ }records.set(id, r);return session(r);},@@ -136,33 +216,13 @@ export function installFakeE2b(): FakeE2b {r.expired = true;rmSync(r.home, { recursive: true, force: true });},- };-- return {- client,- current: (name) => {- const r = byName(name);- return r && !r.expired ? { sandboxId: r.sandboxId, state: r.state, metadata: r.metadata } : null;- },- createdCount: (name) => [...records.values()].filter((r) => r.metadata.name === name).length,- homeDir: (name) => {- const r = byName(name);- if (!r) throw new Error(`fake-e2b: no sandbox named ${name}`);- return r.home;+ async deleteSnapshot(snapshotId): Promise<void> {+ const dir = snapshots.get(snapshotId);+ if (!dir) return;+ rmSync(dir, { recursive: true, force: true });+ snapshots.delete(snapshotId);},- pause: (name) => {- const r = byName(name);- if (r) r.state = "paused";- },- expirePaused: () => {- for (const r of records.values()) {- if (r.state === "paused" && !r.expired) {- r.expired = true;- rmSync(r.home, { recursive: true, force: true });- }- }- },- execScripts: () => [...execScripts],- cleanup: () => rmSync(root, { recursive: true, force: true }),};++ return fake;}