Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

Preserve sandbox working state with native provider recovery (#1038)

Regan Bell43986f2e1c27
diff --git a/docs/sandbox-preservation.md b/docs/sandbox-preservation.md
new file mode 100644
index 000000000..7b9514e9a
--- /dev/null
+++ b/docs/sandbox-preservation.md
@@ -0,0 +1,15 @@
+# Sandbox recovery
+
+The agent runs in core. Sandbox home state is working state with a provider-specific recovery window. Publish code to git and deliverables to Files when they must remain durable and accessible independently of compute.
+
+`MODAL_NATIVE_SNAPSHOTS_ENABLED` defaults to false. First deploy this reader-compatible release with native capture disabled; keep portable checkpointing until every active core and the retained rollback release can read native checkpoints. Enable the flag only in a later deployment after that condition holds. Once a scope has a native checkpoint, disabling the flag continues its native capture and recovery; it does not revert that scope to stale portable state. After activation, rolling back to releases without the native reader is unsupported and can lose working state.
+
+When enabled, Modal uses native directory checkpoints of `/root`, mounted into a fresh sandbox when the current sandbox is replaced. It does not stream the home through core during routine checkpointing or rollover. The base image supplies system dependencies outside the home; a home checkpoint does not preserve processes, RAM, or system-wide package installs. Runtime credential files under `/tmp` are outside the directory checkpoint; ordinary home files and symlinks are included.
+
+`MODAL_NATIVE_SNAPSHOT_INTERVAL_SEC` defaults to 300 and controls checkpoint throttling independently of the legacy `MODAL_SNAPSHOT_INTERVAL_SEC`. Used-turn teardown, replacement, and idle reaping capture checkpoints. The existing deep-idle maintenance sweep also checkpoints active machines when due, including machines running background work. This is best-effort recovery, not a guarantee that a process running into the provider deadline has saved its latest writes. Background maintenance must be enabled; the sweep cadence depends on the configured idle intervals. Captures while processes write are not application-consistent database backups.
+
+`MODAL_SNAPSHOT_RETENTION_SEC` defaults to 2592000 (30 days) and must be positive. Checkpoint IDs, capture timestamps, errors and expiration times are stored with sandbox records. `DATABASE_URL` must point to durable Postgres to retain these references across core restarts; without it the sandbox record map is memory-only. Status reports `recovery.checkpointExpiresAtMs`; using a checkpoint does not renew its expiry. Expired or failed native restores stop with an explicit recovery error instead of replacing the home with an empty directory or silently using an older portable backup. Idle machines whose checkpoints expire are not automatically renewed. A core crash during hydration leaves a durable pending marker; another core refuses to expose that potentially incomplete home until an operator confirms the original core is no longer hydrating and invokes `execute` with `computer: "restart"`. Restart terminates only the incomplete replacement, retains checkpoint references, and clears the pending marker; the next provision retries that retained checkpoint. It does not archive or promote the incomplete home. Handled restore failures can retry after successful cleanup of the failed replacement.
+
+E2B creates persistent machines with an explicit pause-on-timeout lifecycle and pauses them at teardown unless `keepWarm` retains running background work. The provider timeout still applies; `keepWarm` does not extend that timeout. Routine native pause does not create a second tar backup. Provider pause retains disk and normally memory; provider fallback may retain disk only, so reconnectable processes must not depend on RAM surviving. Status can inspect a paused machine without waking it. Pause failures are recorded and returned, with the running machine retained for retry. A lost native machine requires explicit recovery rather than an automatic blank replacement. Existing E2B machines whose lifecycle does not confirm pause-on-timeout retain legacy portable checkpointing.
+
+Explicit home export/import and `persistHomeSnapshot` continue to use the portable tar format for cross-provider moves and chosen recovery. Configure `MODAL_SNAPSHOT_S3_BUCKET` or `E2B_SNAPSHOT_S3_BUCKET` to retain those portable copies durably. They are not silently selected instead of a failed newer provider checkpoint.
diff --git a/src/config.ts b/src/config.ts
index cb5bdf28f..ad5a92366 100644
--- a/src/config.ts
+++ b/src/config.ts
@@ -366,6 +366,9 @@ function e2bSandboxEnv(env: NodeJS.ProcessEnv): E2bSandboxEnv {
}
interface ModalSandboxEnv {
+ nativeSnapshotsEnabled?: boolean;
+ nativeSnapshotIntervalSec?: number;
+ snapshotRetentionSec?: number;
tokenId?: string;
tokenSecret?: string;
appName?: string;
@@ -387,6 +390,14 @@ interface ModalSandboxEnv {
function modalSandboxEnv(env: NodeJS.ProcessEnv): ModalSandboxEnv {
const num = (name: string): number | undefined => numEnvStrict(name, env[name]);
return {
+ nativeSnapshotsEnabled:
+ boolEnvStrict("MODAL_NATIVE_SNAPSHOTS_ENABLED", env.MODAL_NATIVE_SNAPSHOTS_ENABLED) ?? false,
+ ...(num("MODAL_NATIVE_SNAPSHOT_INTERVAL_SEC") !== undefined
+ ? { nativeSnapshotIntervalSec: num("MODAL_NATIVE_SNAPSHOT_INTERVAL_SEC") }
+ : {}),
+ ...(num("MODAL_SNAPSHOT_RETENTION_SEC") !== undefined
+ ? { snapshotRetentionSec: num("MODAL_SNAPSHOT_RETENTION_SEC") }
+ : {}),
...(env.MODAL_TOKEN_ID ? { tokenId: env.MODAL_TOKEN_ID } : {}),
...(env.MODAL_TOKEN_SECRET ? { tokenSecret: env.MODAL_TOKEN_SECRET } : {}),
...(env.MODAL_APP_NAME ? { appName: env.MODAL_APP_NAME } : {}),
@@ -993,12 +1004,12 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {
}
if (env.E2B_API_KEY && !env.E2B_SNAPSHOT_S3_BUCKET) {
console.warn(
- "[config] e2b sandbox backend enabled without E2B_SNAPSHOT_S3_BUCKET — home snapshots are held in memory only, so a core restart plus an expired pause LOSES scope files; set E2B_SNAPSHOT_S3_BUCKET for durable snapshots.",
+ "[config] e2b sandbox backend enabled without E2B_SNAPSHOT_S3_BUCKET — provider pause preserves the machine, but portable recovery snapshots are memory-only; set E2B_SNAPSHOT_S3_BUCKET for durable portable recovery.",
);
}
if (env.MODAL_TOKEN_ID && env.MODAL_TOKEN_SECRET && !env.MODAL_SNAPSHOT_S3_BUCKET) {
console.warn(
- "[config] modal sandbox backend enabled without MODAL_SNAPSHOT_S3_BUCKET — home snapshots are held in memory only, and modal has no pause: a core restart, an idle reap, or the 24h lifetime wall LOSES scope files; set MODAL_SNAPSHOT_S3_BUCKET for durable snapshots.",
+ "[config] modal sandbox backend enabled without MODAL_SNAPSHOT_S3_BUCKET — native home checkpoints have limited retention; portable recovery snapshots are memory-only. Set MODAL_SNAPSHOT_S3_BUCKET for durable portable recovery and configure DATABASE_URL for durable checkpoint references.",
);
}
if (env.NODE_ENV === "production" && harnessEnvStrict(env.HARNESS) === "mock") {
diff --git a/src/harness/agent-tools.ts b/src/harness/agent-tools.ts
index a883683ec..faf0c9672 100644
--- a/src/harness/agent-tools.ts
+++ b/src/harness/agent-tools.ts
@@ -631,6 +631,24 @@ export function createAgentTools(ref: ToolContextRef, opts?: AgentToolsOptions):
const verdict = computerVerdict(s);
const machineLine = s.listed && s.listed !== s.machine ? `${s.machine} (listed: ${s.listed})` : s.machine;
const pressureLine = s.pressure ? `; io pressure: ${s.pressure.ioFull60}% (load ${s.pressure.load1})` : "";
+ let shellLine = s.guestResponsive ? "answering" : `NOT answering${s.probeError ? ` (${s.probeError})` : ""}`;
+ if (s.lifecycleState === "paused") shellLine = "paused (not probed)";
+ const recoveryLines: string[] = [];
+ if (s.lifecycleState) recoveryLines.push(`lifecycle: ${s.lifecycleState}`);
+ if (s.expiresAtMs !== undefined)
+ recoveryLines.push(`machine expires: ${new Date(s.expiresAtMs).toISOString()}`);
+ if (s.recovery) {
+ recoveryLines.push(`recovery strategy: ${s.recovery.strategy}`);
+ if (s.recovery.state) recoveryLines.push(`recovery state: ${s.recovery.state}`);
+ if (s.recovery.checkpointId) recoveryLines.push(`checkpoint: ${s.recovery.checkpointId}`);
+ if (s.recovery.checkpointAtMs !== undefined)
+ recoveryLines.push(`checkpoint captured: ${new Date(s.recovery.checkpointAtMs).toISOString()}`);
+ if (s.recovery.checkpointExpiresAtMs === null)
+ recoveryLines.push("checkpoint expires: no provider expiry reported");
+ else if (s.recovery.checkpointExpiresAtMs !== undefined)
+ recoveryLines.push(`checkpoint expires: ${new Date(s.recovery.checkpointExpiresAtMs).toISOString()}`);
+ if (s.recovery.error) recoveryLines.push(`recovery error: ${s.recovery.error}`);
+ }
const verdictLine =
verdict === "wedged"
? " — WEDGED: a machine exists but its shell is not answering; the platform's health reporting goes stale in exactly this state, so trust the shell probe over any healthy/running claim and restart the computer"
@@ -639,7 +657,7 @@ export function createAgentTools(ref: ToolContextRef, opts?: AgentToolsOptions):
callId,
{ tool: "execute", computer: "status", verdict, ...s },
text(
- `machine: ${machineLine}; shell: ${s.guestResponsive ? "answering" : `NOT answering${s.probeError ? ` (${s.probeError})` : ""}`}${pressureLine}${verdictLine}`,
+ [`machine: ${machineLine}; shell: ${shellLine}${pressureLine}${verdictLine}`, ...recoveryLines].join("\n"),
),
);
} catch (e) {
diff --git a/src/sandbox/e2b-client.ts b/src/sandbox/e2b-client.ts
index 000aefc51..51a2fb04d 100644
--- a/src/sandbox/e2b-client.ts
+++ b/src/sandbox/e2b-client.ts
@@ -38,6 +38,9 @@ interface E2bCreateOpts {
}
export interface E2bClient {
+ readonly nativePause?: boolean;
+ info?(sandboxId: string): Promise<{ state: string; expiresAtMs: number; onTimeout?: string }>;
+
create(opts: E2bCreateOpts): Promise<E2bSession>;
connect(sandboxId: string): Promise<E2bSession>;
@@ -160,11 +163,7 @@ export function createSdkE2bClient(opts: SdkE2bClientOptions): E2bClient {
}
},
async pause(): Promise<void> {
- try {
- await sbx.pause();
- } catch {
- await sbx.pause();
- }
+ await sbx.pause({ keepMemory: true });
},
async kill(): Promise<void> {
await sbx.kill().catch((err) => {
@@ -174,13 +173,24 @@ export function createSdkE2bClient(opts: SdkE2bClientOptions): E2bClient {
});
return {
+ nativePause: true,
+ async info(sandboxId) {
+ const { Sandbox } = await loadSdk();
+ try {
+ const info = await Sandbox.getInfo(sandboxId, common);
+ return { state: info.state, expiresAtMs: info.endAt.getTime(), onTimeout: info.lifecycle?.onTimeout };
+ } catch (err) {
+ if (isGoneError(err)) throw new E2bSandboxGoneError(sandboxId, String((err as Error).message));
+ throw err;
+ }
+ },
async create(createOpts): Promise<E2bSession> {
const { Sandbox } = await loadSdk();
const sbx = await Sandbox.create(templateId, {
...common,
timeoutMs: createOpts.timeoutMs ?? sandboxTtlMs,
metadata: createOpts.metadata,
- ...(createOpts.autoPause !== undefined ? { autoPause: createOpts.autoPause } : {}),
+ lifecycle: { onTimeout: createOpts.autoPause ? "pause" : "kill", autoResume: false },
});
return wrap(sbx);
},
diff --git a/src/sandbox/e2b-sandbox.ts b/src/sandbox/e2b-sandbox.ts
index 537d29b94..2b19c6a5e 100644
--- a/src/sandbox/e2b-sandbox.ts
+++ b/src/sandbox/e2b-sandbox.ts
@@ -57,6 +57,9 @@ const SNAPSHOT_PRUNE = HOME_SNAPSHOT_PRUNE;
export interface StoredE2bSandbox {
sandboxId: string;
+ nativePause?: boolean;
+ preservationState?: "running" | "paused" | "pause_failed";
+ preservationError?: string;
createdAtMs: number;
lastSnapshotMs?: number;
homeDirty?: boolean;
@@ -135,6 +138,11 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti
if (stored) {
try {
const session = await client.connect(stored.sandboxId);
+ const info = await client.info?.(session.sandboxId);
+ await store.merge(scope, {
+ preservationState: "running",
+ ...(info ? { nativePause: info.onTimeout === "pause" } : {}),
+ });
return adopt(session);
} catch (err) {
if (!(err instanceof E2bSandboxGoneError)) throw err;
@@ -145,13 +153,21 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti
for (const summary of listed) {
try {
const session = await client.connect(summary.sandboxId);
- await store.put(scope, { sandboxId: session.sandboxId, createdAtMs: Date.now() });
+ const info = await client.info?.(session.sandboxId);
+ await store.put(scope, {
+ sandboxId: session.sandboxId,
+ createdAtMs: Date.now(),
+ nativePause: info?.onTimeout === "pause",
+ });
return adopt(session);
} catch (err) {
if (!(err instanceof E2bSandboxGoneError)) throw err;
}
}
+ if (stored?.nativePause) {
+ throw new Error("e2b sandbox is gone; explicitly import a recovery snapshot before replacing its home");
+ }
try {
onStatus?.("Creating the sandbox…");
} catch (error) {
@@ -159,7 +175,12 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti
}
const session = await client.create({ metadata: { name }, autoPause: true });
sessionByName.set(name, session);
- await store.put(scope, { sandboxId: session.sandboxId, createdAtMs: Date.now() });
+ await store.put(scope, {
+ sandboxId: session.sandboxId,
+ createdAtMs: Date.now(),
+ nativePause: false,
+ preservationState: "running",
+ });
let hydrated: boolean;
try {
hydrated = await hydrateHome(scope, session);
@@ -171,6 +192,7 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti
cause: e,
});
}
+ await store.merge(scope, { nativePause: client.nativePause });
return { session, coldStart: !hydrated };
});
}
@@ -225,11 +247,11 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti
const profile: AgentComputerProfile = {
backend: "e2b",
- writablePersistence: "snapshot_to_workspace",
+ writablePersistence: client.nativePause ? "provider_managed" : "snapshot_to_workspace",
processSessions: true,
egressEnforcement: "none",
spec: {
- os: "Ubuntu — E2B Firecracker sandbox (pauses when idle; home is snapshotted and restored)",
+ os: "Ubuntu — E2B Firecracker sandbox (provider pause preserves state; publish durable work to git or Files)",
runtimes: ["Node", "Python 3"],
get tools() {
return visibleTools(["git", "curl", "jq", "tar", "python3", ...(opts.extraTools ?? [])]);
@@ -424,7 +446,27 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti
if (!stored) return { machine: "no sandbox provisioned yet", provisioned: false, guestResponsive: false };
const machine = `e2b sandbox ${stored.sandboxId}`;
+ let expiresAtMs: number | undefined;
+ const recovery = {
+ strategy: stored.nativePause ? ("provider_pause" as const) : ("workspace_snapshot" as const),
+ state: stored.preservationState,
+ ...(stored.preservationError ? { error: stored.preservationError } : {}),
+ ...(stored.lastSnapshotMs ? { checkpointAtMs: stored.lastSnapshotMs } : {}),
+ };
try {
+ if (client.info) {
+ const info = await client.info(stored.sandboxId);
+ expiresAtMs = info.state === "running" ? info.expiresAtMs : undefined;
+ if (info.state === "paused")
+ return {
+ machine,
+ listed: info.state,
+ lifecycleState: "paused",
+ provisioned: true,
+ guestResponsive: false,
+ recovery: { ...recovery, state: info.state, checkpointExpiresAtMs: null },
+ };
+ }
scopeByName.set(name, scopeId);
let session = sessionByName.get(name);
if (!session) {
@@ -432,9 +474,16 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti
sessionByName.set(name, session);
}
const r = await session.runCommand("echo responsive", { timeoutMs: 30_000 });
- return { machine, provisioned: true, guestResponsive: r.exitCode === 0 && /responsive/.test(r.stdout) };
+ return {
+ machine,
+ expiresAtMs,
+ recovery,
+ provisioned: true,
+ guestResponsive: r.exitCode === 0 && /responsive/.test(r.stdout),
+ };
} catch (e) {
return {
+ recovery,
machine: `${machine} (${errMessage(e).slice(0, 120)})`,
provisioned: !(e instanceof E2bSandboxGoneError),
guestResponsive: false,
@@ -480,15 +529,27 @@ export function createE2bSandbox(workspace: WorkspaceStore, opts: E2bSandboxOpti
const stored = await store.get(scope);
if (!tdOpts?.homeUnchanged) await store.merge(scope, { homeDirty: true });
- if (snapshotDue(stored, tdOpts, snapshotIntervalMs)) {
+ if (!stored?.nativePause && snapshotDue(stored, tdOpts, snapshotIntervalMs)) {
try {
await snapshotHome(scope, session);
} catch (e) {
reportError("sandbox_snapshot", "teardown_snapshot_failed", errMessage(e), scope);
}
}
- sessionByName.delete(handle.id);
- await session.pause().catch(swallowAs("e2b-sandbox: pause on teardown", undefined));
+ if (tdOpts?.keepWarm) return;
+ try {
+ await session.pause();
+ await store.merge(scope, {
+ preservationState: "paused",
+ preservationError: undefined,
+ ...(stored?.nativePause ? { homeDirty: false } : {}),
+ });
+ sessionByName.delete(handle.id);
+ } catch (error) {
+ await store.merge(scope, { preservationState: "pause_failed", preservationError: errMessage(error) });
+ reportError("sandbox_preservation", "pause_failed", errMessage(error), scope);
+ throw error;
+ }
}
return sandbox;
diff --git a/src/sandbox/home-snapshot.ts b/src/sandbox/home-snapshot.ts
index 268c8bb77..449e3bf83 100644
--- a/src/sandbox/home-snapshot.ts
+++ b/src/sandbox/home-snapshot.ts
@@ -286,8 +286,9 @@ export function createHomeSnapshotOps<S>(opts: HomeSnapshotOpsOptions<S>): HomeS
async function tarSize(session: S, left: () => number): Promise<number> {
const stat = await run(session, `wc -c < ${shq(homeTarPath)}`, 30_000, left);
- const size = Number.parseInt(stat.stdout.trim(), 10);
- if (stat.exitCode !== 0 || !Number.isFinite(size)) throw new Error(`${label} snapshot read-back empty`);
+ const size = Number(stat.stdout.trim());
+ if (stat.exitCode !== 0 || !Number.isSafeInteger(size) || size <= 0)
+ throw new Error(`${label} snapshot read-back empty`);
return size;
}
@@ -338,7 +339,7 @@ export function createHomeSnapshotOps<S>(opts: HomeSnapshotOpsOptions<S>): HomeS
const prune = prunePaths.length
? `\\( ${prunePaths.map((p) => `-path ${shq(p)}`).join(" -o ")} \\) -prune -o `
: "";
- const script = `cd ${shq(homeDir)} 2>/dev/null || exit 0; find . ${prune}-type f -print0 > ${shq(listPath)} 2>/dev/null; tar --null -T ${shq(listPath)} -cf ${shq(homeTarPath)}; rc=$?; rm -f ${shq(listPath)}; exit $rc`;
+ const script = `cd ${shq(homeDir)} 2>/dev/null || exit 0; find . ${prune}\\( ! -type d -o -exec test -r {} \\; \\) -print0 > ${shq(listPath)} 2>/dev/null; tar --no-recursion --null -T ${shq(listPath)} -cf ${shq(homeTarPath)}; rc=$?; rm -f ${shq(listPath)}; exit $rc`;
try {
const made = await run(session, script, 180_000, left);
if (made.exitCode !== 0) throw new Error(`${label} snapshot tar failed: ${made.stderr.slice(0, 200)}`);
@@ -363,7 +364,9 @@ export function createHomeSnapshotOps<S>(opts: HomeSnapshotOpsOptions<S>): HomeS
async hydrateHome(scope, session): Promise<boolean> {
const left = startClock();
const stored = await withTimeout(() => store.open(scope), left(), `${label} hydrate open`);
- if (!stored || stored.size === 0) return false;
+ if (!stored) return false;
+ if (!Number.isSafeInteger(stored.size) || stored.size <= 0)
+ throw new Error(`${label} hydrate: invalid snapshot size ${stored.size}`);
try {
const started = await run(session, `mkdir -p ${shq(homeDir)} && : > ${shq(homeTarPath)}`, 30_000, left);
if (started.exitCode !== 0)
@@ -371,10 +374,19 @@ export function createHomeSnapshotOps<S>(opts: HomeSnapshotOpsOptions<S>): HomeS
let total = 0;
let i = 0;
for await (const piece of coalesce(stored.parts, partBytes)) {
- await writePart(session, i++, piece, left);
total += piece.length;
+ if (total > stored.size)
+ throw new Error(`${label} hydrate: received ${total} bytes, expected ${stored.size}`);
+ await writePart(session, i++, piece, left);
}
- if (total === 0) return false;
+ if (total !== stored.size)
+ throw new Error(`${label} hydrate: received ${total} bytes, expected ${stored.size}`);
+ const written = await tarSize(session, left);
+ if (written !== stored.size)
+ throw new Error(`${label} hydrate: wrote ${written} bytes, expected ${stored.size}`);
+ const validated = await run(session, `tar -tf ${shq(homeTarPath)} > /dev/null`, 180_000, left);
+ if (validated.exitCode !== 0)
+ throw new Error(`${label} hydrate archive invalid: ${validated.stderr.slice(0, 200)}`);
const r = await run(
session,
`cd ${shq(homeDir)} && tar -xf ${shq(homeTarPath)}; rc=$?; rm -f ${shq(homeTarPath)}; exit $rc`,
diff --git a/src/sandbox/modal-client.ts b/src/sandbox/modal-client.ts
index a1fa7d882..5fae65dc9 100644
--- a/src/sandbox/modal-client.ts
+++ b/src/sandbox/modal-client.ts
@@ -13,6 +13,8 @@ export interface ModalSession {
runCommand(command: string, opts?: ModalRunOpts): Promise<ModalCommandResult>;
readFileBytes(absPath: string): Promise<Uint8Array | null>;
writeFileBytes(absPath: string, data: Uint8Array): Promise<void>;
+ snapshotHome?(): Promise<{ imageId: string; expiresAtMs: number }>;
+ restoreHome?(imageId: string): Promise<void>;
terminate(): Promise<void>;
}
export class ModalSandboxGoneError extends Error {
@@ -33,6 +35,8 @@ interface ModalCreateOpts {
}
export interface ModalClient {
+ readonly lifetimeMs?: number;
+ readonly nativeSnapshots?: boolean;
create(opts: ModalCreateOpts): Promise<ModalSession>;
fromId(sandboxId: string): Promise<ModalSession>;
fromName(name: string): Promise<ModalSession | null>;
@@ -51,6 +55,7 @@ export interface SdkModalClientOptions {
regions?: string[];
sandboxTimeoutMs?: number;
maxCommandMs?: number;
+ snapshotRetentionMs?: number;
}
const MAX_LIFETIME_MS = 24 * 3600_000;
@@ -97,6 +102,9 @@ function isDeadlineError(err: unknown): boolean {
export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
const sandboxTimeoutMs = Math.min(opts.sandboxTimeoutMs ?? MAX_LIFETIME_MS, MAX_LIFETIME_MS);
const maxCommandMs = opts.maxCommandMs ?? DEFAULT_MAX_COMMAND_MS;
+ const snapshotRetentionMs = opts.snapshotRetentionMs ?? 30 * 24 * 3600_000;
+ if (!Number.isFinite(snapshotRetentionMs) || snapshotRetentionMs <= 0)
+ throw new Error("Modal snapshot retention must be a positive finite duration");
type ModalSdk = typeof import("modal");
type SdkClient = InstanceType<ModalSdk["ModalClient"]>;
type SdkApp = Awaited<ReturnType<SdkClient["apps"]["fromName"]>>;
@@ -158,6 +166,15 @@ export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
throw err;
}
},
+ async snapshotHome() {
+ const expiresAtMs = Date.now() + snapshotRetentionMs;
+ const image = await sbx.snapshotDirectory("/root", { ttlMs: snapshotRetentionMs });
+ return { imageId: image.imageId, expiresAtMs };
+ },
+ async restoreHome(imageId): Promise<void> {
+ const { client } = await loadCtx();
+ await sbx.mountImage("/root", await client.images.fromId(imageId));
+ },
async terminate(): Promise<void> {
await sbx.terminate().catch((err) => {
if (!isSandboxGoneError(err)) throw err;
@@ -166,6 +183,8 @@ export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
});
return {
+ lifetimeMs: sandboxTimeoutMs,
+ nativeSnapshots: true,
async create(createOpts): Promise<ModalSession> {
const { client, app, image } = await loadCtx();
try {
diff --git a/src/sandbox/modal-sandbox.ts b/src/sandbox/modal-sandbox.ts
index 013832626..8309c5e23 100644
--- a/src/sandbox/modal-sandbox.ts
+++ b/src/sandbox/modal-sandbox.ts
@@ -59,6 +59,12 @@ const SNAPSHOT_PRUNE = ["./.qm-hydrated", ...HOME_SNAPSHOT_PRUNE];
export interface StoredModalSandbox {
sandboxId: string;
+ hydrationPending?: boolean;
+ nativeSnapshotId?: string;
+ nativeSnapshotExpiresAtMs?: number;
+ expiresAtMs?: number;
+ recoveryError?: string;
+ snapshotGeneration?: number;
createdAtMs: number;
lastSnapshotMs?: number;
lastActivityMs?: number;
@@ -71,6 +77,8 @@ export interface ModalSandboxOptions extends BlobStagingOptions {
namePrefix?: string;
defaultTimeoutSec?: number;
snapshotIntervalMs?: number;
+ nativeSnapshotsEnabled?: boolean;
+ nativeSnapshotIntervalMs?: number;
rotateAfterMs?: number;
reapIdleMs?: number;
egressProxyUrl?: string;
@@ -88,7 +96,10 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
const client = opts.client;
const prefix = opts.namePrefix ?? "qm";
const defaultTimeoutSec = opts.defaultTimeoutSec ?? 600;
- const snapshotIntervalMs = opts.snapshotIntervalMs ?? 0;
+ const usesNativeSnapshots = (stored?: StoredModalSandbox | null): boolean =>
+ !!stored?.nativeSnapshotId || !!(client.nativeSnapshots && opts.nativeSnapshotsEnabled);
+ const snapshotIntervalMs = (stored?: StoredModalSandbox | null): number =>
+ usesNativeSnapshots(stored) ? (opts.nativeSnapshotIntervalMs ?? 5 * 60_000) : (opts.snapshotIntervalMs ?? 0);
const rotateAfterMs = opts.rotateAfterMs ?? 20 * 3600_000;
const reapIdleMs = opts.reapIdleMs ?? 6 * 3600_000;
const fileChunkBytes = opts.fileChunkBytes ?? 64 * 1024 * 1024;
@@ -122,17 +133,63 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
partBytes: fileChunkBytes,
});
- async function snapshotHome(scope: string, session: ModalSession): Promise<void> {
+ async function assertHydrated(session: ModalSession): Promise<void> {
const marked = await session.runCommand(`test -f ${shq(HYDRATED_MARKER)}`, { timeoutMs: 30_000 });
if (marked.exitCode !== 0)
throw new Error(
"modal snapshot refused: this sandbox never finished hydrating, its home is not a trustworthy source",
);
- await homeSnapshots.snapshotHome(scope, session);
- await store.merge(scope, { lastSnapshotMs: Date.now(), homeDirty: false });
+ }
+
+ async function snapshotHome(scope: string, session: ModalSession): Promise<void> {
+ await assertHydrated(session);
+ if (usesNativeSnapshots(await store.get(scope))) {
+ if (!session.snapshotHome) throw new Error("native Modal home checkpoint is not supported by this client");
+ if (!store.update) throw new Error("native Modal checkpoints require an atomic durable store");
+ const requested = await store.update(scope, (current) => {
+ if (current.sandboxId !== session.sandboxId) throw new Error("Modal checkpoint source has been replaced");
+ return { ...current, snapshotGeneration: (current.snapshotGeneration ?? 0) + 1 };
+ });
+ if (!requested) throw new Error("Modal checkpoint source is no longer tracked");
+ const capturedAtMs = Date.now();
+ try {
+ const snapshot = await session.snapshotHome();
+ const committed = await store.update(scope, (current) =>
+ current.sandboxId === session.sandboxId && current.snapshotGeneration === requested.snapshotGeneration
+ ? {
+ ...current,
+ nativeSnapshotId: snapshot.imageId,
+ nativeSnapshotExpiresAtMs: snapshot.expiresAtMs,
+ recoveryError: undefined,
+ lastSnapshotMs: capturedAtMs,
+ homeDirty: false,
+ }
+ : current,
+ );
+ if (committed?.nativeSnapshotId !== snapshot.imageId)
+ throw new Error("Modal checkpoint was superseded; keeping the source running");
+ } catch (error) {
+ await store.update(scope, (current) =>
+ current.sandboxId === session.sandboxId && current.snapshotGeneration === requested.snapshotGeneration
+ ? { ...current, recoveryError: errMessage(error) }
+ : current,
+ );
+ throw error;
+ }
+ } else {
+ await homeSnapshots.snapshotHome(scope, session);
+ await store.merge(scope, { lastSnapshotMs: Date.now(), homeDirty: false });
+ }
}
async function createHydrated(scope: string, name: string): Promise<{ session: ModalSession; coldStart: boolean }> {
+ const previous = await store.get(scope);
+ if (previous?.nativeSnapshotExpiresAtMs && previous.nativeSnapshotExpiresAtMs <= Date.now()) {
+ const recoveryError =
+ "Modal home checkpoint has expired; explicitly import a recovery snapshot before creating a replacement";
+ await store.merge(scope, { recoveryError });
+ throw new Error(recoveryError);
+ }
let session: ModalSession;
try {
session = await client.create({ name });
@@ -140,6 +197,9 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
if (!(err instanceof ModalNameConflictError)) throw err;
const adopted = await client.fromName(name);
if (!adopted) throw err;
+ if ((await store.get(scope))?.hydrationPending)
+ throw new Error("Modal home hydration is incomplete; refusing to adopt the replacement", { cause: err });
+ await assertHydrated(adopted);
sessionByName.set(name, adopted);
const merged = await store.merge(scope, { sandboxId: adopted.sandboxId, lastActivityMs: Date.now() });
if (!merged) {
@@ -152,8 +212,10 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
}
return { session: adopted, coldStart: false };
}
- sessionByName.set(name, session);
await store.put(scope, {
+ ...previous,
+ hydrationPending: true,
+ ...(client.lifetimeMs ? { expiresAtMs: Date.now() + client.lifetimeMs } : {}),
sandboxId: session.sandboxId,
createdAtMs: Date.now(),
lastActivityMs: Date.now(),
@@ -161,11 +223,21 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
});
let hydrated: boolean;
try {
- hydrated = await homeSnapshots.hydrateHome(scope, session);
+ if (previous?.nativeSnapshotId) {
+ if (!session.restoreHome) throw new Error("native Modal home restore is not supported by this client");
+ await session.restoreHome(previous.nativeSnapshotId);
+ hydrated = true;
+ } else {
+ hydrated = await homeSnapshots.hydrateHome(scope, session);
+ }
} catch (e) {
+ await store.merge(scope, { recoveryError: errMessage(e) });
reportError("sandbox_hydrate", "hydrate_failed", errMessage(e), scope);
sessionByName.delete(name);
- await session.terminate().catch(() => undefined);
+ await session.terminate().then(
+ () => store.merge(scope, { hydrationPending: false }),
+ () => undefined,
+ );
throw new Error(`modal provision: home hydration failed (${errMessage(e)}); not risking the stored snapshot`, {
cause: e,
});
@@ -173,10 +245,15 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
const marked = await session.runCommand(`touch ${shq(HYDRATED_MARKER)}`, { timeoutMs: 30_000 });
if (marked.exitCode !== 0) {
sessionByName.delete(name);
- await session.terminate().catch(() => undefined);
+ await session.terminate().then(
+ () => store.merge(scope, { hydrationPending: false }),
+ () => undefined,
+ );
throw new Error(`modal provision: could not mark the sandbox hydrated: ${marked.stderr.slice(0, 200)}`);
}
- if (hydrated) await store.merge(scope, { lastSnapshotMs: Date.now() });
+ await store.merge(scope, { hydrationPending: false });
+ sessionByName.set(name, session);
+ if (hydrated && !previous?.nativeSnapshotId) await store.merge(scope, { lastSnapshotMs: Date.now() });
return { session, coldStart: !hydrated };
}
@@ -187,12 +264,17 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
): Promise<{ session: ModalSession; coldStart: boolean }> {
return provisionQueue(scope, async () => {
const adopt = async (session: ModalSession): Promise<{ session: ModalSession; coldStart: boolean }> => {
+ await assertHydrated(session);
sessionByName.set(name, session);
await store.merge(scope, { lastActivityMs: Date.now() });
return { session, coldStart: false };
};
const stored = await store.get(scope);
+ if (stored?.hydrationPending)
+ throw new Error(
+ "Modal home hydration was interrupted; use computer restart to discard the incomplete replacement and retry the retained checkpoint",
+ );
const stale =
!!stored &&
Date.now() - stored.createdAtMs > rotateAfterMs &&
@@ -222,6 +304,7 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
if (!(err instanceof ModalSandboxGoneError)) throw err;
}
if (session) {
+ await assertHydrated(session);
try {
await snapshotHome(scope, session);
} catch (e) {
@@ -246,12 +329,13 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
}
}
sessionByName.delete(name);
- await store.delete(scope).catch(() => undefined);
+ if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope).catch(() => undefined);
}
if (!stored) {
const found = await client.fromName(name);
if (found) {
+ await assertHydrated(found);
await store.put(scope, {
sandboxId: found.sandboxId,
createdAtMs: Date.now(),
@@ -332,11 +416,11 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
const profile: AgentComputerProfile = {
backend: "modal",
- writablePersistence: "snapshot_to_workspace",
+ writablePersistence: usesNativeSnapshots() ? "provider_managed" : "snapshot_to_workspace",
processSessions: true,
egressEnforcement: "none",
spec: {
- os: "Ubuntu — Modal sandbox (24h max lifetime; home is snapshotted and restored onto fresh sandboxes)",
+ os: "Ubuntu — Modal sandbox (24h max lifetime; home checkpoints have limited retention; publish durable work to git or Files)",
runtimes: ["Python 3"],
get tools() {
return visibleTools(["git", "curl", "jq", "tar", "python3", ...(opts.extraTools ?? [])]);
@@ -532,25 +616,75 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
const name = sandboxScopeName(prefix, scopeId);
scopeByName.set(name, scopeId);
const { session } = await ensureSession(scopeId, name);
- await provisionQueue(scopeId, () => snapshotHome(scopeId, session));
+ await provisionQueue(scopeId, async () => {
+ await assertHydrated(session);
+ await homeSnapshots.snapshotHome(scopeId, session);
+ });
+ },
+
+ async restartComputer(scopeId: string): Promise<void> {
+ await provisionQueue(scopeId, async () => {
+ const stored = await store.get(scopeId);
+ if (!stored) return;
+ const name = sandboxScopeName(prefix, scopeId);
+ let session: ModalSession | undefined;
+ try {
+ session = await client.fromId(stored.sandboxId);
+ } catch (error) {
+ if (!(error instanceof ModalSandboxGoneError)) throw error;
+ }
+ if (session) {
+ if (!stored.hydrationPending) await snapshotHome(scopeId, session);
+ await session.terminate();
+ }
+ sessionByName.delete(name);
+ await store.merge(scopeId, { hydrationPending: false, recoveryError: undefined });
+ });
},
async computerStatus(scopeId: string): Promise<ComputerStatus> {
const name = sandboxScopeName(prefix, scopeId);
const stored = await store.get(scopeId);
if (!stored) return { machine: "no sandbox provisioned yet", provisioned: false, guestResponsive: false };
+ const recovery = {
+ strategy: stored.nativeSnapshotId ? ("provider_snapshot" as const) : ("workspace_snapshot" as const),
+ ...(stored.nativeSnapshotId
+ ? { checkpointId: stored.nativeSnapshotId, checkpointExpiresAtMs: stored.nativeSnapshotExpiresAtMs }
+ : {}),
+ ...(stored.lastSnapshotMs ? { checkpointAtMs: stored.lastSnapshotMs } : {}),
+ ...(stored.recoveryError ? { error: stored.recoveryError } : {}),
+ };
const machine = `modal sandbox ${stored.sandboxId}`;
+ if (stored.hydrationPending)
+ return {
+ machine,
+ provisioned: true,
+ guestResponsive: false,
+ recovery: {
+ ...recovery,
+ state: "failed",
+ error:
+ "Home hydration is incomplete. After confirming no core is still hydrating, use computer restart to discard the incomplete replacement and retry the retained checkpoint.",
+ },
+ };
try {
scopeByName.set(name, scopeId);
let session = sessionByName.get(name);
if (!session) {
session = await client.fromId(stored.sandboxId);
- sessionByName.set(name, session);
}
const r = await session.runCommand("echo responsive", { timeoutMs: 30_000 });
- return { machine, provisioned: true, guestResponsive: r.exitCode === 0 && /responsive/.test(r.stdout) };
+ return {
+ machine,
+ expiresAtMs: stored.expiresAtMs,
+ recovery,
+ provisioned: true,
+ guestResponsive: r.exitCode === 0 && /responsive/.test(r.stdout),
+ };
} catch (e) {
return {
+ expiresAtMs: stored.expiresAtMs,
+ recovery,
machine: `${machine} (${errMessage(e).slice(0, 120)})`,
provisioned: !(e instanceof ModalSandboxGoneError),
guestResponsive: false,
@@ -591,7 +725,7 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
if (!session) return;
const stored = await store.get(scope);
if (!tdOpts?.homeUnchanged) await store.merge(scope, { homeDirty: true });
- if (snapshotDue(stored, tdOpts, snapshotIntervalMs)) {
+ if (snapshotDue(stored, tdOpts, snapshotIntervalMs(stored))) {
try {
await snapshotHome(scope, session);
await store.merge(scope, { lastActivityMs: Date.now() });
@@ -611,6 +745,22 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
let reaped = 0;
for (const [scope, rec] of await store.entries()) {
if (rec.orgId && rec.orgId !== configOrgId()) continue;
+ if (rec.hydrationPending) continue;
+ if (
+ usesNativeSnapshots(rec) &&
+ (!rec.expiresAtMs || rec.expiresAtMs > Date.now()) &&
+ (!rec.lastSnapshotMs || Date.now() - rec.lastSnapshotMs > snapshotIntervalMs(rec))
+ ) {
+ await provisionQueue(scope, async () => {
+ try {
+ const session = await client.fromId(rec.sandboxId);
+ await snapshotHome(scope, session);
+ } catch (error) {
+ if (!(error instanceof ModalSandboxGoneError))
+ reportError("sandbox_snapshot", "periodic_snapshot_failed", errMessage(error), scope);
+ }
+ });
+ }
if (!rec.lastActivityMs || rec.lastActivityMs > cutoff) continue;
const name = sandboxScopeName(prefix, scope);
scopeByName.set(name, scope);
@@ -623,7 +773,7 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
} catch (e) {
if (e instanceof ModalSandboxGoneError) {
sessionByName.delete(name);
- await store.delete(scope).catch(() => undefined);
+ if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope).catch(() => undefined);
} else {
reportError("sandbox_reap", "deep_idle_probe_failed", errMessage(e), scope);
}
@@ -637,12 +787,12 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
await snapshotHome(scope, session);
await session.terminate();
sessionByName.delete(name);
- await store.delete(scope);
+ if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope);
return 1;
} catch (e) {
if (e instanceof ModalSandboxGoneError) {
sessionByName.delete(name);
- await store.delete(scope).catch(() => undefined);
+ if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope).catch(() => undefined);
} else {
reportError("sandbox_reap", "deep_idle_reap_failed", errMessage(e), scope);
}
diff --git a/src/sandbox/sandbox.ts b/src/sandbox/sandbox.ts
index e01ac2cf5..394ae11c8 100644
--- a/src/sandbox/sandbox.ts
+++ b/src/sandbox/sandbox.ts
@@ -15,7 +15,7 @@ export function hasParentPathSegment(path: string): boolean {
return path.split("/").includes("..");
}
-type WritablePersistence = "snapshot_to_workspace" | "resident_disk";
+type WritablePersistence = "snapshot_to_workspace" | "resident_disk" | "provider_managed";
export type EgressEnforcement = "none" | "ip_port" | "domain";
export interface AgentComputerSpec {
@@ -137,6 +137,16 @@ export interface ProcessSession {
}
export interface ComputerStatus {
+ lifecycleState?: "running" | "paused";
+ expiresAtMs?: number;
+ recovery?: {
+ strategy: "provider_snapshot" | "provider_pause" | "workspace_snapshot";
+ checkpointId?: string;
+ checkpointAtMs?: number;
+ checkpointExpiresAtMs?: number | null;
+ state?: string;
+ error?: string;
+ };
machine: string;
listed?: string;
provisioned?: boolean;
@@ -148,7 +158,7 @@ export interface ComputerStatus {
export type ComputerVerdict = "ok" | "wedged" | "down";
export function computerVerdict(s: ComputerStatus): ComputerVerdict {
- if (s.guestResponsive) return "ok";
+ if (s.guestResponsive || s.lifecycleState === "paused") return "ok";
return s.provisioned ? "wedged" : "down";
}
diff --git a/src/wiring.ts b/src/wiring.ts
index 755396f7b..30cb9eeb5 100644
--- a/src/wiring.ts
+++ b/src/wiring.ts
@@ -795,10 +795,15 @@ export function buildApp(
...(modal.memoryMb !== undefined ? { memoryMb: modal.memoryMb } : {}),
...(modal.regions?.length ? { regions: modal.regions } : {}),
...(modal.sandboxTimeoutSec ? { sandboxTimeoutMs: modal.sandboxTimeoutSec * 1000 } : {}),
+ ...(modal.snapshotRetentionSec !== undefined ? { snapshotRetentionMs: modal.snapshotRetentionSec * 1000 } : {}),
}),
...(modal.namePrefix ? { namePrefix: modal.namePrefix } : {}),
...(modal.defaultTimeoutSec ? { defaultTimeoutSec: modal.defaultTimeoutSec } : {}),
...(modal.snapshotIntervalSec !== undefined ? { snapshotIntervalMs: modal.snapshotIntervalSec * 1000 } : {}),
+ nativeSnapshotsEnabled: modal.nativeSnapshotsEnabled ?? false,
+ ...(modal.nativeSnapshotIntervalSec !== undefined
+ ? { nativeSnapshotIntervalMs: modal.nativeSnapshotIntervalSec * 1000 }
+ : {}),
...(modal.rotateAfterSec ? { rotateAfterMs: modal.rotateAfterSec * 1000 } : {}),
...(modal.reapIdleSec ? { reapIdleMs: modal.reapIdleSec * 1000 } : {}),
...(modal.egressProxyUrl ? { egressProxyUrl: modal.egressProxyUrl } : {}),
diff --git a/test/agent-tools.test.ts b/test/agent-tools.test.ts
index deee4d997..f9e18ebf3 100644
--- a/test/agent-tools.test.ts
+++ b/test/agent-tools.test.ts
@@ -548,6 +548,49 @@ test("computer status surfaces list-view disagreement and guest pressure", async
assert.match(status.content[0]!.text!, /io pressure: 86\.14% \(load 30\.78\)/);
});
+test("computer status exposes paused lifecycle and recovery deadlines without claiming a failed shell", async () => {
+ const ref: ToolContextRef = {
+ current: {
+ ...fakeToolContext(),
+ computerStatus: async () => ({
+ machine: "e2b sandbox",
+ provisioned: true,
+ guestResponsive: false,
+ lifecycleState: "paused",
+ expiresAtMs: Date.parse("2026-09-10T00:00:00Z"),
+ recovery: {
+ strategy: "provider_pause",
+ checkpointId: "checkpoint-1",
+ checkpointAtMs: Date.parse("2026-09-09T00:00:00Z"),
+ checkpointExpiresAtMs: Date.parse("2026-10-09T00:00:00Z"),
+ state: "failed",
+ error: "resume failed; retry required",
+ },
+ }),
+ },
+ emit: () => {},
+ scopeLabel: "personal:U1",
+ };
+ const [execute] = createAgentTools(ref);
+ const out = (await call(execute, { command: "", computer: "status", purpose: "p" })) as {
+ content: Array<{ text?: string }>;
+ };
+ const output = out.content[0]!.text!;
+ assert.match(output, /shell: paused \(not probed\)/);
+ assert.doesNotMatch(output, /NOT answering|WEDGED/);
+ for (const expected of [
+ "lifecycle: paused",
+ "machine expires: 2026-09-10T00:00:00.000Z",
+ "recovery strategy: provider_pause",
+ "recovery state: failed",
+ "checkpoint: checkpoint-1",
+ "checkpoint captured: 2026-09-09T00:00:00.000Z",
+ "checkpoint expires: 2026-10-09T00:00:00.000Z",
+ "recovery error: resume failed; retry required",
+ ])
+ assert.ok(output.includes(expected), expected);
+});
+
test("computer status verdicts: answering guest is ok, dead guest without a machine is down", async () => {
const base = fakeToolContext();
const cases: Array<{ status: ComputerStatus; wedged: boolean }> = [
diff --git a/test/config.test.ts b/test/config.test.ts
index 471ad5196..ee700ed6e 100644
--- a/test/config.test.ts
+++ b/test/config.test.ts
@@ -657,3 +657,21 @@ test("retired brain environment does not configure a runtime integration and war
assert.match(retired[0]!, /BRAIN, BRAIN_MCP_URL, BRAIN_RO_CLIENT_ID are retired/);
assert.match(retired[0]!, /MEMORY_PROVIDER_CONFIG/);
});
+
+test("Modal native retention and interval configuration are independent of legacy portable checkpoint throttling", () => {
+ const config = loadConfig({
+ MODAL_NATIVE_SNAPSHOT_INTERVAL_SEC: "60",
+ MODAL_SNAPSHOT_RETENTION_SEC: "86400",
+ MODAL_SNAPSHOT_INTERVAL_SEC: "315360000",
+ });
+ assert.equal(config.modalSandbox.nativeSnapshotIntervalSec, 60);
+ assert.equal(config.modalSandbox.snapshotRetentionSec, 86400);
+ assert.equal(config.modalSandbox.snapshotIntervalSec, 315360000);
+});
+
+test("Modal native activation is default-off and uses strict boolean configuration", () => {
+ assert.equal(loadConfig({}).modalSandbox.nativeSnapshotsEnabled, false);
+ for (const value of ["true", "on", "1"])
+ assert.equal(loadConfig({ MODAL_NATIVE_SNAPSHOTS_ENABLED: value }).modalSandbox.nativeSnapshotsEnabled, true);
+ assert.throws(() => loadConfig({ MODAL_NATIVE_SNAPSHOTS_ENABLED: "enable" }), /not a recognized boolean/);
+});
diff --git a/test/e2b-sandbox.test.ts b/test/e2b-sandbox.test.ts
index 7e7c13366..863f567e3 100644
--- a/test/e2b-sandbox.test.ts
+++ b/test/e2b-sandbox.test.ts
@@ -183,11 +183,11 @@ test("teardown pauses the sandbox; destroy kills it", async () => {
assert.equal(fake.current(h.id), null);
});
-test("keepWarm teardown pauses; next provision resumes the same sandbox", async () => {
+test("keepWarm teardown leaves the sandbox running for background work", async () => {
const a = await sandbox.provision(layers);
await sandbox.writeFile(a, "keep.txt", "resident\n");
await sandbox.teardown(a, { keepWarm: true });
- assert.equal(fake.current(a.id)?.state, "paused");
+ assert.equal(fake.current(a.id)?.state, "running");
const b = await sandbox.provision(layers);
assert.equal(b.coldStart, false);
assert.equal(fake.createdCount(scopeName()), 1);
@@ -363,3 +363,126 @@ test("stageIn pulls a blob into the guest atomically (temp then mv)", async () =
assert.match(script, /mv -f /, "and only then moves it into place");
assert.match(script, /curl -fsS/, "-f so an HTTP error fails loudly instead of writing the error body");
});
+
+test("native pause skips tar checkpoints and status does not wake a paused sandbox", async () => {
+ const store = createMemoryMap<StoredE2bSandbox>();
+ const portable = instrumentedSnapshotStore();
+ const client = {
+ ...fake.client,
+ nativePause: true,
+ async info() {
+ const current = fake.current(scopeName())!;
+ return { state: current.state, expiresAtMs: Date.now() + 60_000, onTimeout: "pause" };
+ },
+ };
+ const first = make({ client, store, snapshots: portable.store });
+ const handle = await first.provision(layers);
+ await first.writeFile(handle, "work.txt", "keep");
+ await first.teardown(handle);
+ assert.equal(portable.puts(), 0);
+ assert.equal((await store.get(scope))?.preservationState, "paused");
+ const status = await first.computerStatus!(scope);
+ assert.equal(status.lifecycleState, "paused");
+ assert.equal(fake.current(scopeName())?.state, "paused");
+ const restarted = make({ client, store, snapshots: portable.store });
+ const resumed = await restarted.provision(layers);
+ assert.equal(await restarted.readFile(resumed, "work.txt"), "keep");
+});
+
+test("legacy metadata adopts paused native state without reading a broken portable snapshot", async () => {
+ const store = createMemoryMap<StoredE2bSandbox>();
+ const portable = instrumentedSnapshotStore();
+ const first = make({ store, snapshots: portable.store });
+ const handle = await first.provision(layers);
+ await first.writeFile(handle, "unpublished.txt", "newest native contents");
+ await first.teardown(handle);
+ const legacy = (await store.get(scope))!;
+ await store.put(scope, { sandboxId: legacy.sandboxId, createdAtMs: legacy.createdAtMs });
+ portable.failReads(true);
+ portable.failWrites(true);
+ const client = {
+ ...fake.client,
+ nativePause: true,
+ async info() {
+ return { state: fake.current(scopeName())!.state, expiresAtMs: Date.now() + 60_000, onTimeout: "pause" };
+ },
+ };
+ const restarted = make({ client, store, snapshots: portable.store });
+ const resumed = await restarted.provision(layers);
+ assert.equal(await restarted.readFile(resumed, "unpublished.txt"), "newest native contents");
+ await restarted.teardown(resumed);
+ assert.equal(fake.createdCount(scopeName()), 1);
+ assert.equal((await store.get(scope))?.nativePause, true);
+ assert.equal((await store.get(scope))?.preservationState, "paused");
+});
+
+test("pause failures are durable and visible and leave the source available for retry", async () => {
+ const store = createMemoryMap<StoredE2bSandbox>();
+ let fail = true;
+ const client = {
+ ...fake.client,
+ nativePause: true,
+ async create(options: Parameters<typeof fake.client.create>[0]) {
+ const session = await fake.client.create(options);
+ return {
+ ...session,
+ async pause() {
+ if (fail) throw new Error("provider pause unavailable");
+ await session.pause();
+ },
+ };
+ },
+ };
+ const first = make({ client, store });
+ const handle = await first.provision(layers);
+ await assert.rejects(first.teardown(handle), /pause unavailable/);
+ assert.equal((await store.get(scope))?.preservationState, "pause_failed");
+ assert.equal(fake.current(scopeName())?.state, "running");
+ fail = false;
+ await first.teardown(handle);
+ assert.equal((await store.get(scope))?.preservationState, "paused");
+ assert.equal((await store.get(scope))?.preservationError, undefined);
+});
+
+test("a lost native E2B sandbox requires explicit recovery instead of a blank replacement", async () => {
+ const store = createMemoryMap<StoredE2bSandbox>();
+ const client = { ...fake.client, nativePause: true };
+ const first = make({ client, store });
+ const handle = await first.provision(layers);
+ await first.teardown(handle);
+ fake.expirePaused();
+ const restarted = make({ client, store });
+ await assert.rejects(restarted.provision(layers), /explicitly import a recovery snapshot/);
+ assert.equal(fake.createdCount(scopeName()), 1);
+});
+
+test("legacy pause preserves dirty home after failed portable checkpoint and retries on an unused turn", async () => {
+ const store = createMemoryMap<StoredE2bSandbox>();
+ const portable = instrumentedSnapshotStore();
+ const first = make({ store, snapshots: portable.store });
+ const handle = await first.provision(layers);
+ await first.teardown(handle);
+ const resumed = await first.provision(layers);
+ await first.writeFile(resumed, "unsaved.txt", "needs checkpoint");
+ portable.failWrites(true);
+ await first.teardown(resumed);
+ assert.equal((await store.get(scope))?.homeDirty, true);
+ portable.failWrites(false);
+ const unused = await first.provision(layers);
+ await first.teardown(unused, { homeUnchanged: true });
+ assert.equal(portable.puts(), 2);
+ assert.equal((await store.get(scope))?.homeDirty, false);
+});
+
+test("legacy pause preserves dirty home while portable checkpoints are throttled", async () => {
+ const store = createMemoryMap<StoredE2bSandbox>();
+ const portable = instrumentedSnapshotStore();
+ const first = make({ store, snapshots: portable.store, snapshotIntervalMs: 60_000 });
+ const handle = await first.provision(layers);
+ await first.teardown(handle);
+ const resumed = await first.provision(layers);
+ await first.writeFile(resumed, "unsaved.txt", "needs checkpoint");
+ await first.teardown(resumed);
+ assert.equal(portable.puts(), 1);
+ assert.equal((await store.get(scope))?.homeDirty, true);
+});
diff --git a/test/home-snapshot.test.ts b/test/home-snapshot.test.ts
index 3a2d0d9e5..0dccff66f 100644
--- a/test/home-snapshot.test.ts
+++ b/test/home-snapshot.test.ts
@@ -1,7 +1,18 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
-import { mkdtempSync, readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, chmodSync } from "node:fs";
+import {
+ mkdtempSync,
+ readFileSync,
+ writeFileSync,
+ existsSync,
+ mkdirSync,
+ readdirSync,
+ chmodSync,
+ symlinkSync,
+ readlinkSync,
+ statSync,
+} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
@@ -217,7 +228,7 @@ test("the whole snapshot is bounded by one deadline", async () => {
assert.equal(rec.aborted, 1);
});
-test("an empty or missing snapshot hydrates nothing", async () => {
+test("a missing snapshot hydrates nothing", async () => {
const b = box();
assert.equal(await ops(b, createMemorySnapshotStore()).hydrateHome("scope", b), false);
});
@@ -268,3 +279,89 @@ test("snapshotDue skips an unused turn only when the stored home is known clean"
assert.equal(snapshotDue({ lastSnapshotMs: 100 }, { homeUnchanged: true }, 0, 101), true, "unknown counts as dirty");
assert.equal(snapshotDue(null, { homeUnchanged: true }, 0), true, "never snapshotted counts as dirty");
});
+
+test("snapshots preserve symlinks and empty directories without following links or archiving pruned contents", async () => {
+ const a = box();
+ fill(a.home, "project/source", 32);
+ fill(a.home, "project/node_modules/excluded", 32);
+ mkdirSync(join(a.home, "empty"));
+ symlinkSync("project/source", join(a.home, "link"));
+ symlinkSync("missing", join(a.home, "dangling"));
+ const store = createMemorySnapshotStore();
+ await ops(a, store).snapshotHome("scope", a);
+ const b = box();
+ await ops(b, store).hydrateHome("scope", b);
+ assert.equal(readlinkSync(join(b.home, "link")), "project/source");
+ assert.equal(readlinkSync(join(b.home, "dangling")), "missing");
+ assert.ok(statSync(join(b.home, "empty")).isDirectory());
+ assert.equal(existsSync(join(b.home, "project/node_modules")), false);
+});
+
+for (const lengthDelta of [-1, 1]) {
+ test(`hydrate rejects a stream whose size differs by ${lengthDelta} before touching home files`, async () => {
+ const a = box();
+ fill(a.home, "existing", 32);
+ const store = createMemorySnapshotStore();
+ await ops(a, store).snapshotHome("scope", a);
+ const mismatched: HomeSnapshotStore = {
+ ...store,
+ async open(scope) {
+ const snapshot = await store.open(scope);
+ return snapshot && { ...snapshot, size: snapshot.size + lengthDelta };
+ },
+ };
+ const b = box();
+ writeFileSync(join(b.home, "existing"), "keep me");
+ await assert.rejects(ops(b, mismatched).hydrateHome("scope", b), /received .* bytes, expected/);
+ assert.equal(readFileSync(join(b.home, "existing"), "utf8"), "keep me");
+ assert.equal(existsSync(b.tar), false);
+ });
+}
+
+test("hydrate detects truncated guest writes before extraction", async () => {
+ const a = box();
+ fill(a.home, "existing", 32);
+ const store = createMemorySnapshotStore();
+ await ops(a, store).snapshotHome("scope", a);
+ const b = box();
+ writeFileSync(join(b.home, "existing"), "keep me");
+ const truncatedIo = {
+ ...io,
+ async writeFileBytes(target: Box, path: string, bytes: Uint8Array) {
+ await io.writeFileBytes(target, path, bytes.subarray(0, bytes.length - 1));
+ },
+ };
+ await assert.rejects(ops(b, store, { io: truncatedIo }).hydrateHome("scope", b), /wrote .* bytes, expected/);
+ assert.equal(readFileSync(join(b.home, "existing"), "utf8"), "keep me");
+ assert.equal(existsSync(b.tar), false);
+});
+
+test("hydrate validates archive structure before extraction", async () => {
+ const store = createMemorySnapshotStore();
+ await store.put("scope", Buffer.from("not a tar archive"));
+ const b = box();
+ writeFileSync(join(b.home, "existing"), "keep me");
+ await assert.rejects(ops(b, store).hydrateHome("scope", b), /archive invalid/);
+ assert.equal(readFileSync(join(b.home, "existing"), "utf8"), "keep me");
+ assert.equal(existsSync(b.tar), false);
+});
+
+for (const size of [0, -1, NaN, Infinity, 1.5]) {
+ test(`hydrate refuses invalid declared size ${size}`, async () => {
+ const store = createMemorySnapshotStore();
+ const invalid: HomeSnapshotStore = {
+ ...store,
+ async open() {
+ return {
+ size,
+ parts: (async function* () {
+ yield Buffer.alloc(1);
+ })(),
+ };
+ },
+ };
+ const b = box();
+ await assert.rejects(ops(b, invalid).hydrateHome("scope", b), /invalid snapshot size/);
+ assert.equal(existsSync(b.tar), false);
+ });
+}
diff --git a/test/modal-sandbox.test.ts b/test/modal-sandbox.test.ts
index 1c8e56531..be33a54f0 100644
--- a/test/modal-sandbox.test.ts
+++ b/test/modal-sandbox.test.ts
@@ -26,6 +26,7 @@ function make(extra: Record<string, unknown> = {}): Sandbox {
return createModalSandbox(createLocalWorkspaceStore(mkdtempSync(join(tmpdir(), "modal-ws-"))), {
client: fake.client,
namePrefix: "qmt",
+ nativeSnapshotsEnabled: true,
...extra,
});
}
@@ -136,7 +137,7 @@ test("sandbox is reused across provisions and warm start is reported", async ()
});
test("an existing live sandbox holding the scope name is adopted after a restart", async () => {
- await fake.client.create({ name: scopeName() });
+ await make().provision(layers);
const h = await sandbox.provision(layers);
assert.equal(h.coldStart, false);
assert.equal(fake.createdCount(scopeName()), 1);
@@ -156,7 +157,7 @@ test("a create race with another core instance adopts the winner instead of erro
return fake.client.fromName(name);
},
};
- await fake.client.create({ name: scopeName() });
+ await make().provision(layers);
const s = createModalSandbox(createLocalWorkspaceStore(mkdtempSync(join(tmpdir(), "modal-ws-"))), {
client: racing,
namePrefix: "qmt",
@@ -480,16 +481,13 @@ test("reapDeepIdle spares a box running a detached background job", async () =>
assert.equal(fake.current(scopeName())?.state, "running");
});
-test("snapshots are refused for a box adopted before it ever finished hydrating", async () => {
+test("unhydrated named homes are refused before adoption and never overwrite checkpoints", async () => {
await fake.client.create({ name: scopeName() });
const counting = instrumentedSnapshotStore();
- const errors: string[] = [];
- const s = make({ snapshots: counting.store, onError: (e: { code: string }) => errors.push(e.code) });
- const h = await s.provision(layers);
- await s.teardown(h);
- assert.equal(counting.puts(), 0, "an unhydrated home must never overwrite the stored snapshot");
- assert.ok(errors.includes("teardown_snapshot_failed"));
- assert.equal(fake.current(h.id)?.state, "running", "the box survives; only the snapshot is refused");
+ const s = make({ snapshots: counting.store });
+ await assert.rejects(s.provision(layers), /never finished hydrating/);
+ assert.equal(counting.puts(), 0);
+ assert.equal(fake.current(scopeName())?.state, "running");
});
test("rotation aborts and keeps the old box when terminating it fails", async () => {
@@ -542,3 +540,256 @@ test("teardown of a box the turn never used skips the snapshot only while the st
assert.equal(counting.puts(), 2, "the next unused turn catches up the missed snapshot");
assert.equal((await store.get(scope))?.homeDirty, false);
});
+
+test("native checkpoints restore across rotation and core restarts without transferring a tar", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const portable = instrumentedSnapshotStore();
+ const first = make({ store, snapshots: portable.store });
+ const handle = await first.provision(layers);
+ await first.writeFile(handle, "uncommitted.txt", "keep me");
+ await first.teardown(handle);
+ const checkpoint = (await store.get(scope))?.nativeSnapshotId;
+ assert.ok(checkpoint);
+ assert.equal(portable.puts(), 0);
+ await store.merge(scope, { createdAtMs: 0 });
+ const restarted = make({ store, snapshots: portable.store });
+ const next = await restarted.provision(layers);
+ assert.equal(await restarted.readFile(next, "uncommitted.txt"), "keep me");
+ assert.equal(portable.puts(), 0);
+ assert.ok(!fake.execScripts().some((script) => script.includes("tar --null")));
+ const status = await restarted.computerStatus!(scope);
+ assert.equal(status.recovery?.strategy, "provider_snapshot");
+ assert.ok(status.recovery?.checkpointExpiresAtMs);
+});
+
+test("native checkpoint references survive deep-idle reaping", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const first = make({ store });
+ const handle = await first.provision(layers);
+ await first.writeFile(handle, "work.txt", "durable checkpoint");
+ await store.merge(scope, { lastActivityMs: 1 });
+ assert.equal((await first.reapDeepIdle!(1)).reaped, 1);
+ assert.ok((await store.get(scope))?.nativeSnapshotId);
+ const restarted = make({ store });
+ const restored = await restarted.provision(layers);
+ assert.equal(await restarted.readFile(restored, "work.txt"), "durable checkpoint");
+});
+
+test("expired native checkpoints block replacement without falling back to stale portable data", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const first = make({ store });
+ const handle = await first.provision(layers);
+ await first.teardown(handle);
+ fake.terminate(scopeName());
+ await store.merge(scope, { nativeSnapshotExpiresAtMs: 1 });
+ const restarted = make({ store });
+ await assert.rejects(restarted.provision(layers), /checkpoint has expired/);
+ assert.equal(fake.createdCount(scopeName()), 1);
+ assert.match((await store.get(scope))?.recoveryError ?? "", /expired/);
+});
+
+test("native checkpoint failure preserves the previous reference and keeps the source running", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ let fail = false;
+ const wrapped: ModalClient = {
+ ...fake.client,
+ async create(options) {
+ const session = await fake.client.create(options);
+ return {
+ ...session,
+ async snapshotHome() {
+ if (fail) throw new Error("provider checkpoint unavailable");
+ return session.snapshotHome!();
+ },
+ };
+ },
+ };
+ const first = make({ store, client: wrapped, rotationHoldMs: 0 });
+ const handle = await first.provision(layers);
+ await first.teardown(handle);
+ const checkpoint = (await store.get(scope))?.nativeSnapshotId;
+ fail = true;
+ await store.merge(scope, { createdAtMs: 0 });
+ await first.provision(layers);
+ assert.equal(fake.createdCount(scopeName()), 1);
+ assert.equal((await store.get(scope))?.nativeSnapshotId, checkpoint);
+ assert.match((await store.get(scope))?.recoveryError ?? "", /unavailable/);
+});
+
+test("native scheduling ignores disabled legacy tar intervals and refreshes active homes in maintenance", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const first = make({ store, client: { ...fake.client, nativeSnapshots: true }, snapshotIntervalMs: 1e15 });
+ const handle = await first.provision(layers);
+ await first.teardown(handle);
+ const initial = (await store.get(scope))?.nativeSnapshotId;
+ assert.ok(initial);
+ await store.merge(scope, { lastSnapshotMs: 1 });
+ await first.writeFile(handle, "background.txt", "new background output");
+ await first.reapDeepIdle!(3600_000);
+ assert.notEqual((await store.get(scope))?.nativeSnapshotId, initial);
+ fake.terminate(scopeName());
+ const restarted = make({ store });
+ const recovered = await restarted.provision(layers);
+ assert.equal(await restarted.readFile(recovered, "background.txt"), "new background output");
+});
+
+test("a late checkpoint from another core cannot replace a newer committed checkpoint", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ let release!: () => void;
+ let started!: () => void;
+ const wait = new Promise<void>((resolve) => {
+ release = resolve;
+ });
+ const entered = new Promise<void>((resolve) => {
+ started = resolve;
+ });
+ let calls = 0;
+ const wrap = (session: Awaited<ReturnType<ModalClient["create"]>>) => ({
+ ...session,
+ async snapshotHome() {
+ const snapshot = await session.snapshotHome!();
+ if (++calls === 1) {
+ started();
+ await wait;
+ }
+ return snapshot;
+ },
+ });
+ const client: ModalClient = {
+ ...fake.client,
+ create: async (options) => wrap(await fake.client.create(options)),
+ fromId: async (id) => wrap(await fake.client.fromId(id)),
+ };
+ const first = make({ store, client });
+ const second = make({ store, client });
+ const one = await first.provision(layers);
+ const two = await second.provision(layers);
+ const older = first.teardown(one);
+ await entered;
+ await second.teardown(two);
+ const newest = (await store.get(scope))?.nativeSnapshotId;
+ release();
+ await older;
+ assert.equal((await store.get(scope))?.nativeSnapshotId, newest);
+ assert.equal((await store.get(scope))?.snapshotGeneration, 2);
+});
+
+test("maintenance checkpoints an idle scope with a live background job without reaping it", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const first = make({ store, client: { ...fake.client, nativeSnapshots: true } });
+ const handle = await first.provision(layers);
+ assert.ok(supportsProcessSessions(first));
+ if (!supportsProcessSessions(first)) return;
+ await first.startProcess(handle, "sleep 5");
+ await first.teardown(handle);
+ const checkpoint = (await store.get(scope))?.nativeSnapshotId;
+ await first.writeFile(handle, "background.txt", "new output");
+ await store.merge(scope, { lastActivityMs: Date.now() - 7 * 3600_000, lastSnapshotMs: 1 });
+ const result = await first.reapDeepIdle!(72 * 3600_000);
+ assert.equal(result.reaped, 0);
+ assert.equal(fake.current(scopeName())?.state, "running");
+ assert.notEqual((await store.get(scope))?.nativeSnapshotId, checkpoint);
+});
+
+test("native capture requires activation and adopted native scopes remain native after flag rollback", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const portable = instrumentedSnapshotStore();
+ const first = make({ store, snapshots: portable.store, nativeSnapshotsEnabled: undefined });
+ const handle = await first.provision(layers);
+ await first.writeFile(handle, "working.txt", "portable generation");
+ await first.teardown(handle);
+ assert.equal((await store.get(scope))?.nativeSnapshotId, undefined);
+ assert.equal(portable.puts(), 1);
+
+ const activated = make({ store, snapshots: portable.store, nativeSnapshotsEnabled: true });
+ const active = await activated.provision(layers);
+ await activated.writeFile(active, "working.txt", "native generation");
+ await store.merge(scope, { lastSnapshotMs: 0 });
+ await activated.teardown(active);
+ assert.ok((await store.get(scope))?.nativeSnapshotId);
+ assert.equal(portable.puts(), 1);
+ fake.terminate(scopeName());
+
+ const rollback = make({ store, snapshots: portable.store, nativeSnapshotsEnabled: false });
+ const recovered = await rollback.provision(layers);
+ assert.equal(await rollback.readFile(recovered, "working.txt"), "native generation");
+ await rollback.writeFile(recovered, "working.txt", "reader rollback generation");
+ await store.merge(scope, { lastSnapshotMs: 0 });
+ await rollback.teardown(recovered);
+ assert.equal(portable.puts(), 1);
+ fake.terminate(scopeName());
+ const restarted = make({ store, snapshots: portable.store, nativeSnapshotsEnabled: false });
+ const restored = await restarted.provision(layers);
+ assert.equal(await restarted.readFile(restored, "working.txt"), "reader rollback generation");
+});
+
+test("interrupted hydration cannot expose a partially restored home through stored adoption", async () => {
+ const store = createMemoryMap<StoredModalSandbox>();
+ const first = make({ store });
+ const handle = await first.provision(layers);
+ await first.writeFile(handle, "working.txt", "do not overwrite");
+ await store.merge(scope, { hydrationPending: true });
+ const restarted = make({ store });
+ await assert.rejects(restarted.provision(layers), /hydration was interrupted/);
+ assert.equal(fake.createdCount(scopeName()), 1);
+ const status = await restarted.computerStatus!(scope);
+ assert.match(status.recovery?.error ?? "", /computer restart/);
+ assert.equal((await store.get(scope))?.hydrationPending, true);
+});
+
+test("explicit restart of interrupted hydration retains the checkpoint and retries it", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const first = make({ store });
+ const handle = await first.provision(layers);
+ await first.writeFile(handle, "working.txt", "last complete checkpoint");
+ await first.teardown(handle);
+ const checkpoint = (await store.get(scope))!.nativeSnapshotId;
+ await first.writeFile(handle, "working.txt", "incomplete replacement content");
+ await store.merge(scope, { hydrationPending: true });
+ const restarted = make({ store, nativeSnapshotsEnabled: false });
+ await restarted.restartComputer!(scope);
+ assert.equal((await store.get(scope))!.nativeSnapshotId, checkpoint);
+ const restored = await restarted.provision(layers);
+ assert.equal(await restarted.readFile(restored, "working.txt"), "last complete checkpoint");
+});
+
+for (const path of ["stored", "name-conflict"] as const) {
+ test(`unhydrated homes cannot enter through ${path} adoption`, async () => {
+ const session = await fake.client.create({ name: scopeName() });
+ const store = createMemoryMap<StoredModalSandbox>();
+ if (path === "stored") await store.put(scope, { sandboxId: session.sandboxId, createdAtMs: Date.now() });
+ let first = true;
+ const client = {
+ ...fake.client,
+ async fromName(name: string) {
+ if (path === "name-conflict" && first) {
+ first = false;
+ return null;
+ }
+ return fake.client.fromName(name);
+ },
+ };
+ const backend = make({ store, client });
+ await assert.rejects(backend.provision(layers), /never finished hydrating/);
+ assert.equal(fake.current(scopeName())?.state, "running");
+ assert.equal(fake.createdCount(scopeName()), 1);
+ });
+}
diff --git a/test/native-provider-clients.test.ts b/test/native-provider-clients.test.ts
new file mode 100644
index 000000000..8bd9e78ae
--- /dev/null
+++ b/test/native-provider-clients.test.ts
@@ -0,0 +1,97 @@
+import { test, mock } from "node:test";
+import assert from "node:assert/strict";
+import { createSdkModalClient } from "../src/sandbox/modal-client.ts";
+import { createSdkE2bClient } from "../src/sandbox/e2b-client.ts";
+
+const modalCalls: unknown[][] = [];
+const modalSandbox = {
+ sandboxId: "sb-native",
+ async snapshotDirectory(path: string, options: unknown) {
+ modalCalls.push(["snapshot", path, options]);
+ return { imageId: "im-native" };
+ },
+ async mountImage(path: string, image: unknown) {
+ modalCalls.push(["mount", path, image]);
+ },
+};
+mock.module("modal", {
+ namedExports: {
+ ModalClient: class {
+ apps = { fromName: async () => ({}) };
+ images = { fromRegistry: () => ({}), fromId: async (imageId: string) => ({ imageId }) };
+ sandboxes = { create: async () => modalSandbox };
+ },
+ },
+});
+
+const e2bCalls: unknown[][] = [];
+let pauseError: Error | undefined;
+mock.module("e2b", {
+ namedExports: {
+ Sandbox: class {
+ static async create(template: string, options: unknown) {
+ e2bCalls.push(["create", template, options]);
+ return {
+ sandboxId: "e2b-native",
+ async pause(options: unknown) {
+ e2bCalls.push(["pause", options]);
+ if (pauseError) throw pauseError;
+ return false;
+ },
+ };
+ }
+ static async getInfo() {
+ e2bCalls.push(["info"]);
+ return { state: "paused", endAt: new Date(1000), lifecycle: { onTimeout: "pause" } };
+ }
+ },
+ },
+});
+
+test("Modal native directory snapshot uses explicit finite retention and restores the exact image", async () => {
+ const client = createSdkModalClient({
+ tokenId: "id",
+ tokenSecret: "secret",
+ appName: "test",
+ image: "ubuntu",
+ snapshotRetentionMs: 60_000,
+ });
+ const session = await client.create({ name: "test" });
+ const before = Date.now();
+ const snapshot = await session.snapshotHome!();
+ assert.deepEqual(modalCalls[0], ["snapshot", "/root", { ttlMs: 60_000 }]);
+ assert.ok(snapshot.expiresAtMs >= before + 60_000);
+ await session.restoreHome!(snapshot.imageId);
+ assert.deepEqual(modalCalls[1], ["mount", "/root", { imageId: "im-native" }]);
+ assert.equal(client.lifetimeMs, 24 * 3600_000);
+});
+
+test("Modal rejects invalid checkpoint retention", () => {
+ for (const snapshotRetentionMs of [0, -1, Infinity, NaN]) {
+ assert.throws(
+ () =>
+ createSdkModalClient({
+ tokenId: "id",
+ tokenSecret: "secret",
+ appName: "test",
+ image: "ubuntu",
+ snapshotRetentionMs,
+ }),
+ /positive finite/,
+ );
+ }
+});
+
+test("E2B sends explicit pause lifecycle, reads state without connect, and surfaces failed pause", async () => {
+ const client = createSdkE2bClient({ apiKey: "test" });
+ const session = await client.create({ metadata: { name: "test" }, autoPause: true });
+ assert.deepEqual((e2bCalls[0]![2] as { lifecycle: unknown }).lifecycle, { onTimeout: "pause", autoResume: false });
+ assert.equal(await client.info!(session.sandboxId).then((info) => info.state), "paused");
+ await session.pause();
+ assert.deepEqual(e2bCalls.at(-1), ["pause", { keepMemory: true }]);
+ pauseError = new Error("snapshot backlog");
+ await assert.rejects(session.pause(), /snapshot backlog/);
+ pauseError = undefined;
+ await client.create({ metadata: {}, autoPause: false });
+ assert.deepEqual((e2bCalls.at(-1)![2] as { lifecycle: unknown }).lifecycle, { onTimeout: "kill", autoResume: false });
+});
diff --git a/test/support/fake-modal.ts b/test/support/fake-modal.ts
index 26da849f5..96855fa5d 100644
--- a/test/support/fake-modal.ts
+++ b/test/support/fake-modal.ts
@@ -1,5 +1,5 @@
import { spawnSync } from "node:child_process";
-import { existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { cpSync, existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import {
@@ -32,7 +32,7 @@ export interface FakeModal {
cleanup(): void;
}
-export function installFakeModal(): FakeModal {
+export function installFakeModal(opts: { native?: boolean } = {}): FakeModal {
const root = mkdtempSync(join(tmpdir(), "fake-modal-"));
const records = new Map<string, FakeRecord>();
const execScripts: string[] = [];
@@ -68,6 +68,20 @@ export function installFakeModal(): FakeModal {
const session = (r: FakeRecord): ModalSession => ({
sandboxId: r.sandboxId,
+ ...(opts.native
+ ? {
+ async snapshotHome(): Promise<{ imageId: string; expiresAtMs: number }> {
+ alive(r);
+ const imageId = `im-${nextId++}`;
+ cpSync(r.home, join(root, imageId), { recursive: true });
+ return { imageId, expiresAtMs: Date.now() + 30 * 24 * 3600_000 };
+ },
+ async restoreHome(imageId: string): Promise<void> {
+ alive(r);
+ cpSync(join(root, imageId), r.home, { recursive: true });
+ },
+ }
+ : {}),
async runCommand(command): Promise<ModalCommandResult> {
alive(r);
execScripts.push(command);
@@ -105,6 +119,7 @@ export function installFakeModal(): FakeModal {
});
const client: ModalClient = {
+ nativeSnapshots: opts.native ?? false,
async create(opts): Promise<ModalSession> {
if (opts.name) {
const existing = byName(opts.name);