Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

Align the Modal backend with Modal sandbox docs (#1420)

Regan Bell440d5a33b102
diff --git a/docs/modal-image.md b/docs/modal-image.md
index 3c788fc79..973bf6639 100644
--- a/docs/modal-image.md
+++ b/docs/modal-image.md
@@ -32,6 +32,14 @@ or replace image-installed dependencies. Core validates the baked artifact again
its own bundle and exposes the matching copy through the usual home launcher path;
an older image can use the runtime bundle fallback.
+Native home checkpoints are directory snapshots of `/root`, so everything installed
+under the home is captured again in every checkpoint. Deployment-layer tool files are
+converged under `/usr/local/bin` and `/usr/local/lib` and stay out of checkpoints;
+the runtime connector bundle fallback lands under `/root/.qm/composio` and adds about
+1.3 MB to each checkpoint until `MODAL_IMAGE` points at a baked image, after which
+only a symlink remains in the home. Set `MODAL_IMAGE` before enabling native
+checkpoints on a deployment with many active users.
+
Rebuild the image when updating the SDK lockfile or builder, and retain the previous
image ID with the previous core release for rollback. Existing running sandboxes
keep their current image until normal replacement. Do not delete image IDs still
diff --git a/docs/sandbox-preservation.md b/docs/sandbox-preservation.md
index 9b42f864e..94f032c49 100644
--- a/docs/sandbox-preservation.md
+++ b/docs/sandbox-preservation.md
@@ -9,9 +9,9 @@ The agent runs in core. Sandbox home state is working state with a provider-spec
When enabled, Modal uses native directory checkpoints of `/root`, mounted into a fresh sandbox when the current sandbox is replaced. It does not stream the home through core during routine checkpointing or rollover. The base image supplies system dependencies outside the home; a home checkpoint does not preserve processes, RAM, or system-wide package installs. Runtime credential files under `/tmp` are outside the directory checkpoint; ordinary home files and symlinks are included.
-`MODAL_NATIVE_SNAPSHOT_INTERVAL_SEC` defaults to 300 and controls checkpoint throttling independently of the legacy `MODAL_SNAPSHOT_INTERVAL_SEC`. Used-turn teardown, replacement, and idle reaping capture checkpoints. The existing deep-idle maintenance sweep also checkpoints active machines when due, including machines running background work. This is best-effort recovery, not a guarantee that a process running into the provider deadline has saved its latest writes. Background maintenance must be enabled; the sweep cadence depends on the configured idle intervals. Captures while processes write are not application-consistent database backups.
+`MODAL_NATIVE_SNAPSHOT_INTERVAL_SEC` defaults to 300 and controls checkpoint throttling independently of the legacy `MODAL_SNAPSHOT_INTERVAL_SEC`. Used-turn teardown, replacement, and idle reaping capture checkpoints. The sandbox maintenance sweep runs every five minutes whenever background maintenance is enabled and either idle window is positive; it checkpoints active machines when due, including machines running background work. A machine within thirty minutes of Modal's 24-hour sandbox lifetime is checkpointed and terminated by the sweep even while background work runs, because Modal would otherwise kill that work unsaved; the next provision restores the checkpoint into a fresh machine. This is best-effort recovery, not a guarantee that a process running into the provider deadline has saved its latest writes. Captures while processes write are not application-consistent database backups.
-`MODAL_SNAPSHOT_RETENTION_SEC` defaults to 2592000 (30 days) and must be positive. Checkpoint IDs, capture timestamps, errors and expiration times are stored with sandbox records. `DATABASE_URL` must point to durable Postgres to retain these references across core restarts; without it the sandbox record map is memory-only. Status reports `recovery.checkpointExpiresAtMs`; using a checkpoint does not renew its expiry. Expired or failed native restores stop with an explicit recovery error instead of replacing the home with an empty directory or silently using an older portable backup. Idle machines whose checkpoints expire are not automatically renewed. A core crash during hydration leaves a durable pending marker; another core refuses to expose that potentially incomplete home until an operator confirms the original core is no longer hydrating and invokes `sandbox` with `action: "restart"`. Restart terminates only the incomplete replacement, retains checkpoint references, and clears the pending marker; the next provision retries that retained checkpoint. It does not archive or promote the incomplete home. Handled restore failures can retry after successful cleanup of the failed replacement.
+`MODAL_SNAPSHOT_RETENTION_SEC` defaults to 2592000 (30 days) and must be positive. Checkpoint IDs, capture timestamps, errors and expiration times are stored with sandbox records. `DATABASE_URL` must point to durable Postgres to retain these references across core restarts; without it the sandbox record map is memory-only. Status reports `recovery.checkpointExpiresAtMs`; using a checkpoint does not renew its expiry, because Modal retains a directory snapshot for a hard cutoff measured from creation. The maintenance sweep therefore renews the checkpoint of a reaped idle scope once less than seven days (or half the checkpoint's lifetime, whichever is shorter) remain: it restores the checkpoint into a fresh machine, captures a new checkpoint, and terminates that machine again, so an idle user never loses a home to retention alone. Renewal holds the same lifecycle lock as provisioning and leaves the scope's idle history untouched. Expired or failed native restores stop with an explicit recovery error instead of replacing the home with an empty directory or silently using an older portable backup. A core crash during hydration leaves a durable pending marker; another core refuses to expose that potentially incomplete home until an operator confirms the original core is no longer hydrating and invokes `sandbox` with `action: "restart"`. Restart terminates only the incomplete replacement, retains checkpoint references, and clears the pending marker; the next provision retries that retained checkpoint. It does not archive or promote the incomplete home. Handled restore failures can retry after successful cleanup of the failed replacement.
E2B creates persistent machines with an explicit pause-on-timeout lifecycle and pauses them at teardown unless `keepWarm` retains running background work. The provider timeout still applies; `keepWarm` does not extend that timeout. Routine native pause does not create a second tar backup. Provider pause retains disk and normally memory; provider fallback may retain disk only, so reconnectable processes must not depend on RAM surviving. Status can inspect a paused machine without waking it. Pause failures are recorded and returned, with the running machine retained for retry. A lost native machine requires explicit recovery rather than an automatic blank replacement. Existing E2B machines whose lifecycle does not confirm pause-on-timeout retain legacy portable checkpointing.
@@ -20,3 +20,9 @@ Sprites keep the whole disk across sleeps, so the sprite itself is the durable h
Smolmachines machines keep their disk across stop and start, and an idle machine stops after `SMOLMACHINES_AUTOSTOP_SEC` seconds without activity when that knob is set (the provider sweeps every 15 seconds; an in-flight exec keeps the machine alive). The provider documents that stopping is not a backup and that the cloud API has no machine snapshot; export to a `.smolmachine` artifact is the only native capture and it is a whole-disk registry artifact, so core keeps portable home tars instead. Every non-scratch teardown snapshots the home to `SMOLMACHINES_SNAPSHOT_S3_BUCKET` when the home changed, throttled by `SMOLMACHINES_SNAPSHOT_INTERVAL_SEC`; a replacement machine for a scope hydrates from that tar and a failed hydration deletes the incomplete replacement instead of adopting it. Initialization is marked pending in the durable sandbox record before create, so a lost create response or core restart cannot make the next core skip restoration. Provisioning, capture, and destruction share a lifecycle lock across cores; initialization is only marked complete after hydration succeeds. Status reports `recovery.strategy: "workspace_snapshot"` with the last capture time and error. Without the bucket the backend offers no `persistHomeSnapshot` and reports no recovery. Scratch machines are created ephemeral with a 24-hour `ttlSeconds` so a core crash cannot leave them running.
Explicit home export/import and `persistHomeSnapshot` continue to use the portable tar format for cross-provider moves and chosen recovery. Configure `MODAL_SNAPSHOT_S3_BUCKET`, `E2B_SNAPSHOT_S3_BUCKET`, `SPRITES_SNAPSHOT_S3_BUCKET` or `SMOLMACHINES_SNAPSHOT_S3_BUCKET` to retain those portable copies durably. They are not silently selected instead of a failed newer provider checkpoint.
+
+Modal machines are created with a one-core, 2 GiB reservation (`MODAL_CPUS`, `MODAL_MEMORY_MB`), optional placement (`MODAL_REGIONS`, a comma-separated list; place machines beside `MODAL_SNAPSHOT_S3_BUCKET`), a 24-hour lifetime, and a provider idle timeout of twice `MODAL_REAP_IDLE_SEC` (12 hours by default) as a backstop when no core is sweeping. Every machine carries `qm-org`, `qm-prefix`, and `qm-kind` tags; the sweep lists running scope machines by tag and terminates any that no sandbox record or live session has claimed for ten minutes, so machines orphaned by store loss stop billing without operator action. Scratch machines are excluded because other cores hold them only in memory.
+
+When `MODAL_EGRESS_PROXY_URL` is set, machines are created with Modal's outbound allowlist limited to the proxy host (a domain allowlist for an https proxy on port 443, a CIDR allowlist for an IP address; other proxy URLs are rejected at startup because Modal domain allowlists pass only TLS on 443), and the profile reports `egressEnforcement: "domain"`. Command execution, file transfer, and checkpoints travel over Modal's control plane and are unaffected. Without the setting, core warns that Modal sandboxes run fail-open.
+
+Turn environment reaches Modal commands through the exec `env` parameter rather than inlined `export` statements, exec deadlines are rounded up to whole seconds with a single thirty-second grace beyond the in-guest `timeout`, and a command longer than 64 KiB is written to `/tmp` through the filesystem API and executed from there.
diff --git a/package-lock.json b/package-lock.json
index 42d08a9fd..f77ef7788 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -36,7 +36,7 @@
"fastify": "^5.12.1",
"jose": "^6.2.3",
"lru-cache": "^11.5.2",
- "modal": "^0.10.0",
+ "modal": "^0.10.1",
"officeparser": "^8.0.0",
"opencode-ai": "1.17.18",
"pg": "^8.13.1",
@@ -9162,9 +9162,9 @@
}
},
"node_modules/modal": {
- "version": "0.10.0",
- "resolved": "https://registry.npmjs.org/modal/-/modal-0.10.0.tgz",
- "integrity": "sha512-RNjl05KEizIYHRa3t43PBEOhiU2w6O+MXCzMLgkBdx3qSw+oA4dSspG3YiNFdCOZZEg8ikmFMcmtsr/xj+rASQ==",
+ "version": "0.10.1",
+ "resolved": "https://registry.npmjs.org/modal/-/modal-0.10.1.tgz",
+ "integrity": "sha512-nnsWoVZ4XNLHtCV5/wuu2avQsFkGK6e/KazGPil3VwXI1aKh8jJkpP++lMPKwKQhMpe4N6WJtSRR6NwiolCEkw==",
"license": "Apache-2.0",
"dependencies": {
"cbor-x": "^1.6.0",
diff --git a/package.json b/package.json
index 7c5c4b7ce..492d0c168 100644
--- a/package.json
+++ b/package.json
@@ -90,7 +90,7 @@
"fastify": "^5.12.1",
"jose": "^6.2.3",
"lru-cache": "^11.5.2",
- "modal": "^0.10.0",
+ "modal": "^0.10.1",
"officeparser": "^8.0.0",
"opencode-ai": "1.17.18",
"pg": "^8.13.1",
diff --git a/src/config.ts b/src/config.ts
index 4b11df2e8..5a1d5440f 100644
--- a/src/config.ts
+++ b/src/config.ts
@@ -1181,6 +1181,11 @@ export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config {
"[config] SANDBOX_BACKEND=sprites without SPRITES_EGRESS_PROXY_URL — sandboxes run with NO egress enforcement (fail-open); set SPRITES_EGRESS_PROXY_URL to the public egress proxy to force sandbox traffic through it.",
);
}
+ if (env.MODAL_TOKEN_ID && env.MODAL_TOKEN_SECRET && !env.MODAL_EGRESS_PROXY_URL) {
+ console.warn(
+ "[config] modal sandbox backend enabled without MODAL_EGRESS_PROXY_URL — sandboxes run with NO egress enforcement (fail-open); set MODAL_EGRESS_PROXY_URL to the public https egress proxy so Modal's outbound allowlist admits only that host.",
+ );
+ }
if (env.SANDBOX_BACKEND === "sprites" && !env.SPRITES_SNAPSHOT_S3_BUCKET) {
console.warn(
"[config] SANDBOX_BACKEND=sprites without SPRITES_SNAPSHOT_S3_BUCKET — retiring a computer deletes its sprite and every checkpoint irreversibly with no exported home; set SPRITES_SNAPSHOT_S3_BUCKET to export the home to S3 before a sprite is destroyed and to rehydrate a replacement.",
diff --git a/src/sandbox/exec-kill.ts b/src/sandbox/exec-kill.ts
index de0fe9235..ffadff18a 100644
--- a/src/sandbox/exec-kill.ts
+++ b/src/sandbox/exec-kill.ts
@@ -1,3 +1,4 @@
+import { randomUUID } from "node:crypto";
import { shq } from "../util/shell.ts";
export function pgidMarkerPath(uid: string): string {
@@ -10,7 +11,14 @@ export function killableScript(innerScript: string, uid: string): string {
__pi_exec_rc=$?
rm -f ${marker} 2>/dev/null
exit $__pi_exec_rc`;
- return `exec setsid sh -c ${shq(inner)}`;
+ if (Buffer.byteLength(inner, "utf8") <= 64 * 1024) return `exec setsid sh -c ${shq(inner)}`;
+ const script = shq(`/tmp/.exec-${uid}.sh`);
+ const delimiter = `QM_EXEC_${randomUUID().replaceAll("-", "")}`;
+ return `(umask 077; cat > ${script} <<'${delimiter}'
+rm -f ${script}
+${inner}
+${delimiter}
+) && exec setsid sh ${script}`;
}
export function killScript(uid: string): string {
diff --git a/src/sandbox/modal-client.ts b/src/sandbox/modal-client.ts
index 5c4136e42..4be571e5f 100644
--- a/src/sandbox/modal-client.ts
+++ b/src/sandbox/modal-client.ts
@@ -1,3 +1,5 @@
+import { randomUUID } from "node:crypto";
+import { isIP } from "node:net";
import { resolveModalImage } from "./modal-image.ts";
export interface ModalCommandResult {
@@ -8,6 +10,7 @@ export interface ModalCommandResult {
interface ModalRunOpts {
timeoutMs?: number;
+ env?: Record<string, string>;
}
export interface ModalSession {
@@ -34,6 +37,7 @@ export class ModalNameConflictError extends Error {
interface ModalCreateOpts {
name?: string;
+ tags?: Record<string, string>;
}
export interface ModalClient {
@@ -43,6 +47,7 @@ export interface ModalClient {
fromId(sandboxId: string): Promise<ModalSession>;
fromName(name: string): Promise<ModalSession | null>;
terminate(sandboxId: string): Promise<void>;
+ listRunning?(tags: Record<string, string>): AsyncIterable<string>;
}
export interface SdkModalClientOptions {
@@ -55,12 +60,24 @@ export interface SdkModalClientOptions {
memoryMb?: number;
regions?: string[];
sandboxTimeoutMs?: number;
+ idleTimeoutMs?: number;
maxCommandMs?: number;
snapshotRetentionMs?: number;
+ egressProxyUrl?: string;
}
const MAX_LIFETIME_MS = 24 * 3600_000;
+const DEFAULT_IDLE_TIMEOUT_MS = 12 * 3600_000;
const DEFAULT_MAX_COMMAND_MS = 3600_000;
+const DEFAULT_CPUS = 1;
+const DEFAULT_MEMORY_MB = 2048;
+export const MODAL_EXEC_GRACE_MS = 30_000;
+export const MODAL_MAX_EXEC_ARG_BYTES = 64 * 1024;
+const SNAPSHOT_CALL_TIMEOUT_MS = 600_000;
+
+function wholeSeconds(ms: number): number {
+ return Math.max(1000, Math.ceil(ms / 1000) * 1000);
+}
function errName(err: unknown): string {
return String((err as Error)?.name ?? "");
@@ -74,6 +91,10 @@ function isFileNotFoundError(err: unknown): boolean {
return errName(err) === "SandboxFilesystemNotFoundError" || /no such file or directory/i.test(errText(err));
}
+function isFileTooLargeError(err: unknown): boolean {
+ return errName(err) === "SandboxFilesystemFileTooLargeError";
+}
+
function isSandboxGoneError(err: unknown): boolean {
if (isFileNotFoundError(err)) return false;
const name = errName(err);
@@ -100,12 +121,28 @@ function isDeadlineError(err: unknown): boolean {
return /timeouterror$/i.test(errName(err)) || /deadline exceeded/i.test(errText(err));
}
+export function modalEgressAllowlist(
+ egressProxyUrl: string,
+): { outboundDomainAllowlist: string[] } | { outboundCidrAllowlist: string[] } {
+ const url = new URL(egressProxyUrl);
+ const host = url.hostname.replace(/^\[|\]$/g, "");
+ const ipVersion = isIP(host);
+ if (ipVersion) return { outboundCidrAllowlist: [`${host}/${ipVersion === 4 ? 32 : 128}`] };
+ if (url.protocol === "https:" && (url.port === "" || url.port === "443")) return { outboundDomainAllowlist: [host] };
+ throw new Error(
+ "MODAL_EGRESS_PROXY_URL must be an https URL on port 443 or an IP address: Modal domain allowlists pass only TLS traffic on port 443",
+ );
+}
+
export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
- const sandboxTimeoutMs = Math.min(opts.sandboxTimeoutMs ?? MAX_LIFETIME_MS, MAX_LIFETIME_MS);
+ const sandboxTimeoutMs = wholeSeconds(Math.min(opts.sandboxTimeoutMs ?? MAX_LIFETIME_MS, MAX_LIFETIME_MS));
+ const idleTimeoutMs = wholeSeconds(Math.min(opts.idleTimeoutMs ?? DEFAULT_IDLE_TIMEOUT_MS, sandboxTimeoutMs));
const maxCommandMs = opts.maxCommandMs ?? DEFAULT_MAX_COMMAND_MS;
const snapshotRetentionMs = opts.snapshotRetentionMs ?? 30 * 24 * 3600_000;
if (!Number.isFinite(snapshotRetentionMs) || snapshotRetentionMs <= 0)
throw new Error("Modal snapshot retention must be a positive finite duration");
+ const snapshotTtlMs = wholeSeconds(snapshotRetentionMs);
+ const network = opts.egressProxyUrl ? modalEgressAllowlist(opts.egressProxyUrl) : {};
type ModalSdk = typeof import("modal");
type SdkClient = InstanceType<ModalSdk["ModalClient"]>;
type SdkApp = Awaited<ReturnType<SdkClient["apps"]["fromName"]>>;
@@ -138,9 +175,20 @@ export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
const wrap = (sbx: SdkSandbox): ModalSession => ({
sandboxId: sbx.sandboxId,
async runCommand(command, runOpts): Promise<ModalCommandResult> {
- const timeoutMs = runOpts?.timeoutMs ?? maxCommandMs;
+ const commandMs = wholeSeconds(runOpts?.timeoutMs ?? maxCommandMs);
+ const timeoutMs = commandMs + MODAL_EXEC_GRACE_MS;
try {
- const p = await sbx.exec(["sh", "-c", command], { mode: "text", timeoutMs: timeoutMs + 30_000 });
+ let args = ["timeout", String(commandMs / 1000), "sh", "-c", command];
+ if (Buffer.byteLength(command, "utf8") > MODAL_MAX_EXEC_ARG_BYTES) {
+ const spooled = `/tmp/.qm-exec-${randomUUID()}.sh`;
+ await sbx.filesystem.writeBytes(Buffer.from(command, "utf8"), spooled);
+ args = ["sh", "-c", `timeout ${commandMs / 1000} sh ${spooled}; rc=$?; rm -f ${spooled}; exit $rc`];
+ }
+ const p = await sbx.exec(args, {
+ mode: "text",
+ timeoutMs,
+ ...(runOpts?.env && Object.keys(runOpts.env).length ? { env: runOpts.env } : {}),
+ });
const [stdout, stderr, exitCode] = await Promise.all([p.stdout.readText(), p.stderr.readText(), p.wait()]);
return { stdout, stderr, exitCode };
} catch (err) {
@@ -155,6 +203,11 @@ export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
} catch (err) {
if (isSandboxGoneError(err)) throw new ModalSandboxGoneError(sbx.sandboxId, errText(err));
if (isFileNotFoundError(err)) return null;
+ if (isFileTooLargeError(err))
+ throw new Error(
+ `modal read ${absPath}: the file exceeds Modal's filesystem read limit; move it with blob staging or split it`,
+ { cause: err },
+ );
throw err;
}
},
@@ -167,8 +220,8 @@ export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
}
},
async snapshotHome() {
- const expiresAtMs = Date.now() + snapshotRetentionMs;
- const image = await sbx.snapshotDirectory("/root", { ttlMs: snapshotRetentionMs, timeoutMs: 600_000 });
+ const expiresAtMs = Date.now() + snapshotTtlMs;
+ const image = await sbx.snapshotDirectory("/root", { ttlMs: snapshotTtlMs, timeoutMs: SNAPSHOT_CALL_TIMEOUT_MS });
return { imageId: image.imageId, expiresAtMs };
},
async restoreHome(imageId): Promise<void> {
@@ -190,10 +243,13 @@ export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
try {
const sbx = await client.sandboxes.create(app, image, {
...(createOpts.name ? { name: createOpts.name } : {}),
+ ...(createOpts.tags ? { tags: createOpts.tags } : {}),
timeoutMs: sandboxTimeoutMs,
- ...(opts.cpus !== undefined ? { cpu: opts.cpus } : {}),
- ...(opts.memoryMb !== undefined ? { memoryMiB: opts.memoryMb } : {}),
+ idleTimeoutMs,
+ cpu: opts.cpus ?? DEFAULT_CPUS,
+ memoryMiB: opts.memoryMb ?? DEFAULT_MEMORY_MB,
...(opts.regions?.length ? { regions: opts.regions } : {}),
+ ...network,
});
return wrap(sbx);
} catch (err) {
@@ -247,5 +303,18 @@ export function createSdkModalClient(opts: SdkModalClientOptions): ModalClient {
throw err;
}
},
+ async *listRunning(tags): AsyncIterable<string> {
+ const { client, app } = await loadCtx();
+ for await (const sbx of client.sandboxes.list({ appId: app.appId, tags })) {
+ let exited: number | null;
+ try {
+ exited = await sbx.poll();
+ } catch (err) {
+ if (isSandboxGoneError(err)) continue;
+ throw err;
+ }
+ if (exited === null) yield sbx.sandboxId;
+ }
+ },
};
}
diff --git a/src/sandbox/modal-sandbox.ts b/src/sandbox/modal-sandbox.ts
index b32ada11c..f9c10920b 100644
--- a/src/sandbox/modal-sandbox.ts
+++ b/src/sandbox/modal-sandbox.ts
@@ -57,6 +57,9 @@ const HOME_TAR = `${HOME_DIR}/.qm-home.tar`;
const HYDRATED_MARKER = `${HOME_DIR}/.qm-hydrated`;
const IN_MEMORY_ADOPT_MAX_BYTES = 256 * 1024 * 1024;
const ACTIVITY_TOUCH_INTERVAL_MS = 10 * 60_000;
+const swallowGone = (error: unknown): void => {
+ if (!(error instanceof ModalSandboxGoneError)) throw error;
+};
const SNAPSHOT_PRUNE = ["./.qm-hydrated", ...HOME_SNAPSHOT_PRUNE];
export interface StoredModalSandbox {
@@ -92,6 +95,9 @@ export interface ModalSandboxOptions extends BlobStagingOptions {
layerToolFiles?: () => readonly LayerInstallFile[];
fileChunkBytes?: number;
rotationHoldMs?: number;
+ lifetimeMarginMs?: number;
+ checkpointRenewMarginMs?: number;
+ orphanGraceMs?: number;
store?: DurableMap<StoredModalSandbox>;
snapshots?: HomeSnapshotStore;
onError?: (e: { category: string; code: string; message: string; scopeLabel?: string }) => void;
@@ -111,7 +117,16 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
const reapIdleMs = opts.reapIdleMs ?? 6 * 3600_000;
const fileChunkBytes = opts.fileChunkBytes ?? 64 * 1024 * 1024;
const rotationHoldMs = opts.rotationHoldMs ?? 10 * 60_000;
+ const lifetimeMarginMs = opts.lifetimeMarginMs ?? 30 * 60_000;
+ const checkpointRenewMarginMs = opts.checkpointRenewMarginMs ?? 7 * 24 * 3600_000;
+ const orphanGraceMs = opts.orphanGraceMs ?? 10 * 60_000;
const rotationHoldUntil = new Map<string, number>();
+ const orphanFirstSeenMs = new Map<string, number>();
+ const tags = (kind: "scope" | "scratch"): Record<string, string> => ({
+ "qm-org": configOrgId(),
+ "qm-prefix": prefix,
+ "qm-kind": kind,
+ });
const workspaceDir = `${HOME_DIR}/${WORKSPACE_BASENAME}`;
const store = opts.store ?? createMemoryMap<StoredModalSandbox>();
const snapshots = opts.snapshots ?? createMemorySnapshotStore();
@@ -207,7 +222,7 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
}
let session: ModalSession;
try {
- session = await client.create({ name });
+ session = await client.create({ name, tags: tags("scope") });
} catch (err) {
if (!(err instanceof ModalNameConflictError)) throw err;
const adopted = await client.fromName(name);
@@ -380,7 +395,7 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
scratchKeyByName.set(name, key);
const active = activeScratch.get(name) ?? 0;
if (active === 0 && !sessionByName.has(name)) {
- const session = await client.create({});
+ const session = await client.create({ tags: tags("scratch") });
sessionByName.set(name, session);
}
activeScratch.set(name, active + 1);
@@ -391,7 +406,7 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
async function withSessionUnlocked<T>(name: string, action: (session: ModalSession) => Promise<T>): Promise<T> {
const scratchKey = scratchKeyByName.get(name);
const reviveScratch = async (): Promise<ModalSession> => {
- const session = await client.create({});
+ const session = await client.create({ tags: tags("scratch") });
sessionByName.set(name, session);
return session;
};
@@ -432,21 +447,34 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
await store.merge(scope, { lastActivityMs: now }).catch(() => undefined);
}
- async function execRaw(name: string, script: string, timeoutSec: number): Promise<ExecResult> {
- await touchActivity(name);
- return withSession(name, async (session) => {
- const r = await session.runCommand(`timeout ${timeoutSec} sh -c ${shq(script)}`, {
- timeoutMs: timeoutSec * 1000 + 30_000,
- });
- return { stdout: r.stdout, stderr: r.stderr, code: r.exitCode, timedOut: r.exitCode === 124 };
+ async function runTimed(
+ session: ModalSession,
+ script: string,
+ timeoutSec: number,
+ env?: Record<string, string>,
+ ): Promise<ExecResult> {
+ const r = await session.runCommand(script, {
+ timeoutMs: timeoutSec * 1000,
+ ...(env ? { env } : {}),
});
+ return { stdout: r.stdout, stderr: r.stderr, code: r.exitCode, timedOut: r.exitCode === 124 };
+ }
+
+ async function execRaw(
+ name: string,
+ script: string,
+ timeoutSec: number,
+ env?: Record<string, string>,
+ ): Promise<ExecResult> {
+ await touchActivity(name);
+ return withSession(name, (session) => runTimed(session, script, timeoutSec, env));
}
const profile: AgentComputerProfile = {
backend: "modal",
writablePersistence: usesNativeSnapshots() ? "provider_managed" : "snapshot_to_workspace",
processSessions: true,
- egressEnforcement: "none",
+ egressEnforcement: opts.egressProxyUrl ? "domain" : "none",
spec: {
os: "Debian 12 — Modal sandbox (24h max lifetime; home checkpoints have limited retention; publish durable work to git or Files)",
runtimes: ["Node 24", "Python 3"],
@@ -470,12 +498,9 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
const procSessions = createExecProcessSessions(procIo);
const directProcIo = (session: ModalSession): ExecProcessIo => ({
- async run(_handle, command, execOpts): Promise<ExecResult> {
+ run(_handle, command, execOpts): Promise<ExecResult> {
const timeoutSec = execOpts?.timeoutMs ? Math.ceil(execOpts.timeoutMs / 1000) : defaultTimeoutSec;
- const r = await session.runCommand(`timeout ${timeoutSec} sh -c ${shq(command)}`, {
- timeoutMs: timeoutSec * 1000 + 30_000,
- });
- return { stdout: r.stdout, stderr: r.stderr, code: r.exitCode, timedOut: r.exitCode === 124 };
+ return runTimed(session, command, timeoutSec);
},
});
@@ -525,6 +550,131 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
lastTouchMs.delete(scope);
}
+ function checkpointRenewalDue(rec: StoredModalSandbox, now: number): boolean {
+ if (!rec.nativeSnapshotId || !rec.nativeSnapshotExpiresAtMs) return false;
+ const remaining = rec.nativeSnapshotExpiresAtMs - now;
+ const lifetime = rec.nativeSnapshotExpiresAtMs - (rec.lastSnapshotMs ?? 0);
+ return remaining > 0 && remaining <= Math.min(checkpointRenewMarginMs, lifetime / 2);
+ }
+
+ function renewCheckpoint(scope: string, name: string, rec: StoredModalSandbox): Promise<void> {
+ return advisoryLock.withLock(`modal-use:${scope}`, () =>
+ provisionQueue(scope, async () => {
+ const current = await store.get(scope);
+ if (
+ !current ||
+ current.hydrationPending ||
+ current.nativeSnapshotId !== rec.nativeSnapshotId ||
+ !checkpointRenewalDue(current, Date.now())
+ )
+ return;
+ try {
+ const { session } = await createHydrated(scope, name);
+ try {
+ await snapshotHome(scope, session);
+ } finally {
+ sessionByName.delete(name);
+ await session.terminate().catch(swallowGone);
+ }
+ await store.merge(scope, { lastActivityMs: current.lastActivityMs });
+ } catch (e) {
+ if (!(e instanceof ModalSandboxGoneError))
+ reportError("sandbox_snapshot", "checkpoint_renewal_failed", errMessage(e), scope);
+ }
+ }),
+ );
+ }
+
+ function refreshCheckpoint(scope: string, rec: StoredModalSandbox, session: ModalSession): Promise<void> {
+ return provisionQueue(scope, async () => {
+ try {
+ const current = await store.get(scope);
+ if (
+ !current ||
+ current.sandboxId !== rec.sandboxId ||
+ Date.now() - lastSnapshotAttemptMs(current) <= snapshotIntervalMs(current)
+ )
+ return;
+ await snapshotHome(scope, session);
+ } catch (error) {
+ if (!(error instanceof ModalSandboxGoneError))
+ reportError("sandbox_snapshot", "periodic_snapshot_failed", errMessage(error), scope);
+ }
+ });
+ }
+
+ function retire(
+ scope: string,
+ name: string,
+ rec: StoredModalSandbox,
+ session: ModalSession,
+ idleCutoffMs?: number,
+ ): Promise<number> {
+ return advisoryLock.withLock(`modal-use:${scope}`, () =>
+ provisionQueue(scope, async (): Promise<number> => {
+ const current = await store.get(scope);
+ if (!current || current.sandboxId !== rec.sandboxId) return 0;
+ if (idleCutoffMs !== undefined && (!current.lastActivityMs || current.lastActivityMs > idleCutoffMs)) return 0;
+ try {
+ await snapshotHome(scope, session);
+ await session.terminate();
+ sessionByName.delete(name);
+ if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope);
+ return 1;
+ } catch (e) {
+ if (e instanceof ModalSandboxGoneError) {
+ sessionByName.delete(name);
+ if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope).catch(() => undefined);
+ } else {
+ reportError("sandbox_reap", "deep_idle_reap_failed", errMessage(e), scope);
+ }
+ return 0;
+ }
+ }),
+ );
+ }
+
+ async function sweepOrphans(now: number): Promise<number> {
+ if (!client.listRunning) return 0;
+ const tracked = async (): Promise<Set<string>> => {
+ const ids = new Set<string>();
+ for (const [, rec] of await store.entries()) ids.add(rec.sandboxId);
+ for (const session of sessionByName.values()) ids.add(session.sandboxId);
+ return ids;
+ };
+ let known = await tracked();
+ const listed = new Set<string>();
+ const candidates: string[] = [];
+ try {
+ for await (const sandboxId of client.listRunning(tags("scope"))) {
+ listed.add(sandboxId);
+ if (known.has(sandboxId)) continue;
+ const firstSeen = orphanFirstSeenMs.get(sandboxId) ?? now;
+ orphanFirstSeenMs.set(sandboxId, firstSeen);
+ if (now - firstSeen >= orphanGraceMs) candidates.push(sandboxId);
+ }
+ } catch (e) {
+ reportError("sandbox_reap", "orphan_list_failed", errMessage(e));
+ return 0;
+ }
+ for (const id of orphanFirstSeenMs.keys()) if (!listed.has(id) || known.has(id)) orphanFirstSeenMs.delete(id);
+ if (!candidates.length) return 0;
+ known = await tracked();
+ let reaped = 0;
+ for (const sandboxId of candidates) {
+ if (known.has(sandboxId)) continue;
+ try {
+ await client.terminate(sandboxId);
+ orphanFirstSeenMs.delete(sandboxId);
+ reaped++;
+ } catch (e) {
+ if (e instanceof ModalSandboxGoneError) orphanFirstSeenMs.delete(sandboxId);
+ else reportError("sandbox_reap", "orphan_terminate_failed", errMessage(e));
+ }
+ }
+ return reaped;
+ }
+
const sandbox: Sandbox = {
destroyScope(scopeId: string): Promise<void> {
return provisionQueue(scopeId, () => destroyStoredScope(scopeId));
@@ -601,12 +751,9 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
async run(handle, command, execOpts?: ExecOptions): Promise<ExecResult> {
const timeoutSec = execOpts?.timeoutMs ? Math.ceil(execOpts.timeoutMs / 1000) : defaultTimeoutSec;
- const exports = Object.entries(handle.env ?? {})
- .map(([k, v]) => `export ${k}=${shq(v)}`)
- .join("; ");
- const script = `${nonInteractiveShellPrefix()}${exports ? exports + "; " : ""}cd ${handle.rootDir} 2>/dev/null; ${command}`;
+ const script = `${nonInteractiveShellPrefix()}cd ${handle.rootDir} 2>/dev/null; ${command}`;
const signal = execOpts?.signal;
- if (!signal) return execRaw(handle.id, script, timeoutSec);
+ if (!signal) return execRaw(handle.id, script, timeoutSec, handle.env);
const killUid = randomUUID();
const fireKill = () => {
execRaw(handle.id, killScript(killUid), 15).catch(swallowAs("modal-sandbox: kill in-flight exec", undefined));
@@ -615,7 +762,7 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
const onAbort = () => fireKill();
signal.addEventListener("abort", onAbort, { once: true });
try {
- return await execRaw(handle.id, killableScript(script, killUid), timeoutSec);
+ return await execRaw(handle.id, killableScript(script, killUid), timeoutSec, handle.env);
} finally {
signal.removeEventListener("abort", onAbort);
}
@@ -659,9 +806,6 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
const session = sessionByName.get(name);
sessionByName.delete(name);
const stored = await store.get(scopeId);
- const swallowGone = (e: unknown): void => {
- if (!(e instanceof ModalSandboxGoneError)) throw e;
- };
if (session) await session.terminate().catch(swallowGone);
else if (stored) await client.terminate(stored.sandboxId).catch(swallowGone);
await store.delete(scopeId);
@@ -789,74 +933,57 @@ export function createModalSandbox(workspace: WorkspaceStore, opts: ModalSandbox
async reapDeepIdle(idleMs): Promise<{ reaped: number }> {
if (!(idleMs > 0)) return { reaped: 0 };
- const cutoff = Date.now() - Math.min(idleMs, reapIdleMs);
+ const now = Date.now();
+ const cutoff = now - Math.min(idleMs, reapIdleMs);
let reaped = 0;
for (const [scope, rec] of await store.entries()) {
if (rec.orgId && rec.orgId !== configOrgId()) continue;
if (rec.hydrationPending) continue;
- if (
- usesNativeSnapshots(rec) &&
- (!rec.expiresAtMs || rec.expiresAtMs > Date.now()) &&
- Date.now() - lastSnapshotAttemptMs(rec) > snapshotIntervalMs(rec)
- ) {
- await provisionQueue(scope, async () => {
- try {
- const current = await store.get(scope);
- if (
- !current ||
- current.sandboxId !== rec.sandboxId ||
- Date.now() - lastSnapshotAttemptMs(current) <= snapshotIntervalMs(current)
- )
- return;
- const session = await client.fromId(rec.sandboxId);
- await snapshotHome(scope, session);
- } catch (error) {
- if (!(error instanceof ModalSandboxGoneError))
- reportError("sandbox_snapshot", "periodic_snapshot_failed", errMessage(error), scope);
- }
- });
- }
- if (!rec.lastActivityMs || rec.lastActivityMs > cutoff) continue;
const name = sandboxScopeName(prefix, scope);
scopeByName.set(name, scope);
- let session: ModalSession;
+ let session: ModalSession | undefined;
try {
session = sessionByName.get(name) ?? (await client.fromId(rec.sandboxId));
- const handle: SandboxHandle = { id: name, rootDir: workspaceDir, homeDir: HOME_DIR, coldStart: false };
- const live = await createExecProcessSessions(directProcIo(session)).listProcesses(handle);
- if (live.some((p) => p.status.state === "running")) continue;
} catch (e) {
- if (e instanceof ModalSandboxGoneError) {
- sessionByName.delete(name);
- if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope).catch(() => undefined);
- } else {
+ if (!(e instanceof ModalSandboxGoneError)) {
reportError("sandbox_reap", "deep_idle_probe_failed", errMessage(e), scope);
+ continue;
}
+ }
+ if (!session) {
+ sessionByName.delete(name);
+ if (!rec.nativeSnapshotId) await store.delete(scope).catch(() => undefined);
+ else if (checkpointRenewalDue(rec, now)) await renewCheckpoint(scope, name, rec);
continue;
}
- reaped += await advisoryLock.withLock(`modal-use:${scope}`, () =>
- provisionQueue(scope, async (): Promise<number> => {
- const current = await store.get(scope);
- if (!current || current.sandboxId !== rec.sandboxId) return 0;
- if (!current.lastActivityMs || current.lastActivityMs > cutoff) return 0;
- try {
- await snapshotHome(scope, session);
- await session.terminate();
+ if (rec.expiresAtMs && rec.expiresAtMs - now <= lifetimeMarginMs) {
+ reaped += await retire(scope, name, rec, session);
+ continue;
+ }
+ if (rec.lastActivityMs && rec.lastActivityMs <= cutoff) {
+ let busy: boolean;
+ try {
+ const handle: SandboxHandle = { id: name, rootDir: workspaceDir, homeDir: HOME_DIR, coldStart: false };
+ const live = await createExecProcessSessions(directProcIo(session)).listProcesses(handle);
+ busy = live.some((p) => p.status.state === "running");
+ } catch (e) {
+ if (e instanceof ModalSandboxGoneError) {
sessionByName.delete(name);
- if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope);
- return 1;
- } catch (e) {
- if (e instanceof ModalSandboxGoneError) {
- sessionByName.delete(name);
- if (!(await store.get(scope))?.nativeSnapshotId) await store.delete(scope).catch(() => undefined);
- } else {
- reportError("sandbox_reap", "deep_idle_reap_failed", errMessage(e), scope);
- }
- return 0;
+ if (!rec.nativeSnapshotId) await store.delete(scope).catch(() => undefined);
+ } else {
+ reportError("sandbox_reap", "deep_idle_probe_failed", errMessage(e), scope);
}
- }),
- );
+ continue;
+ }
+ if (!busy) {
+ reaped += await retire(scope, name, rec, session, cutoff);
+ continue;
+ }
+ }
+ if (usesNativeSnapshots(rec) && now - lastSnapshotAttemptMs(rec) > snapshotIntervalMs(rec))
+ await refreshCheckpoint(scope, rec, session);
}
+ reaped += await sweepOrphans(now);
return { reaped };
},
};
diff --git a/src/wiring.ts b/src/wiring.ts
index 240922d16..8e9be8275 100644
--- a/src/wiring.ts
+++ b/src/wiring.ts
@@ -927,7 +927,9 @@ export function buildApp(
...(modal.memoryMb !== undefined ? { memoryMb: modal.memoryMb } : {}),
...(modal.regions?.length ? { regions: modal.regions } : {}),
...(modal.sandboxTimeoutSec ? { sandboxTimeoutMs: modal.sandboxTimeoutSec * 1000 } : {}),
+ ...(modal.reapIdleSec ? { idleTimeoutMs: 2 * modal.reapIdleSec * 1000 } : {}),
...(modal.snapshotRetentionSec !== undefined ? { snapshotRetentionMs: modal.snapshotRetentionSec * 1000 } : {}),
+ ...(modal.egressProxyUrl ? { egressProxyUrl: modal.egressProxyUrl } : {}),
}),
...(modal.namePrefix ? { namePrefix: modal.namePrefix } : {}),
...(modal.defaultTimeoutSec ? { defaultTimeoutSec: modal.defaultTimeoutSec } : {}),
@@ -2490,17 +2492,15 @@ export function buildApp(
const keepWarmSweeper = createSweeper(() => app.keepAlwaysOnWarm(), KEEP_WARM_INTERVAL_MS);
const deepIdleMachineMs = config.deepIdleMachineMs;
const devIdleMachineMs = config.devIdleMachineMs;
- const sweepFractions = [deepIdleMachineMs, devIdleMachineMs]
- .filter((w): w is number => !!w && w > 0)
- .map((w) => Math.floor(w / 24));
- const deepIdleReapEnabled = Boolean(sandbox.reapDeepIdle && sweepFractions.length);
+ const SANDBOX_MAINTENANCE_INTERVAL_MS = 5 * 60_000;
+ const deepIdleReapEnabled = Boolean(sandbox.reapDeepIdle && (deepIdleMachineMs > 0 || devIdleMachineMs > 0));
const deepIdleSweeper = deepIdleReapEnabled
? createSweeper(
() =>
leaderLease.hold("sandbox:deep-idle-reaper", () =>
sandbox.reapDeepIdle!(deepIdleMachineMs, devIdleMachineMs),
),
- Math.max(60_000, Math.min(...sweepFractions)),
+ SANDBOX_MAINTENANCE_INTERVAL_MS,
{ immediate: true },
)
: null;
diff --git a/test/config.test.ts b/test/config.test.ts
index b403c8a94..7b39fdb34 100644
--- a/test/config.test.ts
+++ b/test/config.test.ts
@@ -192,6 +192,19 @@ test("production names a mock harness rather than letting it pass as a real depl
assert.match(mock[1]!, /HARNESS is "mock"/);
});
+test("Modal without an egress proxy warns that sandboxes run fail-open", () => {
+ const warnings: string[] = [];
+ const original = console.warn;
+ console.warn = (msg: unknown) => void warnings.push(String(msg));
+ try {
+ loadConfig({ MODAL_TOKEN_ID: "id", MODAL_TOKEN_SECRET: "secret" });
+ loadConfig({ MODAL_TOKEN_ID: "id", MODAL_TOKEN_SECRET: "secret", MODAL_EGRESS_PROXY_URL: "https://egress.test" });
+ } finally {
+ console.warn = original;
+ }
+ assert.equal(warnings.filter((w) => w.includes("MODAL_EGRESS_PROXY_URL")).length, 1);
+});
+
test("a leftover *=sqlite env throws (no silent downgrade to ephemeral memory)", () => {
assert.throws(() => loadConfig({ SESSION_STORE: "sqlite" }), /SESSION_STORE=sqlite is no longer supported/);
assert.throws(() => loadConfig({ RUN_STORE: "sqlite" }), /RUN_STORE=sqlite is no longer supported/);
diff --git a/test/modal-sandbox.test.ts b/test/modal-sandbox.test.ts
index f2d8488eb..aca31d2e4 100644
--- a/test/modal-sandbox.test.ts
+++ b/test/modal-sandbox.test.ts
@@ -13,6 +13,7 @@ import { supportsBlobStaging, supportsProcessSessions } from "../src/sandbox/san
import { createMemoryMap, type DurableMap } from "../src/persistence/durable-map.ts";
import { createMemoryBlobTransferStore } from "../src/persistence/blob-transfer.ts";
import { scopeId } from "../src/types.ts";
+import { orgId } from "../src/config.ts";
import { mintCapabilityToken, EGRESS_PROXY_AUD } from "../src/auth/capability-token.ts";
import { installFakeModal, type FakeModal } from "./support/fake-modal.ts";
import { ModalSandboxGoneError, type ModalClient } from "../src/sandbox/modal-client.ts";
@@ -321,6 +322,7 @@ test("profile advertises snapshot persistence and process sessions", () => {
assert.equal(sandbox.profile.writablePersistence, "snapshot_to_workspace");
assert.equal(sandbox.profile.processSessions, true);
assert.equal(sandbox.profile.egressEnforcement, "none");
+ assert.equal(make({ egressProxyUrl: "https://proxy.example.com" }).profile.egressEnforcement, "domain");
});
test("file reads and writes revive a sandbox that died mid-turn", async () => {
@@ -1035,3 +1037,110 @@ test("deep idle termination excludes writes from another core until recovery", {
assert.equal(await second.readFile(two, "after-checkpoint.txt"), "must survive");
assert.equal(fake.createdCount(scopeName()), 2);
});
+
+test("turn env reaches commands through the exec env parameter, never as inlined exports", async () => {
+ const h = await sandbox.provision(layers, { env: { MY_TOKEN: "hunter2" } });
+ const r = await sandbox.run(h, "echo TOKEN=$MY_TOKEN");
+ assert.match(r.stdout, /TOKEN=hunter2/);
+ assert.ok(!fake.execScripts().some((script) => script.includes("hunter2")));
+});
+
+test("scope and scratch sandboxes carry ownership tags", async () => {
+ const h = await sandbox.provision(layers);
+ assert.deepEqual(fake.tagsOf(fake.current(h.id)!.sandboxId), {
+ "qm-org": orgId(),
+ "qm-prefix": "qmt",
+ "qm-kind": "scope",
+ });
+ const scratch = await sandbox.provision(layers, { scratch: { key: "worker-1" } });
+ const scratchIds: string[] = [];
+ for await (const id of fake.client.listRunning!({ "qm-kind": "scratch" })) scratchIds.push(id);
+ assert.equal(scratchIds.length, 1);
+ assert.deepEqual(fake.tagsOf(scratchIds[0]!), { "qm-org": orgId(), "qm-prefix": "qmt", "qm-kind": "scratch" });
+ await sandbox.teardown(scratch);
+});
+
+test("a near-expiry checkpoint of a reaped scope is renewed before Modal deletes it", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const s = make({ store });
+ const handle = await s.provision(layers);
+ await s.writeFile(handle, "work.txt", "idle for a month");
+ await store.merge(scope, { lastActivityMs: 1 });
+ assert.equal((await s.reapDeepIdle!(1)).reaped, 1);
+ const parked = (await store.get(scope))!;
+ assert.ok(parked.nativeSnapshotId);
+ const day = 24 * 3600_000;
+ await store.merge(scope, { nativeSnapshotExpiresAtMs: Date.now() + 20 * day, lastSnapshotMs: Date.now() - 10 * day });
+ await s.reapDeepIdle!(1);
+ assert.equal((await store.get(scope))!.nativeSnapshotId, parked.nativeSnapshotId, "far from expiry: untouched");
+ await store.merge(scope, { nativeSnapshotExpiresAtMs: Date.now() + 3600_000, lastSnapshotMs: Date.now() - 29 * day });
+ await s.reapDeepIdle!(1);
+ const renewed = (await store.get(scope))!;
+ assert.notEqual(renewed.nativeSnapshotId, parked.nativeSnapshotId);
+ assert.ok(renewed.nativeSnapshotExpiresAtMs! > Date.now() + 29 * day);
+ assert.equal(renewed.lastActivityMs, 1, "renewal is maintenance, not user activity");
+ assert.equal(renewed.hydrationPending, false);
+ assert.equal(fake.current(scopeName()), null, "the renewal sandbox is terminated again");
+ assert.equal(fake.runningCount(), 0);
+ const restarted = make({ store });
+ const restored = await restarted.provision(layers);
+ assert.equal(await restarted.readFile(restored, "work.txt"), "idle for a month");
+});
+
+test("checkpoint renewal skips expired and short-lived checkpoints instead of looping", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const s = make({ store });
+ await s.provision(layers);
+ await store.merge(scope, { lastActivityMs: 1 });
+ await s.reapDeepIdle!(1);
+ const parked = (await store.get(scope))!;
+ await store.merge(scope, { nativeSnapshotExpiresAtMs: Date.now() + 40_000, lastSnapshotMs: Date.now() - 20_000 });
+ await s.reapDeepIdle!(1);
+ assert.equal((await store.get(scope))!.nativeSnapshotId, parked.nativeSnapshotId, "still in the first half-life");
+ await store.merge(scope, { nativeSnapshotExpiresAtMs: 1 });
+ await s.reapDeepIdle!(1);
+ assert.equal((await store.get(scope))!.nativeSnapshotId, parked.nativeSnapshotId, "expired: nothing to renew");
+ assert.equal(fake.createdCount(scopeName()), 1);
+});
+
+test("a sandbox approaching Modal's lifetime limit is checkpointed and retired even while busy", async () => {
+ fake.cleanup();
+ fake = installFakeModal({ native: true });
+ const store = createMemoryMap<StoredModalSandbox>();
+ const s = make({ store, client: { ...fake.client, lifetimeMs: 24 * 3600_000 } });
+ const handle = await s.provision(layers);
+ await s.writeFile(handle, "job.txt", "in flight");
+ assert.ok(supportsProcessSessions(s));
+ if (!supportsProcessSessions(s)) return;
+ await s.startProcess(handle, "sleep 30");
+ assert.equal((await s.reapDeepIdle!(72 * 3600_000)).reaped, 0, "a fresh busy sandbox is left alone");
+ await store.merge(scope, { expiresAtMs: Date.now() + 60_000 });
+ assert.equal((await s.reapDeepIdle!(72 * 3600_000)).reaped, 1);
+ assert.equal(fake.current(scopeName()), null);
+ assert.ok((await store.get(scope))!.nativeSnapshotId);
+ const next = await s.provision(layers);
+ assert.equal(await s.readFile(next, "job.txt"), "in flight");
+ assert.ok((await store.get(scope))!.expiresAtMs! > Date.now() + 23 * 3600_000);
+});
+
+test("untracked scope sandboxes are terminated after a grace period while scratch and other deployments are kept", async () => {
+ const store = createMemoryMap<StoredModalSandbox>();
+ const s = make({ store, orphanGraceMs: 40 });
+ const handle = await s.provision(layers);
+ const tracked = fake.current(handle.id)!.sandboxId;
+ const orphan = fake.createUntracked({ "qm-org": orgId(), "qm-prefix": "qmt", "qm-kind": "scope" });
+ const scratch = fake.createUntracked({ "qm-org": orgId(), "qm-prefix": "qmt", "qm-kind": "scratch" });
+ const foreign = fake.createUntracked({ "qm-org": orgId(), "qm-prefix": "other", "qm-kind": "scope" });
+ assert.equal((await s.reapDeepIdle!(72 * 3600_000)).reaped, 0, "first sighting starts the grace period");
+ assert.equal(fake.runningCount(), 4);
+ await new Promise((resolve) => setTimeout(resolve, 60));
+ assert.equal((await s.reapDeepIdle!(72 * 3600_000)).reaped, 1);
+ const running = new Set<string>();
+ for await (const id of fake.client.listRunning!({})) running.add(id);
+ assert.deepEqual(running, new Set([tracked, scratch, foreign]));
+ assert.equal(running.has(orphan), false);
+});
diff --git a/test/native-provider-clients.test.ts b/test/native-provider-clients.test.ts
index d03d7b5a9..cfe1a5ea1 100644
--- a/test/native-provider-clients.test.ts
+++ b/test/native-provider-clients.test.ts
@@ -1,11 +1,44 @@
+import { spawnSync } from "node:child_process";
+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join } from "node:path";
+import { createModalSandbox } from "../src/sandbox/modal-sandbox.ts";
+import { createLocalWorkspaceStore } from "../src/workspace/workspace-store.ts";
import { test, mock } from "node:test";
import assert from "node:assert/strict";
-import { createSdkModalClient } from "../src/sandbox/modal-client.ts";
+import {
+ createSdkModalClient,
+ modalEgressAllowlist,
+ MODAL_EXEC_GRACE_MS,
+ MODAL_MAX_EXEC_ARG_BYTES,
+} from "../src/sandbox/modal-client.ts";
import { createSdkE2bClient } from "../src/sandbox/e2b-client.ts";
const modalCalls: unknown[][] = [];
+const modalCreateParams: Record<string, unknown>[] = [];
+const modalExecs: { args: string[]; params: Record<string, unknown> }[] = [];
+const modalWrites: { path: string; bytes: number }[] = [];
+const modalRunning = new Map<string, boolean>();
+class SandboxFilesystemFileTooLargeError extends Error {
+ override name = "SandboxFilesystemFileTooLargeError";
+}
const modalSandbox = {
sandboxId: "sb-native",
+ async exec(args: string[], params: Record<string, unknown>) {
+ if (typeof params.timeoutMs === "number" && params.timeoutMs % 1000 !== 0)
+ throw new Error(`timeoutMs must be a multiple of 1000ms, got ${params.timeoutMs}`);
+ modalExecs.push({ args, params });
+ return { stdout: { readText: async () => "ok" }, stderr: { readText: async () => "" }, wait: async () => 0 };
+ },
+ filesystem: {
+ async writeBytes(data: Uint8Array, path: string) {
+ modalWrites.push({ path, bytes: data.byteLength });
+ },
+ async readBytes(path: string) {
+ if (path.endsWith("huge.bin")) throw new SandboxFilesystemFileTooLargeError("file too large");
+ return new Uint8Array([1]);
+ },
+ },
async snapshotDirectory(path: string, options: unknown) {
modalCalls.push(["snapshot", path, options]);
return { imageId: "im-native" };
@@ -17,9 +50,18 @@ const modalSandbox = {
mock.module("modal", {
namedExports: {
ModalClient: class {
- apps = { fromName: async () => ({}) };
+ apps = { fromName: async () => ({ appId: "ap-1" }) };
images = { fromRegistry: () => ({}), fromId: async (imageId: string) => ({ imageId }) };
- sandboxes = { create: async () => modalSandbox };
+ sandboxes = {
+ create: async (_app: unknown, _image: unknown, params: Record<string, unknown>) => {
+ modalCreateParams.push(params);
+ return modalSandbox;
+ },
+ list: async function* (params: { appId?: string; tags?: Record<string, string> }) {
+ modalCalls.push(["list", params]);
+ for (const [sandboxId, running] of modalRunning) yield { sandboxId, poll: async () => (running ? null : 0) };
+ },
+ };
},
},
});
@@ -66,6 +108,94 @@ test("Modal native directory snapshot has a ten-minute timeout, finite retention
assert.equal(client.lifetimeMs, 24 * 3600_000);
});
+test("Modal exec deadlines are whole seconds with a single grace margin, even near a snapshot deadline", async () => {
+ const client = createSdkModalClient({ tokenId: "id", tokenSecret: "secret", appName: "test", image: "ubuntu" });
+ const session = await client.create({});
+ modalExecs.length = 0;
+ await session.runCommand("wc -c < /root/.qm-home.tar", { timeoutMs: 12_345 });
+ await session.runCommand("true", { timeoutMs: 7 });
+ await session.runCommand("true");
+ assert.deepEqual(
+ modalExecs.map((call) => call.params.timeoutMs),
+ [13_000 + MODAL_EXEC_GRACE_MS, 1000 + MODAL_EXEC_GRACE_MS, 3600_000 + MODAL_EXEC_GRACE_MS],
+ );
+ assert.deepEqual(modalExecs[0]!.args, ["timeout", "13", "sh", "-c", "wc -c < /root/.qm-home.tar"]);
+});
+
+test("Modal passes command env through exec and spools oversized commands through the filesystem", async () => {
+ const client = createSdkModalClient({ tokenId: "id", tokenSecret: "secret", appName: "test", image: "ubuntu" });
+ const session = await client.create({});
+ modalExecs.length = 0;
+ modalWrites.length = 0;
+ await session.runCommand("echo $SECRET_TOKEN", { env: { SECRET_TOKEN: "s3cret" } });
+ assert.deepEqual(modalExecs[0]!.params.env, { SECRET_TOKEN: "s3cret" });
+ assert.ok(!modalExecs[0]!.args[2]!.includes("s3cret"));
+ await session.runCommand("true", { env: {} });
+ assert.equal("env" in modalExecs[1]!.params, false);
+ const huge = `printf '%s' '${"A".repeat(MODAL_MAX_EXEC_ARG_BYTES)}' | base64 -d`;
+ await session.runCommand(huge);
+ assert.equal(modalWrites.length, 1);
+ assert.equal(modalWrites[0]!.bytes, Buffer.byteLength(huge));
+ assert.match(modalWrites[0]!.path, /^\/tmp\/\.qm-exec-[0-9a-f-]{36}\.sh$/);
+ assert.equal(
+ modalExecs[2]!.args[2],
+ `timeout 3600 sh ${modalWrites[0]!.path}; rc=$?; rm -f ${modalWrites[0]!.path}; exit $rc`,
+ );
+ assert.ok(Buffer.byteLength(modalExecs[2]!.args[2]!) < MODAL_MAX_EXEC_ARG_BYTES);
+});
+
+test("Modal creates sandboxes with a real reservation, an idle backstop, tags and the egress allowlist", async () => {
+ modalCreateParams.length = 0;
+ const bare = createSdkModalClient({ tokenId: "id", tokenSecret: "secret", appName: "test", image: "ubuntu" });
+ await bare.create({ name: "qm-scope", tags: { "qm-kind": "scope" } });
+ assert.deepEqual(modalCreateParams[0], {
+ name: "qm-scope",
+ tags: { "qm-kind": "scope" },
+ timeoutMs: 24 * 3600_000,
+ idleTimeoutMs: 12 * 3600_000,
+ cpu: 1,
+ memoryMiB: 2048,
+ });
+ const tuned = createSdkModalClient({
+ tokenId: "id",
+ tokenSecret: "secret",
+ appName: "test",
+ image: "ubuntu",
+ cpus: 4,
+ memoryMb: 8192,
+ regions: ["us-west-2"],
+ idleTimeoutMs: 6 * 3600_000 + 1,
+ egressProxyUrl: "https://egress.example.com",
+ });
+ await tuned.create({});
+ assert.deepEqual(modalCreateParams[1], {
+ timeoutMs: 24 * 3600_000,
+ idleTimeoutMs: 6 * 3600_000 + 1000,
+ cpu: 4,
+ memoryMiB: 8192,
+ regions: ["us-west-2"],
+ outboundDomainAllowlist: ["egress.example.com"],
+ });
+ assert.deepEqual(modalEgressAllowlist("http://10.1.2.3:3128"), { outboundCidrAllowlist: ["10.1.2.3/32"] });
+ assert.deepEqual(modalEgressAllowlist("http://[2001:db8::1]:3128"), { outboundCidrAllowlist: ["2001:db8::1/128"] });
+ assert.throws(() => modalEgressAllowlist("http://egress.example.com:3128"), /https URL on port 443/);
+ assert.throws(() => modalEgressAllowlist("https://egress.example.com:8443"), /https URL on port 443/);
+});
+
+test("Modal lists running tagged sandboxes within the app and reports oversized reads clearly", async () => {
+ const client = createSdkModalClient({ tokenId: "id", tokenSecret: "secret", appName: "test", image: "ubuntu" });
+ modalRunning.set("sb-live", true);
+ modalRunning.set("sb-done", false);
+ modalCalls.length = 0;
+ const ids: string[] = [];
+ for await (const id of client.listRunning!({ "qm-kind": "scope" })) ids.push(id);
+ assert.deepEqual(ids, ["sb-live"]);
+ assert.deepEqual(modalCalls[0], ["list", { appId: "ap-1", tags: { "qm-kind": "scope" } }]);
+ const session = await client.create({});
+ assert.deepEqual(await session.readFileBytes("/root/small.bin"), new Uint8Array([1]));
+ await assert.rejects(session.readFileBytes("/root/huge.bin"), /exceeds Modal's filesystem read limit/);
+});
+
test("Modal rejects invalid checkpoint retention", () => {
for (const snapshotRetentionMs of [0, -1, Infinity, NaN]) {
assert.throws(
@@ -95,3 +225,60 @@ test("E2B sends explicit pause lifecycle, reads state without connect, and surfa
await client.create({ metadata: {}, autoPause: false });
assert.deepEqual((e2bCalls.at(-1)![2] as { lifecycle: unknown }).lifecycle, { onTimeout: "kill", autoResume: false });
});
+
+test("Modal executes a one-megabyte payload through the sandbox and SDK without nested oversized arguments", async (t) => {
+ const root = mkdtempSync(join(tmpdir(), "modal-large-command-"));
+ const home = join(root, "home");
+ mkdirSync(home);
+ const map = (value: string) =>
+ value
+ .replaceAll("/root", home)
+ .replaceAll("/tmp/.qm-exec-", `${root}/.qm-exec-`)
+ .replaceAll("exec setsid ", process.platform === "darwin" ? "exec " : "exec setsid ");
+ const originalExec = modalSandbox.exec;
+ const originalWrite = modalSandbox.filesystem.writeBytes;
+ t.after(() => {
+ modalSandbox.exec = originalExec;
+ modalSandbox.filesystem.writeBytes = originalWrite;
+ rmSync(root, { recursive: true, force: true });
+ });
+ modalSandbox.filesystem.writeBytes = async (data, path) => {
+ const file = map(path);
+ mkdirSync(dirname(file), { recursive: true });
+ writeFileSync(file, map(Buffer.from(data).toString("utf8")));
+ };
+ modalSandbox.exec = async (args, params) => {
+ const mapped = args.map(map);
+ const result = spawnSync(mapped[0]!, mapped.slice(1), {
+ encoding: "utf8",
+ env: { ...process.env, HOME: home, ...(params.env as Record<string, string> | undefined) },
+ timeout: 30_000,
+ });
+ if (result.error) throw result.error;
+ return {
+ stdout: { readText: async () => result.stdout },
+ stderr: { readText: async () => result.stderr },
+ wait: async () => result.status ?? -1,
+ };
+ };
+ const client = createSdkModalClient({ tokenId: "id", tokenSecret: "secret", appName: "test", image: "ubuntu" });
+ const session = await client.create({});
+ const sandbox = createModalSandbox(createLocalWorkspaceStore(join(root, "workspace")), {
+ client: {
+ create: async () => session,
+ fromId: async () => session,
+ fromName: async () => null,
+ terminate: async () => {},
+ },
+ });
+ const handle = await sandbox.provision([{ scopeId: "large-command", mountPath: "", mode: "rw" }]);
+ const size = 1024 * 1024;
+ const result = await sandbox.run(handle, `payload='${"x".repeat(size)}'; printf '%s' "${"${#payload}"}"`);
+ assert.equal(result.code, 0, result.stderr);
+ assert.equal(result.stdout, String(size));
+ const signalled = await sandbox.run(handle, `payload='${"x".repeat(size)}'; printf '%s' "${"${#payload}"}"`, {
+ signal: new AbortController().signal,
+ });
+ assert.equal(signalled.code, 0, signalled.stderr);
+ assert.equal(signalled.stdout, String(size));
+});
diff --git a/test/support/fake-modal.ts b/test/support/fake-modal.ts
index 96855fa5d..768740bf4 100644
--- a/test/support/fake-modal.ts
+++ b/test/support/fake-modal.ts
@@ -13,6 +13,7 @@ import {
interface FakeRecord {
sandboxId: string;
name?: string;
+ tags: Record<string, string>;
state: "running" | "terminated";
home: string;
createdAt: number;
@@ -29,6 +30,8 @@ export interface FakeModal {
terminateAllRunning(): void;
failTerminateOnce(): void;
execScripts(): string[];
+ tagsOf(sandboxId: string): Record<string, string>;
+ createUntracked(tags: Record<string, string>): string;
cleanup(): void;
}
@@ -82,14 +85,14 @@ export function installFakeModal(opts: { native?: boolean } = {}): FakeModal {
},
}
: {}),
- async runCommand(command): Promise<ModalCommandResult> {
+ async runCommand(command, runOpts): Promise<ModalCommandResult> {
alive(r);
execScripts.push(command);
mkdirSync(join(r.home, "tmp"), { recursive: true });
const spawned = spawnSync("sh", ["-c", remap(r, command)], {
encoding: "buffer",
maxBuffer: 128 * 1024 * 1024,
- env: { ...process.env, COPYFILE_DISABLE: "1" },
+ env: { ...process.env, ...runOpts?.env, COPYFILE_DISABLE: "1" },
});
return {
stdout: (spawned.stdout ?? Buffer.alloc(0)).toString("utf8"),
@@ -118,6 +121,21 @@ export function installFakeModal(opts: { native?: boolean } = {}): FakeModal {
},
});
+ const spawn = (name: string | undefined, tags: Record<string, string>): FakeRecord => {
+ const id = `sb-${nextId++}`;
+ const r: FakeRecord = {
+ sandboxId: id,
+ ...(name ? { name } : {}),
+ tags,
+ state: "running",
+ home: join(root, id),
+ createdAt: ++clock,
+ };
+ mkdirSync(r.home, { recursive: true });
+ records.set(id, r);
+ return r;
+ };
+
const client: ModalClient = {
nativeSnapshots: opts.native ?? false,
async create(opts): Promise<ModalSession> {
@@ -127,17 +145,7 @@ export function installFakeModal(opts: { native?: boolean } = {}): FakeModal {
throw new ModalNameConflictError(opts.name, "sandbox with this name already exists");
}
}
- const id = `sb-${nextId++}`;
- const r: FakeRecord = {
- sandboxId: id,
- ...(opts.name ? { name: opts.name } : {}),
- state: "running",
- home: join(root, id),
- createdAt: ++clock,
- };
- mkdirSync(r.home, { recursive: true });
- records.set(id, r);
- return session(r);
+ return session(spawn(opts.name, opts.tags ?? {}));
},
async fromId(sandboxId): Promise<ModalSession> {
const r = records.get(sandboxId);
@@ -154,6 +162,12 @@ export function installFakeModal(opts: { native?: boolean } = {}): FakeModal {
if (!r) return;
kill(r);
},
+ async *listRunning(tags): AsyncIterable<string> {
+ for (const r of records.values()) {
+ if (r.state !== "running") continue;
+ if (Object.entries(tags).every(([k, v]) => r.tags[k] === v)) yield r.sandboxId;
+ }
+ },
};
return {
@@ -181,6 +195,12 @@ export function installFakeModal(opts: { native?: boolean } = {}): FakeModal {
terminateFailures++;
},
execScripts: () => [...execScripts],
+ tagsOf: (sandboxId) => {
+ const r = records.get(sandboxId);
+ if (!r) throw new Error(`fake-modal: no sandbox ${sandboxId}`);
+ return { ...r.tags };
+ },
+ createUntracked: (tags) => spawn(undefined, tags).sandboxId,
cleanup: () => rmSync(root, { recursive: true, force: true }),
};
}