Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(worker): allow a worker to override SANDBOX_MEMORY_LIMIT locally (#16165)

Amr Elmohamady60ae78baa98b
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index d252a3a436b..29aff6e2275 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -43,6 +43,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the
- **A code step's error text must be built from a structured payload, never from a stringified error — the three sandboxes disagree the moment anyone stringifies early.** `AP_EXECUTION_MODE` + the step's `useDeno` pick one of three runners, and each used to hand the host a *different* rendering of the same `throw`: the node child sent `util.inspect(e)`, the deno child sent `error.stack`, and isolated-vm marshals only `message`/`stack`. `formatPieceError`'s `stripStack` then removes `\n at …` lines but leaves the `Error: ` prefix and, for inspect, a mangled own-props tail — so `{{step['error'].message}}` read `"Error: Card declined"` on the default `UNSANDBOXED` mode and `"Card declined"` on `SANDBOX_CODE_ONLY`, and `errorName` was populated only on the isolate (GIT-1898, Pylon 6022). Cloud and self-hosted could differ on the same flow. Fixed at the serializing boundary, not in the parser: both child runners now emit `{ message, name, stack }` (`sandboxError.payloadSource`, injected into both generated programs so the two copies cannot drift) and the host rebuilds a real `Error` via `sandboxError.build`. Do **not** "fix" a future variant of this in `formatPieceError` — it also serves piece and HTTP errors, where a provider legitimately returns text starting `Error: `, and widening `stripStack` puts a lossy heuristic on every error source in the product. Two corollaries worth keeping: captured stdout/stderr goes on the `stack` (→ the UI's *Technical Details*), never on `message`, because which sandbox captures output is itself mode-specific; and a non-`Error` throw still diverges by design — isolated-vm refuses to marshal it at all. `code-sandbox-error-contract.test.ts` drives all three real sandboxes through `formatPieceError` and is what stops a fourth shape appearing.
- **A piece that trips the 5 MB size fallback ships as a "bundle" that still installs its full npm tree, and it dominates engine memory.** `bundlePiece` inlines everything by default, but if the result exceeds `FAIL_BYTES` (5 MB) it rebuilds with `inlineAll: false`, externalizing **all** third-party deps (`bundle-piece-utils.ts`) — deliberate, so every piece keeps building and the tarball stays small. The runtime bill lands elsewhere: `google-sheets` has done this since 0.8.4 (10 externals, growing to 17 by 0.16.1), so its bundle declares `[email protected]`, which bun installs into the shared workspace store and the engine loads — **23.52 MB of a 28.12 MB `require.cache`, 83%, for one piece**, while fully-inlined `[email protected]` contributes 1.58 MB despite being the larger piece. Two `googleapis-common` majors (7.0.1 and 7.2.0) sit resident together, which is exactly the duplication bundling was meant to end. Sampling 60 CDN bundles: 60/60 carry `bundle.cjs`, 55/60 declare **zero** dependencies, and the 5 that don't externalize only native/wasm-ish packages (`pino`, `jimp`, `pdf-lib`, `undici`, `bufferutil`, `tiktoken`). So "bundled" is not a uniform guarantee — check `package.json.dependencies` inside the tarball before assuming a piece is self-contained.
- **Self-contained piece bundles remove cross-piece dedup, so the per-run spike now scales with how many *fat* pieces one flow touches.** A CDN bundle inlines everything: its `package.json` declares **no dependencies** and every `require()` in `src/bundle.cjs` is a Node built-in. Good for install size and it is what finally kills the multi-copy `@activepieces/shared` problem — but two pieces that both use `axios`/`googleapis` no longer share one hoisted copy, they each carry their own. Measured on staging Aug 2026 (`AP_USE_CDN_FOR_BUNDLES=true`, isolate mode): 9 fat bundled pieces in one flow = ~15 MB of uncompressed JS, engine at **926 MB RSS / 788 MB heapUsed** against a V8 `heap_size_limit` of **1216 MB** (`--max-old-space-size=1024` from the default `SANDBOX_MEMORY_LIMIT`) inside a **1 GiB** container already holding a 247 MB worker + 74 MB pm2 → `MEMORY_LIMIT_EXCEEDED`, reproducibly, in ~8 s. The identical flow succeeds with the container raised to 3 GiB. This is the deterministic reproducer for the standing over-commit above: a per-run spike, not accumulation. Sizing rule: budget the engine ceiling against *container minus worker minus pm2*, and treat "many fat pieces in one flow" as the spike driver now that bundles don't share.
+- **Raising `NODE_OPTIONS` never helps an engine OOM, and those OOMs are invisible in the queue.** The sandbox builds the engine's env from scratch (`buildSandboxEnv`), so `NODE_OPTIONS` caps only the worker. The engine's cap is `SANDBOX_MEMORY_LIMIT`: global from the API, or set per worker with a local `AP_SANDBOX_MEMORY_LIMIT` (KB, validated at startup; PR #16165). An OOM run finishes as flow-run status `MEMORY_LIMIT_EXCEEDED`, not a failed BullMQ job, so look for it in `flow_run` grouped by platform, and confirm cgroup-vs-V8 with `dmesg` (`CONSTRAINT_MEMCG`) on the host (SRE-236, SRE-237).
- **The cost of a resident `@activepieces/shared` copy is zod schema construction, not data — ~40 MB per copy, and reused engines collect one per piece version.** Heap-snapshotted a dedicated-worker engine at 596 MB RSS / 402 MB heap (0.88.1 beta, pre-CDN-bundles, `AP_REUSE_SANDBOX`), Aug 2026: after a double forced GC, 393 MB survived, and the histogram was **2.05 M anonymous closures (109 MB) + 143 MB of `(object properties)` arrays + 359 k `system/Context` scopes (18 MB)** — instantiated module graphs, not retained run data (only 16 k distinct functions back those 2 M closures). Every sampled retainer path ended `require.cache → @activepieces+shared@<version>/…/<dto>.js → exports → <ZodSchema>._def → get shape → refine/pipe/optional/brand closures`: `shared` eagerly builds ~500 top-level zod DTO trees at import, and zod v4 attaches per-instance accessor/method closures (120 k `get`, 67 k `set`, 31 k `validate` in that one heap). The engine held **9 shared versions at once** (0.37 → 0.96.2) because each piece bundle pins its own, and reuse + `import()` pins them forever — including **3 versions of `piece-hubspot` and 2 of `piece-slack` simultaneously**, one per flow-pinned piece version, so republish/upgrade churn multiplies copies of the *same* piece. Budget ≈ 60–70 MB engine baseline + ~40 MB per distinct resident shared copy. Two independent fixes attack it: CDN self-contained bundles (no shared inside pieces at all, see above) and require()-based piece LRU eviction; snapshot mechanics for redoing this measurement are in the `profile-worker-memory` skill. Follow-up exact measurement (same process at 542 MB heap, 9 shared copies, graph-cut retained sizes): one cleanly-severable copy weighs **41.3 MB / 445 k nodes**, but the other 8 each show <1 MB *exclusive* retained because copies are **co-retained as clusters, and the pin is the ESM module map, not `require.cache`**: deleting every `require.cache` entry under both `.bun` store roots frees only ~65 MB of 553 MB, while blocking the ESM `ModuleWrap`/`SyntheticModule` entries too frees **406 MB** (engine-only floor: 147 MB, which includes main.js's own bundled shared). Retainer chain: `Global handles → SyntheticModule (piece import()) → Piece object → action run closures → context:pieces_framework_1 / shared_1 (whole exports)` — every action closure captures its module scope, so a piece and its framework+shared copies live and die together. Consequence: any eviction scheme must make the *piece entry itself* collectable (hence require()-based loading in the LRU fix — an import()ed entry can never be dropped); purging shared's cache entries alone reclaims ~nothing.
- **~1.3 GB is the idle floor before a single flow runs.** Same box, freshly booted, zero runs: container 1.325 GiB, of which the API process alone is ~1.18 GB and the worker ~213 MB. Sizing a container off "what a flow needs" is wrong by more than a gigabyte — budget the floor first, then add the sandbox ceiling above.
- **The require()-based LRU in `piece-loader.ts` (PR #14859) makes both CPU and memory *worse* than not evicting, in both the CDN-bundle and raw-npm worlds, and was removed.** The load-path switch from `await import(piecePath)` to `require(piecePath)` doesn't itself reclaim any memory — the resulting loader still holds every loaded piece in `require.cache` until the process exits — but it is the precondition for any eviction mechanism at all, since ESM `ModuleWrap` handles are pinned to the realm regardless of `require.cache` state. The LRU cap-and-evict wrapper Chaker put on top of it was where the memory reclaim was supposed to come from, and it doesn't work in practice. Directly A/B tested `require()` vs `import()` with no eviction on 300-iter rotations: 296 vs 331 MB RSS on CDN, 1963 vs 1969 MB on pre-CDN — the two mechanisms plateau within 6-35 MB of each other, neither reclaims. Two facts made it look like it helped: **(a)** Chaker's PR compared "require() + LRU cap=10" against `await import()` (the old unevictable path), so the entire measured memory win came from the `import → require` switch, not from the LRU; **(b)** his fixture loaded 20 fabricated pieces uniquely (each loaded once, ~10 evictions total), which never exercises the rotate-and-reload churn that real fleets hit. The shipped constant was also `MAX_LOADED_PIECES = 5`, not the 10 in the PR title — a same-day follow-up commit (`a74fc81d58`, empty body) dropped it to 5 after the anchor-pinning bug (`bfce1133b7` — `createRequire(__filename)` anchor held all loaded pieces via its `children` array, engine OOM at 1.1 GB despite the cap) got fixed and left OOM headroom concerns; both landed inside the squash-merge (`71bd79c74d`), so the merged PR reads as if cap=10 shipped. Measured Sep 2026 with a standalone Node harness that inlines the exact eviction algorithm (`scratchpad/lru-bench/`), 300 iterations rotating 15 real published bundles:
diff --git a/packages/server/worker/src/lib/config/configs.ts b/packages/server/worker/src/lib/config/configs.ts
index a87591d160b..f12b68e9597 100644
--- a/packages/server/worker/src/lib/config/configs.ts
+++ b/packages/server/worker/src/lib/config/configs.ts
@@ -50,6 +50,7 @@ export enum WorkerSystemProp {
PROJECT_WORKER = 'AP_PROJECT_WORKER',
WORKER_CONCURRENCY = 'AP_WORKER_CONCURRENCY',
EXECUTION_MODE = 'AP_EXECUTION_MODE',
+ SANDBOX_MEMORY_LIMIT = 'AP_SANDBOX_MEMORY_LIMIT',
REUSE_SANDBOX = 'AP_REUSE_SANDBOX',
CACHE_BASE_PATH = 'AP_CACHE_BASE_PATH',
PREWARM_CACHE_ON_STARTUP = 'AP_PREWARM_CACHE_ON_STARTUP',
diff --git a/packages/server/worker/src/lib/worker.ts b/packages/server/worker/src/lib/worker.ts
index 8f226747bec..417d88ec008 100644
--- a/packages/server/worker/src/lib/worker.ts
+++ b/packages/server/worker/src/lib/worker.ts
@@ -419,6 +419,13 @@ async function fetchAndStoreSettings(sock: Socket): Promise<void> {
if (!isNil(localExecutionMode)) {
response.EXECUTION_MODE = localExecutionMode
}
+ const localSandboxMemoryLimit = system.get(WorkerSystemProp.SANDBOX_MEMORY_LIMIT)
+ if (!isNil(localSandboxMemoryLimit)) {
+ if (!/^[1-9]\d*$/.test(localSandboxMemoryLimit)) {
+ throw new Error(`AP_SANDBOX_MEMORY_LIMIT must be a positive integer in KB. Got: ${localSandboxMemoryLimit}`)
+ }
+ response.SANDBOX_MEMORY_LIMIT = localSandboxMemoryLimit
+ }
const workerGroupId = system.get(WorkerSystemProp.WORKER_GROUP_ID)
if (!isNil(workerGroupId)) {
if (response.EDITION === ApEdition.CLOUD) {
@@ -433,7 +440,7 @@ async function fetchAndStoreSettings(sock: Socket): Promise<void> {
}
}
workerSettings.set(response)
- logger.info({ environment: response.ENVIRONMENT, executionMode: response.EXECUTION_MODE }, 'Worker settings loaded')
+ logger.info({ environment: response.ENVIRONMENT, executionMode: response.EXECUTION_MODE, sandboxMemoryLimitKb: response.SANDBOX_MEMORY_LIMIT }, 'Worker settings loaded')
resolve()
})
})
diff --git a/packages/server/worker/test/lib/worker-settings-override.test.ts b/packages/server/worker/test/lib/worker-settings-override.test.ts
index c152aed32d5..0508f7fef15 100644
--- a/packages/server/worker/test/lib/worker-settings-override.test.ts
+++ b/packages/server/worker/test/lib/worker-settings-override.test.ts
@@ -117,6 +117,7 @@ describe('worker settings override', () => {
delete process.env.AP_EXECUTION_MODE
delete process.env.AP_WORKER_GROUP_ID
delete process.env.AP_REUSE_SANDBOX
+ delete process.env.AP_SANDBOX_MEMORY_LIMIT
await new Promise<void>((resolve) => {
ioServer.close(() => resolve())
})
@@ -191,6 +192,32 @@ describe('worker settings override', () => {
expect(stored.EXECUTION_MODE).toBe(ExecutionMode.SANDBOX_CODE_ONLY)
}, 10_000)
+ it('local AP_SANDBOX_MEMORY_LIMIT overrides server-provided limit', async () => {
+ process.env.AP_SANDBOX_MEMORY_LIMIT = '2097152'
+ const serverSettings = buildWorkerSettingsResponse({ SANDBOX_MEMORY_LIMIT: '1048576' })
+ await connectAndWaitForSettings(serverSettings)
+
+ const stored = mockWorkerSettingsSet.mock.calls[0][0] as WorkerSettingsResponse
+ expect(stored.SANDBOX_MEMORY_LIMIT).toBe('2097152')
+ }, 10_000)
+
+ it('no local AP_SANDBOX_MEMORY_LIMIT keeps server-provided limit', async () => {
+ const serverSettings = buildWorkerSettingsResponse({ SANDBOX_MEMORY_LIMIT: '524288' })
+ await connectAndWaitForSettings(serverSettings)
+
+ const stored = mockWorkerSettingsSet.mock.calls[0][0] as WorkerSettingsResponse
+ expect(stored.SANDBOX_MEMORY_LIMIT).toBe('524288')
+ }, 10_000)
+
+ it.each(['invalid', '2097152KB', '0', '-1'])('invalid local AP_SANDBOX_MEMORY_LIMIT %s throws', async (value) => {
+ process.env.AP_SANDBOX_MEMORY_LIMIT = value
+ const serverSettings = buildWorkerSettingsResponse()
+
+ const err = await connectAndExpectCrash(serverSettings)
+ expect(err.message).toMatch(/AP_SANDBOX_MEMORY_LIMIT must be a positive integer/)
+ expect(mockWorkerSettingsSet).not.toHaveBeenCalled()
+ }, 10_000)
+
it('worker group + SANDBOX_PROCESS passes validation', async () => {
process.env.AP_WORKER_GROUP_ID = 'group-1'
process.env.AP_EXECUTION_MODE = ExecutionMode.SANDBOX_PROCESS