Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(engine): remove per-sandbox piece LRU (#15786)

Amr Elmohamady6b86a38cde23
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index ab6b93ba862..9f911b8a737 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -44,6 +44,18 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the
- **Self-contained piece bundles remove cross-piece dedup, so the per-run spike now scales with how many *fat* pieces one flow touches.** A CDN bundle inlines everything: its `package.json` declares **no dependencies** and every `require()` in `src/bundle.cjs` is a Node built-in. Good for install size and it is what finally kills the multi-copy `@activepieces/shared` problem — but two pieces that both use `axios`/`googleapis` no longer share one hoisted copy, they each carry their own. Measured on staging Aug 2026 (`AP_USE_CDN_FOR_BUNDLES=true`, isolate mode): 9 fat bundled pieces in one flow = ~15 MB of uncompressed JS, engine at **926 MB RSS / 788 MB heapUsed** against a V8 `heap_size_limit` of **1216 MB** (`--max-old-space-size=1024` from the default `SANDBOX_MEMORY_LIMIT`) inside a **1 GiB** container already holding a 247 MB worker + 74 MB pm2 → `MEMORY_LIMIT_EXCEEDED`, reproducibly, in ~8 s. The identical flow succeeds with the container raised to 3 GiB. This is the deterministic reproducer for the standing over-commit above: a per-run spike, not accumulation. Sizing rule: budget the engine ceiling against *container minus worker minus pm2*, and treat "many fat pieces in one flow" as the spike driver now that bundles don't share.
- **The cost of a resident `@activepieces/shared` copy is zod schema construction, not data — ~40 MB per copy, and reused engines collect one per piece version.** Heap-snapshotted a dedicated-worker engine at 596 MB RSS / 402 MB heap (0.88.1 beta, pre-CDN-bundles, `AP_REUSE_SANDBOX`), Aug 2026: after a double forced GC, 393 MB survived, and the histogram was **2.05 M anonymous closures (109 MB) + 143 MB of `(object properties)` arrays + 359 k `system/Context` scopes (18 MB)** — instantiated module graphs, not retained run data (only 16 k distinct functions back those 2 M closures). Every sampled retainer path ended `require.cache → @activepieces+shared@<version>/…/<dto>.js → exports → <ZodSchema>._def → get shape → refine/pipe/optional/brand closures`: `shared` eagerly builds ~500 top-level zod DTO trees at import, and zod v4 attaches per-instance accessor/method closures (120 k `get`, 67 k `set`, 31 k `validate` in that one heap). The engine held **9 shared versions at once** (0.37 → 0.96.2) because each piece bundle pins its own, and reuse + `import()` pins them forever — including **3 versions of `piece-hubspot` and 2 of `piece-slack` simultaneously**, one per flow-pinned piece version, so republish/upgrade churn multiplies copies of the *same* piece. Budget ≈ 60–70 MB engine baseline + ~40 MB per distinct resident shared copy. Two independent fixes attack it: CDN self-contained bundles (no shared inside pieces at all, see above) and require()-based piece LRU eviction; snapshot mechanics for redoing this measurement are in the `profile-worker-memory` skill. Follow-up exact measurement (same process at 542 MB heap, 9 shared copies, graph-cut retained sizes): one cleanly-severable copy weighs **41.3 MB / 445 k nodes**, but the other 8 each show <1 MB *exclusive* retained because copies are **co-retained as clusters, and the pin is the ESM module map, not `require.cache`**: deleting every `require.cache` entry under both `.bun` store roots frees only ~65 MB of 553 MB, while blocking the ESM `ModuleWrap`/`SyntheticModule` entries too frees **406 MB** (engine-only floor: 147 MB, which includes main.js's own bundled shared). Retainer chain: `Global handles → SyntheticModule (piece import()) → Piece object → action run closures → context:pieces_framework_1 / shared_1 (whole exports)` — every action closure captures its module scope, so a piece and its framework+shared copies live and die together. Consequence: any eviction scheme must make the *piece entry itself* collectable (hence require()-based loading in the LRU fix — an import()ed entry can never be dropped); purging shared's cache entries alone reclaims ~nothing.
- **~1.3 GB is the idle floor before a single flow runs.** Same box, freshly booted, zero runs: container 1.325 GiB, of which the API process alone is ~1.18 GB and the worker ~213 MB. Sizing a container off "what a flow needs" is wrong by more than a gigabyte — budget the floor first, then add the sandbox ceiling above.
+- **The require()-based LRU in `piece-loader.ts` (PR #14859) makes both CPU and memory *worse* than not evicting, in both the CDN-bundle and raw-npm worlds, and was removed.** The load-path switch from `await import(piecePath)` to `require(piecePath)` doesn't itself reclaim any memory — the resulting loader still holds every loaded piece in `require.cache` until the process exits — but it is the precondition for any eviction mechanism at all, since ESM `ModuleWrap` handles are pinned to the realm regardless of `require.cache` state. The LRU cap-and-evict wrapper Chaker put on top of it was where the memory reclaim was supposed to come from, and it doesn't work in practice. Directly A/B tested `require()` vs `import()` with no eviction on 300-iter rotations: 296 vs 331 MB RSS on CDN, 1963 vs 1969 MB on pre-CDN — the two mechanisms plateau within 6-35 MB of each other, neither reclaims. Two facts made it look like it helped: **(a)** Chaker's PR compared "require() + LRU cap=10" against `await import()` (the old unevictable path), so the entire measured memory win came from the `import → require` switch, not from the LRU; **(b)** his fixture loaded 20 fabricated pieces uniquely (each loaded once, ~10 evictions total), which never exercises the rotate-and-reload churn that real fleets hit. The shipped constant was also `MAX_LOADED_PIECES = 5`, not the 10 in the PR title — a same-day follow-up commit (`a74fc81d58`, empty body) dropped it to 5 after the anchor-pinning bug (`bfce1133b7` — `createRequire(__filename)` anchor held all loaded pieces via its `children` array, engine OOM at 1.1 GB despite the cap) got fixed and left OOM headroom concerns; both landed inside the squash-merge (`71bd79c74d`), so the merged PR reads as if cap=10 shipped. Measured Sep 2026 with a standalone Node harness that inlines the exact eviction algorithm (`scratchpad/lru-bench/`), 300 iterations rotating 15 real published bundles:
+
+ | scenario | cap | iters | evictions | CPU/load mean | CPU/load p95 | RSS peak | RSS plateau (post-GC) | evict share of loader CPU |
+ |---|---:|---:|---:|---:|---:|---:|---:|---:|
+ | CDN self-contained | 5 (shipped) | 3000 | 2,995 | 63 ms | 80 ms | 315 MB | 313 MB | ~68% |
+ | CDN self-contained | 10 | 3000 | 2,990 | 77 ms | 89 ms | 330 MB | 326 MB | ~68% |
+ | CDN self-contained | 100 (no evict) | 3000 | 0 | 0.3 ms | 0 ms | 303 MB | 289 MB (plateau iter 20) | 0% |
+ | Pre-CDN raw npm | 5 | 300 | 295 | **2131 ms** | 4552 ms | 2209 MB | 2149 MB | 73% |
+ | Pre-CDN raw npm | 10 | 300 | 290 | **2513 ms** | 3075 ms | 2659 MB | 2659 MB | 77% |
+ | Pre-CDN raw npm | 100 (no evict) | 300 | 0 | 35 ms | 47 ms | 1978 MB | 1978 MB (plateau iter 20) | 0% |
+
+ Three overturning findings. **(1)** No-eviction wins on both axes in both worlds — CDN saves 21 MB and 20-25× CPU/load; pre-CDN saves 171-681 MB (yes, less RSS *without* the cap) and 60× CPU/load. The eviction-and-reload churn allocates faster than the doomed-subtree GC + forced `global.gc()` can reclaim, and cap=10 is worse than cap=5 because a bigger survivor require-graph costs more per `collectReachableModules` walk. **(2)** The eviction path is 60-77% of total loader CPU whenever it fires. **(3)** Per-eviction cost is dominated by `global.gc()` on the just-freed piece's object graph (V8 would collect it on its own schedule without the eager stall) — per-piece variance is 4.5× on real pieces (`date-helper` 8.8 ms → `gmail` 40 ms in the CDN world; pre-CDN pieces run 2–4 s each because `collectReachableModules` walks ~1,425 modules per piece instead of 1). Fit shape: **reused sandbox** (either `AP_REUSE_SANDBOX=true`, or `SANDBOX_CODE_ONLY`/`UNSANDBOXED` which default to reuse unless `AP_REUSE_SANDBOX=false` explicitly overrides — see `canReuseSandbox` in `sandbox-manager.ts`) + a fleet whose sandbox lifetime cycles through more pieces than the cap — Alan's 0.91.2 config with 140+ distinct pieces (pieces-report CSV, Sep 2026, CPU-per-worker sharply up vs 0.86), and Funding Societies before them on 0.88.3 (same complaint, silenced by reverting #14889 not by #14859). Historical note: the same shape has been tried and reverted repeatedly — `1c6efa0389` (fresh child per piece, reverted as #15209), `9a98fd05e9` (killed-per-operation piece-host), `3f6e32d885` (size-based cache eviction that kept `import()`, ineffective). The clean shape is the `import → require` switch alone; the LRU wrapper on top adds tax without a matching benefit and is what shipped removed in `fix/piece-loader-remove-lru`.
- **A job-data schema is a transport contract, not a validator — a constraint the piece never had turns a working saved flow into an enqueue-time failure.** Job payloads that carry piece input are shaped by props that were only ever checked in the builder, and often not even there: `Property.Number` accepts `0` and `-1`, a `DynamicProperties` block can hold a value from an older props shape, and `Property.Array` yields whatever was typed. Writing the natural-looking `z.number().int().positive()` or a bare `z.enum([...])` in the job data rejects those payloads at `jobQueue.add` — before the work is attempted, with a validation message no user can act on, on a flow that ran yesterday. Mirror the props' looseness (`z.number()`), and where a value must be one of a set, `z.enum([...]).optional().catch(undefined)` degrades an unknown value to unset instead of failing the job. Validate in the piece if anywhere; the queue is the wrong place to discover bad input.
- **Adding any `WorkerJobType` means five edits, and the compiler only catches three.** In `core/execution/.../job-data.ts`: the enum, the `getDefaultJobPriority` switch (exhaustive with no `default`, so this one is a hard compile error), and the `JobData` union. Then `OneTimeJobAddParams` in `job-queue.ts`, or `jobQueue.add` rejects the payload. Then the lazy handler entry in `worker/.../execute/job-registry.ts` — **that last one is the silent miss**: everything compiles and the job enqueues, it just never runs. Also give the job data a `projectId`, `null` if it has none, because `job-broker.ts` reads `migratedData.projectId as string` for every dequeued job and a member without the property breaks that file and `job-data-migrations.ts` rather than your own. `ExecuteAgentRunJobData` is the shape to copy: nullable `projectId`, and it already flows through that cast.
diff --git a/packages/server/engine/src/lib/helper/piece-loader.ts b/packages/server/engine/src/lib/helper/piece-loader.ts
index 8e8211c2a0d..b5e01946a59 100644
--- a/packages/server/engine/src/lib/helper/piece-loader.ts
+++ b/packages/server/engine/src/lib/helper/piece-loader.ts
@@ -17,7 +17,7 @@ export const pieceLoader = {
devPieces,
})
const piecePath = await pieceLoader.getPiecePath({ packageName, devPieces })
- const module = loadAndCapPieces(piecePath)
+ const module = createRequire(__filename)(piecePath)
const piece = extractPieceFromModule<Piece>({
module,
@@ -131,65 +131,6 @@ export const pieceLoader = {
},
}
-const MAX_LOADED_PIECES = 5
-const engineRequire = createRequire(__filename)
-const loadedPieceEntries = new Map<string, true>()
-const baselineModuleIds = new Set(Object.keys(engineRequire.cache))
-
-function loadAndCapPieces(piecePath: string): Record<string, unknown> {
- const disposableRequire = createRequire(__filename)
- const resolvedEntry = disposableRequire.resolve(piecePath)
- const loadedModule: Record<string, unknown> = disposableRequire(resolvedEntry)
- loadedPieceEntries.delete(resolvedEntry)
- loadedPieceEntries.set(resolvedEntry, true)
- if (loadedPieceEntries.size > MAX_LOADED_PIECES) {
- const leastRecentEntry = loadedPieceEntries.keys().next().value
- if (!isNil(leastRecentEntry)) {
- loadedPieceEntries.delete(leastRecentEntry)
- evictPieceSubtree(leastRecentEntry)
- global.gc?.()
- }
- }
- return loadedModule
-}
-
-function evictPieceSubtree(evictedEntry: string): void {
- const survivorReachable = collectReachableModules([...loadedPieceEntries.keys()])
- const evictedReachable = collectReachableModules([evictedEntry])
- const doomed = new Set([...evictedReachable].filter((id) => !survivorReachable.has(id) && !baselineModuleIds.has(id)))
- if (doomed.size === 0) {
- return
- }
- for (const id of Object.keys(engineRequire.cache)) {
- const mod = engineRequire.cache[id]
- if (!isNil(mod) && !doomed.has(id)) {
- mod.children = mod.children.filter((child) => !doomed.has(child.id))
- }
- }
- if (!isNil(engineRequire.main)) {
- engineRequire.main.children = engineRequire.main.children.filter((child) => !doomed.has(child.id))
- }
- for (const id of doomed) {
- Reflect.deleteProperty(engineRequire.cache, id)
- }
-}
-
-function collectReachableModules(rootIds: string[]): Set<string> {
- const reachable = new Set<string>()
- const stack = rootIds.filter((id) => !isNil(engineRequire.cache[id]))
- while (stack.length > 0) {
- const id = stack.pop()
- if (isNil(id) || reachable.has(id)) {
- continue
- }
- reachable.add(id)
- for (const child of engineRequire.cache[id]?.children ?? []) {
- stack.push(child.id)
- }
- }
- return reachable
-}
-
async function findInDistFolder(packageName: string): Promise<string | null> {
const sourcePiecesPath = path.resolve('packages/pieces')
if (!await utils.folderExists(sourcePiecesPath)) {