AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(engine): remove per-sandbox piece LRU (#15786)
Amr Elmohamady6b86a38cde23
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.mdindex ab6b93ba862..9f911b8a737 100644--- a/brain/knowledge/execution-runtime/workers.md+++ b/brain/knowledge/execution-runtime/workers.md@@ -44,6 +44,18 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the- **Self-contained piece bundles remove cross-piece dedup, so the per-run spike now scales with how many *fat* pieces one flow touches.** A CDN bundle inlines everything: its `package.json` declares **no dependencies** and every `require()` in `src/bundle.cjs` is a Node built-in. Good for install size and it is what finally kills the multi-copy `@activepieces/shared` problem — but two pieces that both use `axios`/`googleapis` no longer share one hoisted copy, they each carry their own. Measured on staging Aug 2026 (`AP_USE_CDN_FOR_BUNDLES=true`, isolate mode): 9 fat bundled pieces in one flow = ~15 MB of uncompressed JS, engine at **926 MB RSS / 788 MB heapUsed** against a V8 `heap_size_limit` of **1216 MB** (`--max-old-space-size=1024` from the default `SANDBOX_MEMORY_LIMIT`) inside a **1 GiB** container already holding a 247 MB worker + 74 MB pm2 → `MEMORY_LIMIT_EXCEEDED`, reproducibly, in ~8 s. The identical flow succeeds with the container raised to 3 GiB. This is the deterministic reproducer for the standing over-commit above: a per-run spike, not accumulation. Sizing rule: budget the engine ceiling against *container minus worker minus pm2*, and treat "many fat pieces in one flow" as the spike driver now that bundles don't share.- **The cost of a resident `@activepieces/shared` copy is zod schema construction, not data — ~40 MB per copy, and reused engines collect one per piece version.** Heap-snapshotted a dedicated-worker engine at 596 MB RSS / 402 MB heap (0.88.1 beta, pre-CDN-bundles, `AP_REUSE_SANDBOX`), Aug 2026: after a double forced GC, 393 MB survived, and the histogram was **2.05 M anonymous closures (109 MB) + 143 MB of `(object properties)` arrays + 359 k `system/Context` scopes (18 MB)** — instantiated module graphs, not retained run data (only 16 k distinct functions back those 2 M closures). Every sampled retainer path ended `require.cache → @activepieces+shared@<version>/…/<dto>.js → exports → <ZodSchema>._def → get shape → refine/pipe/optional/brand closures`: `shared` eagerly builds ~500 top-level zod DTO trees at import, and zod v4 attaches per-instance accessor/method closures (120 k `get`, 67 k `set`, 31 k `validate` in that one heap). The engine held **9 shared versions at once** (0.37 → 0.96.2) because each piece bundle pins its own, and reuse + `import()` pins them forever — including **3 versions of `piece-hubspot` and 2 of `piece-slack` simultaneously**, one per flow-pinned piece version, so republish/upgrade churn multiplies copies of the *same* piece. Budget ≈ 60–70 MB engine baseline + ~40 MB per distinct resident shared copy. Two independent fixes attack it: CDN self-contained bundles (no shared inside pieces at all, see above) and require()-based piece LRU eviction; snapshot mechanics for redoing this measurement are in the `profile-worker-memory` skill. Follow-up exact measurement (same process at 542 MB heap, 9 shared copies, graph-cut retained sizes): one cleanly-severable copy weighs **41.3 MB / 445 k nodes**, but the other 8 each show <1 MB *exclusive* retained because copies are **co-retained as clusters, and the pin is the ESM module map, not `require.cache`**: deleting every `require.cache` entry under both `.bun` store roots frees only ~65 MB of 553 MB, while blocking the ESM `ModuleWrap`/`SyntheticModule` entries too frees **406 MB** (engine-only floor: 147 MB, which includes main.js's own bundled shared). Retainer chain: `Global handles → SyntheticModule (piece import()) → Piece object → action run closures → context:pieces_framework_1 / shared_1 (whole exports)` — every action closure captures its module scope, so a piece and its framework+shared copies live and die together. Consequence: any eviction scheme must make the *piece entry itself* collectable (hence require()-based loading in the LRU fix — an import()ed entry can never be dropped); purging shared's cache entries alone reclaims ~nothing.- **~1.3 GB is the idle floor before a single flow runs.** Same box, freshly booted, zero runs: container 1.325 GiB, of which the API process alone is ~1.18 GB and the worker ~213 MB. Sizing a container off "what a flow needs" is wrong by more than a gigabyte — budget the floor first, then add the sandbox ceiling above.+- **The require()-based LRU in `piece-loader.ts` (PR #14859) makes both CPU and memory *worse* than not evicting, in both the CDN-bundle and raw-npm worlds, and was removed.** The load-path switch from `await import(piecePath)` to `require(piecePath)` doesn't itself reclaim any memory — the resulting loader still holds every loaded piece in `require.cache` until the process exits — but it is the precondition for any eviction mechanism at all, since ESM `ModuleWrap` handles are pinned to the realm regardless of `require.cache` state. The LRU cap-and-evict wrapper Chaker put on top of it was where the memory reclaim was supposed to come from, and it doesn't work in practice. Directly A/B tested `require()` vs `import()` with no eviction on 300-iter rotations: 296 vs 331 MB RSS on CDN, 1963 vs 1969 MB on pre-CDN — the two mechanisms plateau within 6-35 MB of each other, neither reclaims. Two facts made it look like it helped: **(a)** Chaker's PR compared "require() + LRU cap=10" against `await import()` (the old unevictable path), so the entire measured memory win came from the `import → require` switch, not from the LRU; **(b)** his fixture loaded 20 fabricated pieces uniquely (each loaded once, ~10 evictions total), which never exercises the rotate-and-reload churn that real fleets hit. The shipped constant was also `MAX_LOADED_PIECES = 5`, not the 10 in the PR title — a same-day follow-up commit (`a74fc81d58`, empty body) dropped it to 5 after the anchor-pinning bug (`bfce1133b7` — `createRequire(__filename)` anchor held all loaded pieces via its `children` array, engine OOM at 1.1 GB despite the cap) got fixed and left OOM headroom concerns; both landed inside the squash-merge (`71bd79c74d`), so the merged PR reads as if cap=10 shipped. Measured Sep 2026 with a standalone Node harness that inlines the exact eviction algorithm (`scratchpad/lru-bench/`), 300 iterations rotating 15 real published bundles:++ | scenario | cap | iters | evictions | CPU/load mean | CPU/load p95 | RSS peak | RSS plateau (post-GC) | evict share of loader CPU |+ |---|---:|---:|---:|---:|---:|---:|---:|---:|+ | CDN self-contained | 5 (shipped) | 3000 | 2,995 | 63 ms | 80 ms | 315 MB | 313 MB | ~68% |+ | CDN self-contained | 10 | 3000 | 2,990 | 77 ms | 89 ms | 330 MB | 326 MB | ~68% |+ | CDN self-contained | 100 (no evict) | 3000 | 0 | 0.3 ms | 0 ms | 303 MB | 289 MB (plateau iter 20) | 0% |+ | Pre-CDN raw npm | 5 | 300 | 295 | **2131 ms** | 4552 ms | 2209 MB | 2149 MB | 73% |+ | Pre-CDN raw npm | 10 | 300 | 290 | **2513 ms** | 3075 ms | 2659 MB | 2659 MB | 77% |+ | Pre-CDN raw npm | 100 (no evict) | 300 | 0 | 35 ms | 47 ms | 1978 MB | 1978 MB (plateau iter 20) | 0% |++ Three overturning findings. **(1)** No-eviction wins on both axes in both worlds — CDN saves 21 MB and 20-25× CPU/load; pre-CDN saves 171-681 MB (yes, less RSS *without* the cap) and 60× CPU/load. The eviction-and-reload churn allocates faster than the doomed-subtree GC + forced `global.gc()` can reclaim, and cap=10 is worse than cap=5 because a bigger survivor require-graph costs more per `collectReachableModules` walk. **(2)** The eviction path is 60-77% of total loader CPU whenever it fires. **(3)** Per-eviction cost is dominated by `global.gc()` on the just-freed piece's object graph (V8 would collect it on its own schedule without the eager stall) — per-piece variance is 4.5× on real pieces (`date-helper` 8.8 ms → `gmail` 40 ms in the CDN world; pre-CDN pieces run 2–4 s each because `collectReachableModules` walks ~1,425 modules per piece instead of 1). Fit shape: **reused sandbox** (either `AP_REUSE_SANDBOX=true`, or `SANDBOX_CODE_ONLY`/`UNSANDBOXED` which default to reuse unless `AP_REUSE_SANDBOX=false` explicitly overrides — see `canReuseSandbox` in `sandbox-manager.ts`) + a fleet whose sandbox lifetime cycles through more pieces than the cap — Alan's 0.91.2 config with 140+ distinct pieces (pieces-report CSV, Sep 2026, CPU-per-worker sharply up vs 0.86), and Funding Societies before them on 0.88.3 (same complaint, silenced by reverting #14889 not by #14859). Historical note: the same shape has been tried and reverted repeatedly — `1c6efa0389` (fresh child per piece, reverted as #15209), `9a98fd05e9` (killed-per-operation piece-host), `3f6e32d885` (size-based cache eviction that kept `import()`, ineffective). The clean shape is the `import → require` switch alone; the LRU wrapper on top adds tax without a matching benefit and is what shipped removed in `fix/piece-loader-remove-lru`.- **A job-data schema is a transport contract, not a validator — a constraint the piece never had turns a working saved flow into an enqueue-time failure.** Job payloads that carry piece input are shaped by props that were only ever checked in the builder, and often not even there: `Property.Number` accepts `0` and `-1`, a `DynamicProperties` block can hold a value from an older props shape, and `Property.Array` yields whatever was typed. Writing the natural-looking `z.number().int().positive()` or a bare `z.enum([...])` in the job data rejects those payloads at `jobQueue.add` — before the work is attempted, with a validation message no user can act on, on a flow that ran yesterday. Mirror the props' looseness (`z.number()`), and where a value must be one of a set, `z.enum([...]).optional().catch(undefined)` degrades an unknown value to unset instead of failing the job. Validate in the piece if anywhere; the queue is the wrong place to discover bad input.- **Adding any `WorkerJobType` means five edits, and the compiler only catches three.** In `core/execution/.../job-data.ts`: the enum, the `getDefaultJobPriority` switch (exhaustive with no `default`, so this one is a hard compile error), and the `JobData` union. Then `OneTimeJobAddParams` in `job-queue.ts`, or `jobQueue.add` rejects the payload. Then the lazy handler entry in `worker/.../execute/job-registry.ts` — **that last one is the silent miss**: everything compiles and the job enqueues, it just never runs. Also give the job data a `projectId`, `null` if it has none, because `job-broker.ts` reads `migratedData.projectId as string` for every dequeued job and a member without the property breaks that file and `job-data-migrations.ts` rather than your own. `ExecuteAgentRunJobData` is the shape to copy: nullable `projectId`, and it already flows through that cast.diff --git a/packages/server/engine/src/lib/helper/piece-loader.ts b/packages/server/engine/src/lib/helper/piece-loader.tsindex 8e8211c2a0d..b5e01946a59 100644--- a/packages/server/engine/src/lib/helper/piece-loader.ts+++ b/packages/server/engine/src/lib/helper/piece-loader.ts@@ -17,7 +17,7 @@ export const pieceLoader = {devPieces,})const piecePath = await pieceLoader.getPiecePath({ packageName, devPieces })- const module = loadAndCapPieces(piecePath)+ const module = createRequire(__filename)(piecePath)const piece = extractPieceFromModule<Piece>({module,@@ -131,65 +131,6 @@ export const pieceLoader = {},}-const MAX_LOADED_PIECES = 5-const engineRequire = createRequire(__filename)-const loadedPieceEntries = new Map<string, true>()-const baselineModuleIds = new Set(Object.keys(engineRequire.cache))--function loadAndCapPieces(piecePath: string): Record<string, unknown> {- const disposableRequire = createRequire(__filename)- const resolvedEntry = disposableRequire.resolve(piecePath)- const loadedModule: Record<string, unknown> = disposableRequire(resolvedEntry)- loadedPieceEntries.delete(resolvedEntry)- loadedPieceEntries.set(resolvedEntry, true)- if (loadedPieceEntries.size > MAX_LOADED_PIECES) {- const leastRecentEntry = loadedPieceEntries.keys().next().value- if (!isNil(leastRecentEntry)) {- loadedPieceEntries.delete(leastRecentEntry)- evictPieceSubtree(leastRecentEntry)- global.gc?.()- }- }- return loadedModule-}--function evictPieceSubtree(evictedEntry: string): void {- const survivorReachable = collectReachableModules([...loadedPieceEntries.keys()])- const evictedReachable = collectReachableModules([evictedEntry])- const doomed = new Set([...evictedReachable].filter((id) => !survivorReachable.has(id) && !baselineModuleIds.has(id)))- if (doomed.size === 0) {- return- }- for (const id of Object.keys(engineRequire.cache)) {- const mod = engineRequire.cache[id]- if (!isNil(mod) && !doomed.has(id)) {- mod.children = mod.children.filter((child) => !doomed.has(child.id))- }- }- if (!isNil(engineRequire.main)) {- engineRequire.main.children = engineRequire.main.children.filter((child) => !doomed.has(child.id))- }- for (const id of doomed) {- Reflect.deleteProperty(engineRequire.cache, id)- }-}--function collectReachableModules(rootIds: string[]): Set<string> {- const reachable = new Set<string>()- const stack = rootIds.filter((id) => !isNil(engineRequire.cache[id]))- while (stack.length > 0) {- const id = stack.pop()- if (isNil(id) || reachable.has(id)) {- continue- }- reachable.add(id)- for (const child of engineRequire.cache[id]?.children ?? []) {- stack.push(child.id)- }- }- return reachable-}-async function findInDistFolder(packageName: string): Promise<string | null> {const sourcePiecesPath = path.resolve('packages/pieces')if (!await utils.folderExists(sourcePiecesPath)) {