AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(worker): installing pieces no longer depends on the worker reaching the public URL (#16037)
Ahmad Tashdbc140a31af1
diff --git a/brain/knowledge/decisions/000013-sandbox-pool-is-a-pure-execute-cloud-run-pool-superseded.md b/brain/knowledge/decisions/000013-sandbox-pool-is-a-pure-execute-cloud-run-pool-superseded.mdindex f48ecfaaec3..1ae6eb21610 100644--- a/brain/knowledge/decisions/000013-sandbox-pool-is-a-pure-execute-cloud-run-pool-superseded.md+++ b/brain/knowledge/decisions/000013-sandbox-pool-is-a-pure-execute-cloud-run-pool-superseded.md@@ -15,5 +15,5 @@ A pure `execute()` gives local and remote an identical signature and needs no ap## ConsequencesThe remote Cloud Run / detached-pool hosting was removed — see "Worker is the Sandbox". The pure-execute shape survives (the Sandbox still holds no app connection and consumes only resolved inputs), but two things premised on a remote host are now retired:-- **ADR 0002 (pieces are links):** the bundle link being built from `publicApiUrl` was justified by "reachable from a detached remote pool." With no remote pool that reason is gone — the "links" decision still stands on its other merits (transport-uniform install, lazy S3 warm, always-a-working-link for self-hosters), but the public-URL basis is now incidental and could move to the internal URL. ADR files 0001/0002 marked superseded (2026-07-16), not deleted.-- **Worker→app routing:** for a standalone worker (`AP_CONTAINER_TYPE=WORKER`), `AP_FRONTEND_URL` is the base for ALL per-execution engine→app callbacks (`internalApiUrl`: worker/project, populated-flows, connections/props, run-progress, file I/O, piece serverContext.apiUrl). `publicApiUrl` (from the app's `PUBLIC_URL`) is only string injection (webhook URLs, serverContext.publicUrl) plus the cached, provisioning-time bundle download — not the per-request path. So pointing `AP_FRONTEND_URL` at the cluster-internal service keeps every callback off the public endpoint (avoids DNS + TLS + LB hairpin); it's safe because user-facing/public URLs come independently from `PUBLIC_URL`.+- **ADR 0002 (pieces are links):** the bundle link being built from `publicApiUrl` was justified by "reachable from a detached remote pool." With no remote pool that reason is gone — the "links" decision still stands on its other merits (transport-uniform install, lazy S3 warm, always-a-working-link for self-hosters), but the public-URL basis is now incidental and has since moved to the internal URL (GIT-1718). ADR files 0001/0002 marked superseded (2026-07-16), not deleted.+- **Worker→app routing:** for a standalone worker (`AP_CONTAINER_TYPE=WORKER`), `AP_FRONTEND_URL` is the base for ALL per-execution engine→app callbacks (`internalApiUrl`: worker/project, populated-flows, connections/props, run-progress, file I/O, piece serverContext.apiUrl, piece bundle downloads). `publicApiUrl` (from the app's `PUBLIC_URL`) is only string injection (webhook URLs, serverContext.publicUrl) — the worker never fetches from it. So pointing `AP_FRONTEND_URL` at the cluster-internal service keeps every callback off the public endpoint (avoids DNS + TLS + LB hairpin); it's safe because user-facing/public URLs come independently from `PUBLIC_URL`.diff --git a/brain/knowledge/engineering/helm-chart.md b/brain/knowledge/engineering/helm-chart.mdindex 3334622846f..7e261ae8759 100644--- a/brain/knowledge/engineering/helm-chart.md+++ b/brain/knowledge/engineering/helm-chart.md@@ -30,7 +30,7 @@ Only two secrets, both `data: {}` with mittwald `secret-generator` annotations t- **A user-owned key is pinned once it exists.** If `AP_ENCRYPTION_KEY` / `AP_JWT_SECRET` are in the user's secret when the chart renders, their `secretKeyRef` becomes `optional: false`, so a deleted secret stops the pod instead of letting the generated fallback key silently take over.- **Never render one env name twice to get precedence.** Helm 4 installs with server-side apply, which rejects duplicate `env` keys ("duplicate entries for key"). A lower-precedence default has to come through `envFrom`.- **Removing a duplicate env name on a Helm 3 upgrade deletes both entries.** The strategic merge patch matches `env` items by name ("hides previous definition ... may be dropped when using apply"), so a var set in both values keys vanishes from the live pod spec. Never change how an already-duplicated name renders; fix duplicates in the user's values instead.-- **`AP_FRONTEND_URL` must be reachable from inside the pod.** The worker downloads piece bundles through the public URL, so a `localhost` value boots fine and then fails every flow with `fetch failed`.+- **Up to 0.92.x, `AP_FRONTEND_URL` must be reachable from inside the pod.** Those releases download piece bundles through the public URL, so a `localhost` value boots fine and then fails every flow with `fetch failed`. Later releases download over the worker's own app URL (`127.0.0.1` in the chart's single container), see the bundle-download gotcha on *Workers*.- **The mittwald subchart's ClusterRole is named after the release.** Two releases with the same name in different namespaces collide, and deleting a namespace without `helm uninstall` leaves it behind. A second release can set `kubernetes-secret-generator.enabled: false`; the first operator watches every namespace.- **Helm 4 upgrades of a `Rollout` need `--force-conflicts`.** Argo's controller owns the Service `.spec.selector` (it adds `rollouts-pod-template-hash`), so server-side apply conflicts. After a forced upgrade Argo re-adds the hash and traffic keeps routing.- **The HPA targets whatever `activepieces.workloadType` resolves to.** It used to target a `Deployment` the chart never creates.diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.mdindex 29aff6e2275..2d23b2c2a9e 100644--- a/brain/knowledge/execution-runtime/workers.md+++ b/brain/knowledge/execution-runtime/workers.md@@ -20,7 +20,8 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the- `concurrency === 1` primes the sandbox to the full container RAM (cgroup-aware) via `primeFullContainerMemory()`.### Gotchas-- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.+- **Piece bundles download over the worker's own `AP_FRONTEND_URL` (`internalApiUrl`), never the app's `PUBLIC_URL` — keep it that way.** Through 0.92.x the sandbox built `GET /v1/engine/pieces/bundle` from the `PUBLIC_URL` the app pushes over the socket, so a standalone worker ignored its own `AP_FRONTEND_URL=http://app` for this one hop. A `localhost` public URL (the docs' default for compose and `docker run -p 8080:80`) failed every uncached install with `fetch failed` (GIT-1838). A public URL that reached a *different* instance sharing the JWT secret returned 404, so custom-piece upload failed with `PIECE_BUNDLE_NOT_AVAILABLE` while the `PACKAGE_ARCHIVE` row sat intact in Postgres. Tell-tale: the bundle GET never shows up in the app's own request log. Fixed by GIT-1718.+- **`AP_CONTAINER_TYPE=WORKER` moves every engine→app call onto `AP_FRONTEND_URL`.** In `WORKER_AND_APP` the engine reaches the app at `http://127.0.0.1:<port>/api/`; in `WORKER` it uses `AP_FRONTEND_URL + '/api/'` (`getApiUrl` in `worker/src/lib/config/configs.ts`), and the worker's Socket.IO connection follows it. If that URL is the public hostname, file uploads, run-log uploads, piece bundle downloads and the worker socket all cross the ingress/WAF/CDN, so WAF rules and websocket idle timeouts start failing runs that worked the day before. Point separated workers at the in-cluster app service instead. Found on Alan's fleet 2026-09-29, the day after being advised to switch to `WORKER`.- **The piece workspace pins `linker = "isolated"` in its own `bunfig.toml` — do not delete it, the engine's piece resolver depends on the layout bun chooses.** `pieceInstaller` generates a bun workspace at `<cache>/<version>/common` whose members are `pieces/<pieceName>-<pieceVersion>`, and the engine (`piece-loader.ts` → `resolveInstalledPieceEntry`) resolves a piece **only** at `pieces/<alias>/node_modules/<pieceName>`, which just the isolated linker produces. The workspace used to carry no bunfig and inherited whatever ancestor bunfig bun discovered above the cache mount (`/usr/src/app/bunfig.toml`); when bun instead uses the **hoisted** linker it writes the package as a real directory at the workspace root `node_modules/@activepieces/<name>` and leaves the member with no `node_modules` at all, so every run of that piece fails `PieceNotFoundError: Piece not found for package: <name>-<version>` while `bun install` still exits 0. It was silent and self-perpetuating — `markPiecesAsUsed` writes `ready` and `pieceCheckIfAlreadyInstalled` only looks for *a* `node_modules`, so the piece is reinstalled forever and never resolves. Sep 2026 production: 125/125 folders missing `node_modules` were exactly the 125 packages sitting hoisted at the workspace root; 7,400+ `EXECUTE_FLOW` jobs, 15+ platforms, ~60 official pieces, ~300/hr. Two things had to line up — bun 1.4.0 (Aug 31) changed the layout newly-installed members got, and removing the piece-upgrade Redis gate (#15236, deployed 2026-09-03 08:09 UTC) mass-upgraded every platform's flows to newer piece versions so almost every install was suddenly a *first-time* install. Diagnose on a worker with `for d in <cache>/v*/common/pieces/@activepieces/*/; do [ -d "$d/node_modules" ] || basename $d; done`. Fixed by pinning the linker plus a `LATEST_CACHE_VERSION` bump (v14 → v15) so the fleet rebuilds one clean workspace under the pinned layout rather than carrying a mixed one.- **No PM2 anymore — the container is the supervision unit.** `docker-entrypoint.sh` launches the bootstrap scripts with plain `node --enable-source-maps` (removed the `pm2-runtime` + `/tmp/ecosystem.config.js` machinery). `APP`/`WORKER` `exec` a single node as PID 1; `WORKER_AND_APP` runs both and, if either exits, kills the other and exits non-zero so the orchestrator restarts the whole container. This drops PM2's *in-container* crash/OOM restart: an OOM-kill no longer silently recycles a child every ~4 min behind a `RestartCount: 0` (the 2026-07-26 wedge's supply of retry attempts — see below); the container now dies and is rescheduled instead. The historical incident notes below still describe the old PM2 behavior as it happened.- **Version gate (rolling-deploy safety)**: dispatch requires an exact release match, enforced both sides via `versionsAreCompatible` (fail-closed — `undefined` or `UNKNOWN_VERSION` `'0.0.0'` is treated incompatible). App withholds jobs from a mismatched worker (`poll` returns null); worker pauses polling 10s. Ordinary mismatch self-heals on convergence; a read failure does not (cached for process life) and pages on-call once at startup via `assertReleaseReadable`.diff --git a/packages/server/sandbox/src/lib/cache/local-execution-cache.ts b/packages/server/sandbox/src/lib/cache/local-execution-cache.tsindex 74fc56ff29c..698cc284275 100644--- a/packages/server/sandbox/src/lib/cache/local-execution-cache.ts+++ b/packages/server/sandbox/src/lib/cache/local-execution-cache.ts@@ -13,7 +13,7 @@ export const localExecutionCache = (log: ApLogger, basePath: string, getSettingsasync provision({pieces,codeSteps,- publicApiUrl,+ internalApiUrl,engineToken,bestEffort,}: ProvisionParams): Promise<void> {@@ -56,7 +56,7 @@ export const localExecutionCache = (log: ApLogger, basePath: string, getSettingsawait pieceInstaller(log, basePath, getSettings).install({pieces: uniquePieces,includeFilters: true,- publicApiUrl,+ internalApiUrl,engineToken,bestEffort,})@@ -114,7 +114,7 @@ type InstallCodeStepParams = {type ProvisionParams = {pieces: PiecePackage[]codeSteps: CodeArtifact[]- publicApiUrl: string+ internalApiUrl: stringengineToken: stringbestEffort?: boolean}diff --git a/packages/server/sandbox/src/lib/cache/pieces/piece-installer.ts b/packages/server/sandbox/src/lib/cache/pieces/piece-installer.tsindex a07bd1d5c6f..5ea78a695f7 100644--- a/packages/server/sandbox/src/lib/cache/pieces/piece-installer.ts+++ b/packages/server/sandbox/src/lib/cache/pieces/piece-installer.ts@@ -15,10 +15,10 @@ const relativePiecePath = (piece: PiecePackage) => join('./', 'pieces', `${piececonst piecePath = (rootWorkspace: string, piece: PiecePackage) => join(rootWorkspace, 'pieces', `${piece.pieceName}-${piece.pieceVersion}`)export const pieceInstaller = (log: ApLogger, basePath: string, getSettings: () => SandboxSettings) => ({- async install({ pieces, includeFilters, publicApiUrl, engineToken, bestEffort, force }: InstallParams): Promise<void> {+ async install({ pieces, includeFilters, internalApiUrl, engineToken, bestEffort, force }: InstallParams): Promise<void> {const groupedPieces = groupPiecesByPackagePath(pieces, basePath, getSettings)const installPromises = Object.entries(groupedPieces).map(async ([packagePath, piecesInGroup]) => {- await installPieces(packagePath, piecesInGroup, includeFilters, log, { publicApiUrl, engineToken }, getSettings, bestEffort ?? false, force ?? false)+ await installPieces(packagePath, piecesInGroup, includeFilters, log, { internalApiUrl, engineToken }, getSettings, bestEffort ?? false, force ?? false)})await Promise.all(installPromises)},@@ -305,12 +305,12 @@ async function downloadBundleWithRetry({ rootWorkspace, piece, bundleSource }: Dthrow lastError}-async function downloadBundleIfNotCached({ rootWorkspace, piece, bundleSource: { publicApiUrl, engineToken } }: DownloadBundleParams): Promise<void> {+async function downloadBundleIfNotCached({ rootWorkspace, piece, bundleSource: { internalApiUrl, engineToken } }: DownloadBundleParams): Promise<void> {const bundlePath = bundleTgzPath(rootWorkspace, piece)if (await fileSystemUtils.fileExists(bundlePath)) {return}- const url = pieceBundleEndpointUrl(publicApiUrl, piece)+ const url = pieceBundleEndpointUrl(internalApiUrl, piece)const response = await fetch(url, { headers: { Authorization: `Bearer ${engineToken}` } })if (!response.ok) {if (response.status === 404 || response.status === 410) {@@ -325,8 +325,8 @@ async function downloadBundleIfNotCached({ rootWorkspace, piece, bundleSource: {await writeFile(bundlePath, Buffer.from(await response.arrayBuffer()))}-function pieceBundleEndpointUrl(publicApiUrl: string, piece: PiecePackage): string {- const base = `${ensureTrailingSlash(publicApiUrl)}v1/engine/pieces/bundle`+function pieceBundleEndpointUrl(internalApiUrl: string, piece: PiecePackage): string {+ const base = `${ensureTrailingSlash(internalApiUrl)}v1/engine/pieces/bundle`if (piece.packageType === PackageType.ARCHIVE) {return `${base}?archiveId=${encodeURIComponent(piece.archiveId)}`}@@ -416,14 +416,14 @@ type PieceEntryFileExistsParams = {type InstallParams = {pieces: PiecePackage[]includeFilters: boolean- publicApiUrl: string+ internalApiUrl: stringengineToken: stringbestEffort?: booleanforce?: boolean}type BundleSource = {- publicApiUrl: string+ internalApiUrl: stringengineToken: string}diff --git a/packages/server/sandbox/src/lib/engine-rpc-handlers.ts b/packages/server/sandbox/src/lib/engine-rpc-handlers.tsindex 96627c73c83..9a7d5a3073f 100644--- a/packages/server/sandbox/src/lib/engine-rpc-handlers.ts+++ b/packages/server/sandbox/src/lib/engine-rpc-handlers.ts@@ -36,7 +36,7 @@ export const engineRpcHandlers = ({ log, basePath, getSettings, provision }: Engawait installer.install({pieces: piecesToReinstall,includeFilters: true,- publicApiUrl: provision.publicApiUrl,+ internalApiUrl: provision.internalApiUrl,engineToken: provision.engineToken,force: true,})diff --git a/packages/server/sandbox/src/lib/resolver.ts b/packages/server/sandbox/src/lib/resolver.tsindex 968f2ff914a..fb69ca88fc8 100644--- a/packages/server/sandbox/src/lib/resolver.ts+++ b/packages/server/sandbox/src/lib/resolver.ts@@ -40,7 +40,7 @@ export function createResolver({ apiClient, basePath, getSettings, log }: CreateflowVersionId: flowVersion?.id,pieces: uniquePieces,codes,- publicApiUrl: input.publicApiUrl,+ internalApiUrl: input.internalApiUrl,engineToken: input.engineToken,}return { kind: 'ready', provision, flowVersion }diff --git a/packages/server/sandbox/src/lib/sandbox.ts b/packages/server/sandbox/src/lib/sandbox.tsindex 7ed25c9c777..b1e5f1324df 100644--- a/packages/server/sandbox/src/lib/sandbox.ts+++ b/packages/server/sandbox/src/lib/sandbox.ts@@ -47,7 +47,7 @@ export function createSandboxRuntime({ concurrency = 1, basePath, getSettings }:const { error: provisionError } = await tryCatch(() => localExecutionCache(log, basePath, getSettings).provision({pieces: provision.pieces,codeSteps: provision.codes,- publicApiUrl: provision.publicApiUrl,+ internalApiUrl: provision.internalApiUrl,engineToken: provision.engineToken,}))if (provisionError) {@@ -119,8 +119,8 @@ export function createSandboxRuntime({ concurrency = 1, basePath, getSettings }:busy: info.busy,}))},- async prewarm({ log, apiClient, publicApiUrl, flow }: PreWarmSandboxParams): Promise<void> {- if (isNil(apiClient) || isNil(publicApiUrl)) {+ async prewarm({ log, apiClient, internalApiUrl, flow }: PreWarmSandboxParams): Promise<void> {+ if (isNil(apiClient) || isNil(internalApiUrl)) {return}const startedAt = Date.now()@@ -146,11 +146,11 @@ export function createSandboxRuntime({ concurrency = 1, basePath, getSettings }:resolver: createResolver({ apiClient, basePath, getSettings, log }),flows: prewarmData.flows ?? [],platformId,- publicApiUrl,+ internalApiUrl,engineToken,log,})- await localExecutionCache(log, basePath, getSettings).provision({ pieces, codeSteps, publicApiUrl, engineToken, bestEffort: true })+ await localExecutionCache(log, basePath, getSettings).provision({ pieces, codeSteps, internalApiUrl, engineToken, bestEffort: true })log.info({ pieceCount: pieces.length, codeStepCount: codeSteps.length, durationMs: Date.now() - startedAt }, 'Prewarmed sandbox cache')})if (error) {@@ -184,9 +184,9 @@ async function fetchScopeFile({ apiClient, scopeFileId }: FetchScopeFileParams):return { pieces: content.pieces, codeSteps: content.codes }}-async function resolveFlowsForPrewarm({ resolver, flows, platformId, publicApiUrl, engineToken, log }: ResolveFlowsForPrewarmParams): Promise<ResolvedPrewarmInputs> {+async function resolveFlowsForPrewarm({ resolver, flows, platformId, internalApiUrl, engineToken, log }: ResolveFlowsForPrewarmParams): Promise<ResolvedPrewarmInputs> {const resolvedFlows = await Promise.all(flows.map(async (flow) => {- const { data: resolved, error: flowError } = await tryCatch(() => resolver.resolve({ flow, platformId, publicApiUrl, engineToken }))+ const { data: resolved, error: flowError } = await tryCatch(() => resolver.resolve({ flow, platformId, internalApiUrl, engineToken }))if (flowError) {log.warn({ error: String(flowError), flow: { id: flow.id } }, 'Failed to resolve flow for prewarm')return null@@ -217,7 +217,7 @@ type ResolveFlowsForPrewarmParams = {resolver: Resolverflows: { id: string, versionId: string, projectId: string }[]platformId: string- publicApiUrl: string+ internalApiUrl: stringengineToken: stringlog: ApLogger}diff --git a/packages/server/sandbox/src/lib/types.ts b/packages/server/sandbox/src/lib/types.tsindex 3b2be41bdc8..bceac8401f5 100644--- a/packages/server/sandbox/src/lib/types.ts+++ b/packages/server/sandbox/src/lib/types.ts@@ -14,7 +14,7 @@ export type Resolver = {export type ResolveInput = {platformId: string- publicApiUrl: string+ internalApiUrl: stringengineToken: stringflow?: { id: string, versionId: string, projectId: string }pieces?: PiecePackage[]@@ -49,22 +49,22 @@ export type ExecuteParams = {export type PreWarmSandboxParams = {log: ApLoggerapiClient?: WorkerToApiContract- publicApiUrl?: string+ internalApiUrl?: string// Warm just this flow (e.g. on publish) instead of the platform's whole active set.flow?: { id: string, versionId: string, projectId: string }}-// The Resolver's output and the pool's input. The pool installs each piece straight from a link: it-// builds `${publicApiUrl}v1/engine/pieces/bundle?name=&version=&token=` per piece and hands that URL-// to `bun install`, which follows the endpoint's redirect to npm / signed-S3 (or streams the custom-// archive). No bytes cross the worker socket and the pool never imports WorkerToApiContract; the link-// is publicApiUrl-based so it is reachable from a remote pool (Cloud Run). See ADR 0002.+// The Resolver's output and the pool's input. The pool fetches each piece from+// `${internalApiUrl}v1/engine/pieces/bundle` (engine token as a Bearer header), following the redirect+// to npm / signed-S3 (or the streamed custom archive), saves it as a local .tgz, and `bun install`s+// that file. No bytes cross the worker socket and the pool never imports WorkerToApiContract; the URL+// is the worker's own app URL (ends in `/api/`), the one it already reaches the app on. See ADR 0002.export type ProvisionInput = {platformId: stringflowVersionId?: stringpieces: PiecePackage[]codes: CodeArtifact[]- publicApiUrl: string+ internalApiUrl: stringengineToken: string}diff --git a/packages/server/sandbox/test/lib/cache/piece-installer.test.ts b/packages/server/sandbox/test/lib/cache/piece-installer.test.tsindex 0b30dec8110..06a9ae2cd2c 100644--- a/packages/server/sandbox/test/lib/cache/piece-installer.test.ts+++ b/packages/server/sandbox/test/lib/cache/piece-installer.test.ts@@ -81,7 +81,7 @@ const fakeLog = {} as unknown as ApLogger// Every piece is installed from its bundle link; the dependency value is the engine bundle endpoint.-const bundleSource = { publicApiUrl: 'http://localhost:3000/api/', engineToken: 'test-token' }+const bundleSource = { internalApiUrl: 'http://localhost:3000/api/', engineToken: 'test-token' }const fakeGetSettings = () => ({EXECUTION_MODE: 'UNSANDBOXED',@@ -131,6 +131,21 @@ describe('pieceInstaller', () => {expect(await pathExists(readyFilePath(piece2))).toBe(true)})+ it('cold install downloads every bundle from the internal API URL with the engine token', async () => {+ const registryPiece = makePiece('@activepieces/piece-cold')+ const archivePiece = makeArchivePiece('@acme/piece-private')+ const installer = pieceInstaller(fakeLog, testWorkspace, fakeGetSettings)++ mockInstall.mockResolvedValueOnce({ output: '' })++ await installer.install({ pieces: [registryPiece, archivePiece], includeFilters: true, ...bundleSource })++ const expectedInit = { headers: { Authorization: 'Bearer test-token' } }+ expect(vi.mocked(fetch)).toHaveBeenCalledTimes(2)+ expect(vi.mocked(fetch)).toHaveBeenCalledWith('http://localhost:3000/api/v1/engine/pieces/bundle?name=%40activepieces%2Fpiece-cold&version=1.0.0', expectedInit)+ expect(vi.mocked(fetch)).toHaveBeenCalledWith(`http://localhost:3000/api/v1/engine/pieces/bundle?archiveId=${archivePiece.archiveId}`, expectedInit)+ })+it('batch fails with good and bad piece — good piece marked ready, bad piece rolled back', async () => {const good = makePiece('@activepieces/piece-good')const bad = makePiece('@activepieces/piece-bad')diff --git a/packages/server/sandbox/test/lib/sandbox.test.ts b/packages/server/sandbox/test/lib/sandbox.test.tsindex 6f37221ec31..80d633c1560 100644--- a/packages/server/sandbox/test/lib/sandbox.test.ts+++ b/packages/server/sandbox/test/lib/sandbox.test.ts@@ -69,7 +69,7 @@ function buildExecuteParams(workerIndex: number, expiresAt?: number) {flowVersionId: 'fv1',pieces: [],codes: [],- publicApiUrl: 'http://localhost/api/',+ internalApiUrl: 'http://localhost/api/',engineToken: 't',},} as neverdiff --git a/packages/server/worker/src/lib/api-notify-service.ts b/packages/server/worker/src/lib/api-notify-service.tsindex ecf29088739..5d92fcc25f4 100644--- a/packages/server/worker/src/lib/api-notify-service.ts+++ b/packages/server/worker/src/lib/api-notify-service.ts@@ -2,14 +2,14 @@ import { Runtime } from '@activepieces/sandbox'import { ApLogger } from '@activepieces/server-utils'import { ApiToWorkerContract, WorkerToApiContract } from '@activepieces/shared'-export function createApiToWorkerHandlers({ getRuntime, apiClient, getPublicApiUrl, log }: CreateApiToWorkerHandlersParams): ApiToWorkerContract {+export function createApiToWorkerHandlers({ getRuntime, apiClient, getInternalApiUrl, log }: CreateApiToWorkerHandlersParams): ApiToWorkerContract {return {flowPublished({ flowId, flowVersionId, projectId }) {log.info({ flowId, flowVersionId, projectId, message: 'Flow published, prewarming flow cache' })void getRuntime()?.prewarm({log,apiClient,- publicApiUrl: getPublicApiUrl(),+ internalApiUrl: getInternalApiUrl(),flow: { id: flowId, versionId: flowVersionId, projectId },})},@@ -19,6 +19,6 @@ export function createApiToWorkerHandlers({ getRuntime, apiClient, getPublicApiUtype CreateApiToWorkerHandlersParams = {getRuntime: () => Runtime | nullapiClient: WorkerToApiContract- getPublicApiUrl: () => string+ getInternalApiUrl: () => stringlog: ApLogger}diff --git a/packages/server/worker/src/lib/execute/jobs/execute-action.ts b/packages/server/worker/src/lib/execute/jobs/execute-action.tsindex 2e432905c9f..66a9f5d2697 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-action.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-action.ts@@ -10,7 +10,7 @@ export const executeActionJob: JobHandler<ExecuteActionJobData, SynchronousJobRejobType: WorkerJobType.EXECUTE_ACTION,async execute(ctx: JobContext, data: ExecuteActionJobData): Promise<SynchronousJobResult> {const { codes, namespace: codeNamespace } = await resolveCodeStep({ step: data.step, platformId: data.platformId })- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, pieces: data.piece ? [data.piece] : [], codes })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, pieces: data.piece ? [data.piece] : [], codes })if (resolved.kind !== 'ready') {throw new Error(`Unexpected resolve outcome "${resolved.kind}" for action-run action job`)}diff --git a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts b/packages/server/worker/src/lib/execute/jobs/execute-flow.tsindex 176e7c74b96..b82c98880ad 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts@@ -13,7 +13,7 @@ export const executeFlowJob: JobHandler<ExecuteFlowJobData, FireAndForgetJobResuconst timeoutInSeconds = workerSettings.getSettings().FLOW_TIMEOUT_SECONDSconst { data: resolved, error: provisionError } = await tryCatch(() =>- ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionId, projectId: data.projectId } }),+ ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionId, projectId: data.projectId } }),)if (provisionError) {await reportFlowStatus({ ctx, data, status: FlowRunStatus.INTERNAL_ERROR, internalError: toInternalError(RunInternalErrorSource.WORKER, provisionError) })diff --git a/packages/server/worker/src/lib/execute/jobs/execute-polling.ts b/packages/server/worker/src/lib/execute/jobs/execute-polling.tsindex 80b57d4933e..f7cfe507eb1 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-polling.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-polling.ts@@ -10,7 +10,7 @@ export const executePollingJob: JobHandler<PollingJobData, FireAndForgetJobResulasync execute(ctx: JobContext, data: PollingJobData): Promise<FireAndForgetJobResult> {const timeoutInSeconds = workerSettings.getSettings().TRIGGER_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionId, projectId: data.projectId } })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionId, projectId: data.projectId } })if (resolved.kind === 'flow-not-found') {ctx.log.info({ flowVersion: { id: data.flowVersionId } }, 'Flow version not found for polling trigger, skipping')diff --git a/packages/server/worker/src/lib/execute/jobs/execute-property.ts b/packages/server/worker/src/lib/execute/jobs/execute-property.tsindex 2eff40d12ac..1881d365579 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-property.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-property.ts@@ -10,7 +10,7 @@ export const executePropertyJob: JobHandler<ExecutePropertyJobData, SynchronousJasync execute(ctx: JobContext, data: ExecutePropertyJobData): Promise<SynchronousJobResult> {const timeoutInSeconds = workerSettings.getSettings().TRIGGER_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })if (resolved.kind !== 'ready') {throw new Error(`Unexpected resolve outcome "${resolved.kind}" for piece-only job`)}diff --git a/packages/server/worker/src/lib/execute/jobs/execute-token-refresh.ts b/packages/server/worker/src/lib/execute/jobs/execute-token-refresh.tsindex bd67ef1060d..9e31a4292ba 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-token-refresh.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-token-refresh.ts@@ -15,7 +15,7 @@ export const executeTokenRefreshJob: JobHandler<ExecuteTokenRefreshJobData, Syncasync execute(ctx: JobContext, data: ExecuteTokenRefreshJobData): Promise<SynchronousJobResult> {const timeoutInSeconds = workerSettings.getSettings().TRIGGER_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })if (resolved.kind !== 'ready') {throw new Error(`Unexpected resolve outcome "${resolved.kind}" for piece-only job`)}diff --git a/packages/server/worker/src/lib/execute/jobs/execute-trigger-hook.ts b/packages/server/worker/src/lib/execute/jobs/execute-trigger-hook.tsindex 144ab21a9db..449ec457d1a 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-trigger-hook.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-trigger-hook.ts@@ -11,7 +11,7 @@ export const executeTriggerHookJob: JobHandler<ExecuteTriggerHookJobData, Synchrasync execute(ctx: JobContext, data: ExecuteTriggerHookJobData): Promise<SynchronousJobResult> {const timeoutInSeconds = workerSettings.getSettings().TRIGGER_HOOKS_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionId, projectId: data.projectId } })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionId, projectId: data.projectId } })if (resolved.kind === 'flow-not-found') {ctx.log.info({ flowVersion: { id: data.flowVersionId } }, 'Flow version not found for trigger hook, skipping')diff --git a/packages/server/worker/src/lib/execute/jobs/execute-validation.ts b/packages/server/worker/src/lib/execute/jobs/execute-validation.tsindex 5a218a44d6c..a61f4ce1a6b 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-validation.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-validation.ts@@ -9,7 +9,7 @@ export const executeValidationJob: JobHandler<ExecuteValidateAuthJobData, Synchrasync execute(ctx: JobContext, data: ExecuteValidateAuthJobData): Promise<SynchronousJobResult> {const timeoutInSeconds = workerSettings.getSettings().TRIGGER_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })if (resolved.kind !== 'ready') {throw new Error(`Unexpected resolve outcome "${resolved.kind}" for piece-only job`)}diff --git a/packages/server/worker/src/lib/execute/jobs/execute-webhook.ts b/packages/server/worker/src/lib/execute/jobs/execute-webhook.tsindex cc3b057727b..37eeca55efd 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-webhook.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-webhook.ts@@ -27,7 +27,7 @@ export const executeWebhookJob: JobHandler<WebhookJobData, FireAndForgetJobResulconst settings = workerSettings.getSettings()const timeoutInSeconds = settings.TRIGGER_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionIdToRun, projectId: data.projectId } })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionIdToRun, projectId: data.projectId } })if (resolved.kind === 'flow-not-found') {ctx.log.info({ flowVersion: { id: data.flowVersionIdToRun } }, 'Flow version not found for webhook, skipping')diff --git a/packages/server/worker/src/lib/execute/jobs/extract-piece-info.ts b/packages/server/worker/src/lib/execute/jobs/extract-piece-info.tsindex 07b77abfe4a..803dd979cf9 100644--- a/packages/server/worker/src/lib/execute/jobs/extract-piece-info.ts+++ b/packages/server/worker/src/lib/execute/jobs/extract-piece-info.ts@@ -12,7 +12,7 @@ export const extractPieceInfoJob: JobHandler<ExecuteExtractPieceMetadataJobData,async execute(ctx: JobContext, data: ExecuteExtractPieceMetadataJobData): Promise<SynchronousJobResult> {const timeoutInSeconds = workerSettings.getSettings().TRIGGER_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })if (resolved.kind !== 'ready') {throw new Error(`Unexpected resolve outcome "${resolved.kind}" for piece-only job`)}diff --git a/packages/server/worker/src/lib/execute/jobs/renew-webhook.ts b/packages/server/worker/src/lib/execute/jobs/renew-webhook.tsindex 354552bd3b0..c4bc1fd4bcb 100644--- a/packages/server/worker/src/lib/execute/jobs/renew-webhook.ts+++ b/packages/server/worker/src/lib/execute/jobs/renew-webhook.ts@@ -9,7 +9,7 @@ export const renewWebhookJob: JobHandler<RenewWebhookJobData, FireAndForgetJobReasync execute(ctx: JobContext, data: RenewWebhookJobData): Promise<FireAndForgetJobResult> {const timeoutInSeconds = workerSettings.getSettings().TRIGGER_HOOKS_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionId, projectId: data.projectId } })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, flow: { id: data.flowId, versionId: data.flowVersionId, projectId: data.projectId } })if (resolved.kind === 'flow-not-found') {ctx.log.info({ flowVersion: { id: data.flowVersionId } }, 'Flow version not found for renew webhook, skipping')diff --git a/packages/server/worker/src/lib/execute/jobs/resolve-connection-identifier.ts b/packages/server/worker/src/lib/execute/jobs/resolve-connection-identifier.tsindex da07ce8edbd..2bb564517d3 100644--- a/packages/server/worker/src/lib/execute/jobs/resolve-connection-identifier.ts+++ b/packages/server/worker/src/lib/execute/jobs/resolve-connection-identifier.ts@@ -8,7 +8,7 @@ export const resolveConnectionIdentifierJob: JobHandler<ExecuteResolveConnectionasync execute(ctx: JobContext, data: ExecuteResolveConnectionIdentifierJobData): Promise<SynchronousJobResult> {const timeoutInSeconds = workerSettings.getSettings().TRIGGER_TIMEOUT_SECONDS- const resolved = await ctx.resolver.resolve({ platformId: data.platformId, publicApiUrl: ctx.publicApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })+ const resolved = await ctx.resolver.resolve({ platformId: data.platformId, internalApiUrl: ctx.internalApiUrl, engineToken: ctx.engineToken, pieces: [data.piece] })if (resolved.kind !== 'ready') {throw new Error(`Unexpected resolve outcome "${resolved.kind}" for piece-only job`)}diff --git a/packages/server/worker/src/lib/worker.ts b/packages/server/worker/src/lib/worker.tsindex 590cb089528..9e29fc4caf8 100644--- a/packages/server/worker/src/lib/worker.ts+++ b/packages/server/worker/src/lib/worker.ts@@ -123,7 +123,7 @@ export const worker = {createNotifyServer<ApiToWorkerContract>(socket, createApiToWorkerHandlers({getRuntime: () => runtime,apiClient,- getPublicApiUrl: () => ensurePublicApiUrl(workerSettings.getSettings().PUBLIC_URL),+ getInternalApiUrl: getApiUrl,log: logger,}), logger)@@ -196,7 +196,7 @@ async function startPollingWorkers(apiClient: WorkerToApiContract): Promise<voidconst { error: prewarmError } = await tryCatch(() => createdRuntime.prewarm({log: logger,apiClient,- publicApiUrl: ensurePublicApiUrl(workerSettings.getSettings().PUBLIC_URL),+ internalApiUrl: getApiUrl(),}))if (prewarmError) {logger.error({ error: prewarmError }, 'Prewarm failed, continuing without a warm cache')diff --git a/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts b/packages/server/worker/test/lib/execute/jobs/execute-flow.test.tsindex 8e275308061..2c169e0d7f7 100644--- a/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts+++ b/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts@@ -86,7 +86,7 @@ function makeMockContext(opts?: { resolveResult?: unknown, apiOverrides?: Recordresolve: vi.fn().mockResolvedValue(opts?.resolveResult ?? {kind: 'ready',- provision: { platformId: 'plat-1', pieces: [], codes: [], publicApiUrl: 'http://localhost:3000/api/', engineToken: 'test-token' },+ provision: { platformId: 'plat-1', pieces: [], codes: [], internalApiUrl: 'http://localhost:3000/api/', engineToken: 'test-token' },flowVersion: makeFlowVersion(),},),@@ -109,7 +109,7 @@ function makeMockContext(opts?: { resolveResult?: unknown, apiOverrides?: Recordruntime,workerIndex: 0,engineToken: 'test-token',- internalApiUrl: 'http://localhost:3000',+ internalApiUrl: 'http://localhost:3000/api/',publicApiUrl: 'http://localhost:4200',} as any}@@ -152,6 +152,18 @@ describe('executeFlowJob', () => {})})+ describe('piece bundle source', () => {+ it('resolves pieces against the worker internal API URL, not the public one', async () => {+ const ctx = makeMockContext()++ await executeFlowJob.execute(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN }))++ const resolveInput = ctx.resolver.resolve.mock.calls[0][0]+ expect(resolveInput.internalApiUrl).toBe('http://localhost:3000/api/')+ expect(resolveInput).not.toHaveProperty('publicApiUrl')+ })+ })+describe('RESUME validation', () => {it('still throws when logsFileId is missing for RESUME', async () => {const ctx = makeMockContext()