Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(pieces): surface engine errors from POST /v1/pieces/options (#14711)

Amr Elmohamady41635df15a82
diff --git a/brain/knowledge/execution-runtime/workers.md b/brain/knowledge/execution-runtime/workers.md
index 2d23b2c2a9e..873e91c8b39 100644
--- a/brain/knowledge/execution-runtime/workers.md
+++ b/brain/knowledge/execution-runtime/workers.md
@@ -63,6 +63,7 @@ The deep `Resolver`/`Runtime` concurrency and bundle-caching model lives on the
- **Adding any `WorkerJobType` means five edits, and the compiler only catches three.** In `core/execution/.../job-data.ts`: the enum, the `getDefaultJobPriority` switch (exhaustive with no `default`, so this one is a hard compile error), and the `JobData` union. Then `OneTimeJobAddParams` in `job-queue.ts`, or `jobQueue.add` rejects the payload. Then the lazy handler entry in `worker/.../execute/job-registry.ts` — **that last one is the silent miss**: everything compiles and the job enqueues, it just never runs. Also give the job data a `projectId`, `null` if it has none, because `job-broker.ts` reads `migratedData.projectId as string` for every dequeued job and a member without the property breaks that file and `job-data-migrations.ts` rather than your own. `ExecuteAgentRunJobData` is the shape to copy: nullable `projectId`, and it already flows through that cast.
- **A new user-interaction `WorkerJobType` must be added to `USER_INTERACTION_JOB_TYPES`** in `packages/server/api/src/app/workers/job-queue/job-queue.ts`. `jobBroker.completeJob` only publishes the engine response back to the waiting webserver for job types in that set — miss it and the caller hangs to `WATCHER_SAFETY_TIMEOUT_MS` (5 min) with no error. `submitAndWaitForResponse` has only that backstop, so any *best-effort* caller must additionally cap itself (`Promise.race` with a short timeout); the losing engine job still runs to completion, so the cap buys back user latency, not fleet capacity.
+- **Check `status` on every `submitAndWaitForResponse` result before reading `.response`.** A failed job resolves (it does not throw) with `status: INTERNAL_ERROR`, `response: undefined` and the cause in `error`; a timeout gives `status: TIMEOUT`, `response: {}`. Returning `.response` blindly sends HTTP 200 with an empty body and drops the error (GIT-1695, `/v1/pieces/options`). Throw `ENGINE_OPERATION_FAILURE` with `error` as the message.
- **System-job `No handler` = the worker runs the wrong edition.** The single shared `system-job-queue` is consumed by whichever app instance runs `startWorker()`, and EE handlers only register in the CLOUD/ENTERPRISE branches of the edition switch. A worker on a different edition than the instance that *scheduled* the job throws `No handler for job <name>` every tick. Seen July 2026: ~14.6k failures, ~99% `chat-stale-sweep`, because the worker defaulted to community (`AP_EDITION` unset) inside a cloud deployment — CE jobs like `file-cleanup-trigger` ran fine on the same worker, every EE-scheduled job failed identically. Fix on the deployment (`AP_EDITION=cloud`), not by registering EE handlers in CE. The count looks huge because `removeOnComplete: true` hides successes and `removeOnFail` has an age cap but no count cap.
- **When debugging a `MEMORY_LIMIT_EXCEEDED` run in the logs, the worker's `job.execute` line lies — read the app's `completeJob` line instead.** An OOM-killed `EXECUTE_FLOW` still logs `outcome:"success"` in the worker `job.execute` event, because that field is the *worker's* verdict on handling the job, not the flow-run status (the memory path returns `FIRE_AND_FORGET` with `MEMORY_ISSUE`, which the worker treats as handled). The authoritative signal is the app-side line `[workerRpc#completeJob] Job completed by worker job.id=<runId> status:"MEMORY_ISSUE"`. A fast OOM (sandbox dies ~4 s in, deterministic across `FROM_FAILED_STEP` retries) is a per-run heap spike from the flow's own workload — a large file/CSV read into memory or a fat piece output — not infra flakiness; on Cloud the ceiling is fixed so the fix is flow-side. The raw `JavaScript heap out of memory` stderr is captured into the `ActivepiecesError`, **not** shipped to ClickHouse as its own log line, and the per-step run-log body is ZSTD in S3 — to pin the failing step use `debug-failed-job.js --run <id>` on the DevOps box, not the log store. **Localize the death by how many steps the run log actually contains:** a `MEMORY_LIMIT_EXCEEDED` run whose decompressed log holds only the `trigger` step (and `stepsCount: 0`, `failedStep: null`) OOM'd inside the *first action*, before its output could be flushed — the failing step is step_1/step_2, not whatever the flow does later. Classic offender: a `file-helper read_file (text)` → `csv convert_csv_to_json` → grouping-Code chain on a large CSV, which holds the full text, the full JSON row array, and the group as **three coexisting copies** — and because every step output is retained and serialized for the run log, splitting or slimming those intermediates (ideally collapsing read+parse+group into one Code step that returns only the grouped result) is the flow-side fix, since the Cloud sandbox cap can't be raised.
- **The Deno code sandbox (`SANDBOX_PROCESS`) allows tmp-dir read+write and nothing else on the fs, and no subprocess — and that barely covers real Code steps.** `denoCodeSandbox` maps a `DenoPermission[]` to Deno CLI flags (`deno.ts` → `toPermissionFlags`). `SANDBOX_PROCESS`/`UNSANDBOXED` grant `[WRITE_TMP, READ_TMP, NET, ENV, SYS]`, emitting `--allow-net --allow-env --allow-sys --allow-write=$TMPDIR --allow-read=$TMPDIR`. So `child_process` exec/spawn is denied (no `--allow-run`), and any `fs` read/write **outside** `$TMPDIR` is denied. What still works: importing `fs` (Deno checks at the syscall, not the import), npm-heavy code (esbuild bundles deps into one `index.js`, `node_modules` is deleted, and the engine pipes the source to Deno over **stdin** — so no read grant is needed to load user code or deps), network calls, and reads/writes that resolve inside `$TMPDIR`. A production scan of Code steps matching shell/fs patterns found ~93% were false positives (mostly `regex.exec(...)`, not `child_process.exec`); of the genuine breakers, almost all write or read at a **relative/input-supplied path**, not `os.tmpdir()` — so the tmp grants (and the `READ_TMP` fix that changed it from `--allow-write` to the correct `--allow-read`) rescue only a handful. Note relative paths resolve against `cwd` = the compiled step dir, not tmp. See `code-sandbox.ts` (per-mode grants) and `deno-code-sandbox.ts`. The deno binary is located **only** via `AP_DENO_PATH` (`deno.ts` → `resolveDenoPath` throws if unset) — the `deno` npm package is no longer a runtime dependency (kept only as an engine test devDependency), so self-hosters must point `AP_DENO_PATH` at a deno binary in the image. For local dev, `npm start` (`tools/setup-dev.js`) installs deno globally if missing and idempotently writes `AP_DENO_PATH` plus `AP_SANDBOX_PROPAGATED_ENV_VARS=AP_DENO_PATH` into `.env.dev` (both api and worker bootstraps load that file). Careful with the propagation var name: it is `AP_SANDBOX_PROPAGATED_ENV_VARS` (from `AppSystemProp.SANDBOX_PROPAGATED_ENV_VARS`), not `AP_PROPAGATED_ENV_VARS`.
diff --git a/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts b/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts
index 8d4760fbb29..bac6258ee23 100644
--- a/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts
+++ b/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts
@@ -1,6 +1,6 @@
import { ActivepiecesError, ErrorCode, isEmpty, isNil, LocalesEnum } from '@activepieces/core-utils'
import { PieceMetadataModel, PieceMetadataModelSummary } from '@activepieces/pieces-framework'
-import { ALL_PRINCIPAL_TYPES, ApEdition, EngineResponse, GetPieceRequestParams, GetPieceRequestQuery, GetPieceRequestWithScopeParams, ListPiecesRequestQuery, PieceAudienceFilter, PieceCategory, PieceOptionRequest, Principal, PrincipalType, RegistryPiecesRequestQuery, SampleDataFileType, WorkerJobType } from '@activepieces/shared'
+import { ALL_PRINCIPAL_TYPES, ApEdition, EngineResponse, EngineResponseStatus, GetPieceRequestParams, GetPieceRequestQuery, GetPieceRequestWithScopeParams, ListPiecesRequestQuery, PieceAudienceFilter, PieceCategory, PieceOptionRequest, Principal, PrincipalType, RegistryPiecesRequestQuery, SampleDataFileType, WorkerJobType } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'
import { StatusCodes } from 'http-status-codes'
@@ -144,7 +144,7 @@ const basePiecesController: FastifyPluginAsyncZod = async (app) => {
type: SampleDataFileType.OUTPUT,
referencedBy: req.body.input,
})
- const { response } = await userInteractionWatcher.submitAndWaitForResponse<EngineResponse<unknown>>({
+ const engineResponse = await userInteractionWatcher.submitAndWaitForResponse<EngineResponse<unknown>>({
jobType: WorkerJobType.EXECUTE_PROPERTY,
platformId: platform.id,
projectId,
@@ -156,7 +156,18 @@ const basePiecesController: FastifyPluginAsyncZod = async (app) => {
searchValue: req.body.searchValue,
piece: await getPiecePackageWithoutArchive(req.log, platform.id, req.body),
}, req.log)
- return response
+ if (engineResponse.status !== EngineResponseStatus.OK) {
+ throw new ActivepiecesError({
+ code: ErrorCode.ENGINE_OPERATION_FAILURE,
+ params: {
+ message: engineResponse.error ?? (engineResponse.status === EngineResponseStatus.TIMEOUT
+ ? 'Engine timed out while resolving property options'
+ : `Engine failed while resolving property options (${engineResponse.status})`),
+ context: { status: engineResponse.status },
+ },
+ })
+ }
+ return engineResponse.response
},
)
diff --git a/packages/server/api/test/integration/ce/pieces/piece-options-error.test.ts b/packages/server/api/test/integration/ce/pieces/piece-options-error.test.ts
new file mode 100644
index 00000000000..854ea0abdbd
--- /dev/null
+++ b/packages/server/api/test/integration/ce/pieces/piece-options-error.test.ts
@@ -0,0 +1,148 @@
+import { apDayjs } from '@activepieces/server-utils'
+import {
+ EngineResponseStatus,
+ FlowTriggerType,
+ FlowVersionState,
+ PackageType,
+ PieceType,
+ PrincipalType,
+} from '@activepieces/shared'
+import { FastifyInstance } from 'fastify'
+import { StatusCodes } from 'http-status-codes'
+import { vi } from 'vitest'
+import { databaseConnection } from '../../../../src/app/database/database-connection'
+import { userInteractionWatcher } from '../../../../src/app/workers/user-interaction-watcher'
+import { generateMockToken } from '../../../helpers/auth'
+import { db } from '../../../helpers/db'
+import {
+ createMockFlow,
+ createMockFlowVersion,
+ createMockPieceMetadata,
+ mockAndSaveBasicSetup,
+} from '../../../helpers/mocks'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+
+let app: FastifyInstance
+
+beforeAll(async () => {
+ app = await setupTestEnvironment()
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+describe('POST /v1/pieces/options — engine failure propagation', () => {
+ afterEach(() => {
+ vi.restoreAllMocks()
+ })
+
+ const setupCallable = async () => {
+ const { mockPlatform, mockProject, mockOwner } = await mockAndSaveBasicSetup()
+
+ const mockFlow = createMockFlow({ projectId: mockProject.id })
+ await db.save('flow', mockFlow)
+
+ const mockFlowVersion = createMockFlowVersion({
+ flowId: mockFlow.id,
+ state: FlowVersionState.DRAFT,
+ trigger: {
+ type: FlowTriggerType.PIECE,
+ name: 'trigger',
+ displayName: 'Catch Webhook',
+ settings: {
+ pieceName: '@activepieces/piece-webhook',
+ pieceVersion: '~0.1.29',
+ triggerName: 'catch_webhook',
+ input: { authType: 'basic' },
+ propertySettings: {},
+ },
+ valid: true,
+ lastUpdatedDate: apDayjs().toISOString(),
+ },
+ })
+ await db.save('flow_version', mockFlowVersion)
+
+ const mockPiece = createMockPieceMetadata({
+ name: '@activepieces/piece-webhook',
+ version: '0.1.29',
+ platformId: undefined,
+ packageType: PackageType.REGISTRY,
+ pieceType: PieceType.OFFICIAL,
+ })
+ await databaseConnection().getRepository('piece_metadata').save(mockPiece)
+
+ const token = await generateMockToken({
+ id: mockOwner.id,
+ type: PrincipalType.USER,
+ platform: { id: mockPlatform.id },
+ })
+
+ return { mockProject, mockFlow, mockFlowVersion, token }
+ }
+
+ const requestBody = (ids: { projectId: string, flowId: string, flowVersionId: string }) => ({
+ projectId: ids.projectId,
+ flowId: ids.flowId,
+ flowVersionId: ids.flowVersionId,
+ pieceName: '@activepieces/piece-webhook',
+ pieceVersion: '~0.1.29',
+ actionOrTriggerName: 'catch_webhook',
+ propertyName: 'authFields',
+ input: { authType: 'basic' },
+ })
+
+ it('returns ENGINE_OPERATION_FAILURE + engine errorMessage when the property job fails', async () => {
+ const { mockProject, mockFlow, mockFlowVersion, token } = await setupCallable()
+
+ vi.spyOn(userInteractionWatcher, 'submitAndWaitForResponse').mockResolvedValue({
+ status: EngineResponseStatus.INTERNAL_ERROR,
+ response: undefined,
+ error: 'Failed to fetch piece bundle @activepieces/[email protected]: 404 Not Found',
+ })
+
+ const response = await app.inject({
+ method: 'POST',
+ url: '/api/v1/pieces/options',
+ headers: { authorization: `Bearer ${token}` },
+ body: requestBody({
+ projectId: mockProject.id,
+ flowId: mockFlow.id,
+ flowVersionId: mockFlowVersion.id,
+ }),
+ })
+
+ expect(response.statusCode).toBe(StatusCodes.BAD_REQUEST)
+ const body = response.json()
+ expect(body.code).toBe('ENGINE_OPERATION_FAILURE')
+ expect(body.params.message).toContain('404 Not Found')
+ expect(body.params.context.status).toBe(EngineResponseStatus.INTERNAL_ERROR)
+ })
+
+ it('returns a synthesized message on TIMEOUT (no engine errorMessage)', async () => {
+ const { mockProject, mockFlow, mockFlowVersion, token } = await setupCallable()
+
+ vi.spyOn(userInteractionWatcher, 'submitAndWaitForResponse').mockResolvedValue({
+ status: EngineResponseStatus.TIMEOUT,
+ response: {},
+ error: undefined,
+ })
+
+ const response = await app.inject({
+ method: 'POST',
+ url: '/api/v1/pieces/options',
+ headers: { authorization: `Bearer ${token}` },
+ body: requestBody({
+ projectId: mockProject.id,
+ flowId: mockFlow.id,
+ flowVersionId: mockFlowVersion.id,
+ }),
+ })
+
+ expect(response.statusCode).toBe(StatusCodes.BAD_REQUEST)
+ const body = response.json()
+ expect(body.code).toBe('ENGINE_OPERATION_FAILURE')
+ expect(body.params.message).toMatch(/timed out/i)
+ expect(body.params.context.status).toBe(EngineResponseStatus.TIMEOUT)
+ })
+})