AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(worker): record run status and engine error on the job.execute event (#15960)
Amr Elmohamadye3e1cbebb6da
diff --git a/brain/knowledge/flows-execution/flow-runs.md b/brain/knowledge/flows-execution/flow-runs.mdindex fa8898fe14c..5476e851216 100644--- a/brain/knowledge/flows-execution/flow-runs.md+++ b/brain/knowledge/flows-execution/flow-runs.md@@ -54,7 +54,7 @@ A Flow Run records one execution of a specific flow version, from trigger to ter- **Failed-trigger payload** survives past BullMQ job completion only because `buildFailedTriggerContext` writes it into the trigger step's `output` slot.- **The trigger step's status IS the raw-vs-extracted discriminator for retry** — there is no separate field (a `payload` field was tried and removed as redundant). `FAILED` means "`output` holds a raw event, re-run `run()` on it" (`executeTrigger: true`); `SUCCEEDED` means "`output` is already the trigger's result, replay as-is" (`executeTrigger: false`). So any code that *fabricates* a trigger step without the engine having run — the `QUOTA_EXCEEDED` admission gate is the first — must pick the status from where its payload came: raw for sync webhooks, extracted for anything sourced from the worker RPC `submitPayloads` (which passes post-`TriggerHookType.RUN` output). Get it wrong on a polling trigger and retry re-polls against an already-advanced `lastPoll` cursor, so the run gets `undefined` or an unrelated newer item *and* silently consumes those fresh items' own runs.- **Big step outputs**: over 32 KB inline → stored as a `LogSliceRef` pointer to a `FLOW_RUN_LOG_SLICE` file (`outputType === SLICE`); missing backing file throws `ENTITY_NOT_FOUND` (loud retry failure). Step *inputs* over 2 KB (`AP_FLOW_RUN_LOG_INPUT_TRUNCATE_THRESHOLD_KB`) become a display-only truncation placeholder.-- **An INTERNAL_ERROR run does not record why it failed** — read the BullMQ job's `failedReason`, not the run. `reportFlowStatus` (`execute-flow.ts`) only forwards `logsFileId` when `data.logsFileId` is set, which is nil on every `BEGIN` run, so `engineRunCallbackService.uploadRunLog` skips persisting `internalError` into the log file. The run page and the log file both show nothing; the engine's actual error (plus its stderr) survives only in the job that `jobBroker.completeJob` moved to failed. On a dedicated worker group that job lives in `platform-<platformId>-jobs`, **not** `workerJobs` — pass `--queue` to `debug-failed-job.js` or it reports "job not found".+- **An INTERNAL_ERROR's cause lives in three places — check them in this order.** (1) **The run log.** `addToQueue` always assigns a `logsFileId`, `reportFlowStatus` forwards it with the `internalError`, and `engineRunCallbackService.uploadRunLog` writes the error into the log file, so a platform admin sees it through the runs table's **View error** button (`getPopulated`). On Cloud only `ENGINE`-source errors are kept (`internalErrorEnabled`); `WORKER`-source ones are dropped there. (2) **The worker's `job.execute` event** (since 2026-09): `reportFlowStatus` sets `flowRun.status`, `flowRun.willRetry`, `flowRun.environment`, `flowRun.internalErrorSource`, the failed step's `step.name` when known, and attaches the error, so `service:activepieces-worker @flowRun.id:<id>` shows the cause next to `flow.id`/`project.id`. `outcome` still means the worker's handling verdict and stays `success` for an engine-reported failure — filter on `flowRun.status`, not `outcome`. (3) **The failed BullMQ job's `failedReason`** (engine message plus stderr, set by `jobBroker.completeJob`). On a dedicated worker group that job lives in `platform-<platformId>-jobs`, **not** `workerJobs` — pass `--queue` to `debug-failed-job.js` or it reports "job not found". An earlier version of this note said `logsFileId` is nil on every BEGIN run and the run log shows nothing; that was wrong.- Retries only allowed on terminal states within `EXECUTION_DATA_RETENTION_DAYS`.- **Credit metering (Autumn)**: on terminal runs (paid editions), `onFinish` does two tryCatch-wrapped billing steps that never break run completion. (1) A PRODUCTION run not in `QUOTA_EXCEEDED` charges +1 apCredit via `billingProvider.trackCredits` with idempotency key `{runId}:run`. (2) `flowRunAiUsageTracker` pre-scans the flow version for `@activepieces/piece-ai` steps, extracts per-provider/model usage from step outputs (`flow-run-ai-usage-extractor` — recurses into loops, fetches `FLOW_RUN_LOG_SLICE` files, falls back to flow-version settings on `**REDACTED**` models), meters `Σ(messages × model credit weight) + toolCalls` to Autumn (`{runId}:ai`, plus `{runId}:appSumoAi` for the managed-ACTIVEPIECES AppSumo cap), then emits the `AI_USAGE_PER_RUN` PostHog event — the license key is only the PostHog distinctId, no longer a gate on metering.- **Credit gate is fail-open at admission**: the worker RPC `submitPayloads` checks `shouldBlockOnCredits` (blocks only when the platform is `billingEnforced` AND the cached balance is exhausted; CE default and Autumn-outage behavior is false). A blocked run is still admitted — as a `QUOTA_EXCEEDED` run with the trigger payload persisted in its log — so it stays retryable once credits return instead of being dropped. **`AP_EDITION=ee` skips the gate entirely** (`shouldBlockRunOnCredits` returns `false` before any provider call) so self-hosters pay no Redis/Autumn latency on admission — a temporary measure, see decision 000020.diff --git a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts b/packages/server/worker/src/lib/execute/jobs/execute-flow.tsindex 54acb8b64c7..176e7c74b96 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts@@ -1,6 +1,6 @@import { inspect } from 'node:util'import { ActivepiecesError, ErrorCode, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'-import { onCallService } from '@activepieces/server-utils'+import { onCallService, wideEvent } from '@activepieces/server-utils'import { BeginExecuteFlowOperation, EngineOperationType, EngineResponseStatus, ExecuteFlowJobData, ExecutionType, FailedStep, FlowActionType, FlowRunStatus, flowStructureUtil, FlowVersion, ResumeExecuteFlowOperation, RunInternalError, RunInternalErrorSource, WorkerJobType } from '@activepieces/shared'import { system, WorkerSystemProp } from '../../config/configs'import { workerSettings } from '../../config/worker-settings'@@ -189,6 +189,7 @@ function toInternalError(source: RunInternalErrorSource, error: unknown): RunIntasync function reportFlowStatus({ ctx, data, status, internalError, failedStep }: ReportFlowStatusParams): Promise<void> {const willRetry = status === FlowRunStatus.INTERNAL_ERROR && !ctx.lastAttempt+ recordRunOutcomeOnWideEvent({ status, internalError, willRetry, environment: data.environment, failedStepName: failedStep?.name })// A status report has no log file of its own; carry logsFileId only for an internalError the server may// persist into one (see uploadRunLog). Sending it on a plain status report would dangle flow_run.logsFileId.await ctx.apiClient.uploadRunLog({@@ -214,6 +215,20 @@ async function reportFlowStatus({ ctx, data, status, internalError, failedStep }}}+function recordRunOutcomeOnWideEvent({ status, internalError, willRetry, environment, failedStepName }: RecordRunOutcomeParams): void {+ wideEvent.set({ flowRun: { status, willRetry, environment } })+ if (!isNil(failedStepName)) {+ wideEvent.set({ step: { name: failedStepName } })+ }+ if (isNil(internalError)) {+ return+ }+ wideEvent.set({ flowRun: { internalErrorSource: internalError.source, ...spreadIfDefined('internalErrorCode', internalError.code) } })+ const error = new Error(internalError.message)+ error.stack = internalError.message+ wideEvent.error(error)+}+function isDedicatedWorker(): boolean {return !isNil(system.get(WorkerSystemProp.WORKER_GROUP_ID))}@@ -226,6 +241,14 @@ type ReportFlowStatusParams = {failedStep?: FailedStep}+type RecordRunOutcomeParams = {+ status: FlowRunStatus+ internalError?: RunInternalError+ willRetry: boolean+ environment: ExecuteFlowJobData['environment']+ failedStepName?: string+}+type FindStepOwningPieceParams = {flowVersion: FlowVersionpieceName: stringdiff --git a/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts b/packages/server/worker/test/lib/execute/jobs/execute-flow.test.tsindex cfc778434a6..8e275308061 100644--- a/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts+++ b/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts@@ -9,6 +9,7 @@ vi.mock('../../../../src/lib/config/worker-settings', () => ({},}))+import { createLogger, wideEvent } from '@activepieces/server-utils'import { executeFlowJob } from '../../../../src/lib/execute/jobs/execute-flow'import { JobResultKind } from '../../../../src/lib/execute/types'@@ -338,4 +339,59 @@ describe('executeFlowJob', () => {expect(reported).not.toHaveProperty('httpRequestId')})})++ describe('job.execute wide event', () => {+ const ENGINE_ERROR = 'EngineFileUploadError: Failed to upload engine file f-1: 403 Forbidden'++ async function runInsideJobEvent(ctx: ReturnType<typeof makeMockContext>, data: ExecuteFlowJobData) {+ const logger = createLogger({ event: 'job.execute', flowRun: { id: data.runId }, flow: { id: data.flowId } })+ await wideEvent.run({ logger, fn: () => executeFlowJob.execute(ctx, data) })+ return logger.getContext()+ }++ it('emits the run status and engine error next to the run ids', async () => {+ const ctx = makeMockContext()+ ctx.runtime.execute.mockResolvedValue({ status: EngineResponseStatus.INTERNAL_ERROR, error: ENGINE_ERROR })++ const event = await runInsideJobEvent(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN }))++ expect(event.flowRun).toEqual({ id: 'run-1', status: FlowRunStatus.INTERNAL_ERROR, willRetry: true, environment: RunEnvironment.PRODUCTION, internalErrorSource: 'ENGINE' })+ expect(event.flow).toEqual({ id: 'flow-1' })+ expect(event.error).toMatchObject({ message: ENGINE_ERROR })+ })++ it('keeps the original stack instead of pointing at the reporting helper', async () => {+ const ctx = makeMockContext()+ ctx.resolver.resolve.mockRejectedValue(new Error('resolver exploded'))+ const logger = createLogger({ event: 'job.execute', flowRun: { id: 'run-1' } })++ await expect(wideEvent.run({+ logger,+ fn: () => executeFlowJob.execute(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN })),+ })).rejects.toThrow('resolver exploded')++ const { error } = logger.getContext()+ expect(error).toMatchObject({ stack: expect.stringContaining('execute-flow.test.ts') })+ expect(error).not.toMatchObject({ stack: expect.stringContaining('src/lib/execute/jobs/execute-flow.ts') })+ })++ it('emits the status without an error for a user-caused FAILED run', async () => {+ const ctx = makeMockContext({ resolveResult: { kind: 'flow-not-found' } })++ const event = await runInsideJobEvent(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN }))++ expect(event.flowRun).toEqual({ id: 'run-1', status: FlowRunStatus.FAILED, willRetry: false, environment: RunEnvironment.PRODUCTION })+ expect(event.error).toBeUndefined()+ })++ it('emits the failed step name when the worker fails the run on a specific step', async () => {+ const failedStep = { name: 'step_1', displayName: 'HTTP', message: 'The piece @activepieces/[email protected] is not installed' }+ const ctx = makeMockContext({ resolveResult: { kind: 'disabled', failedStep } })++ const event = await runInsideJobEvent(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN }))++ expect(event.flowRun).toMatchObject({ status: FlowRunStatus.FAILED })+ expect(event.step).toEqual({ name: 'step_1' })+ })+ })})