Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(worker): record run status and engine error on the job.execute event (#15960)

Amr Elmohamadye3e1cbebb6da
diff --git a/brain/knowledge/flows-execution/flow-runs.md b/brain/knowledge/flows-execution/flow-runs.md
index fa8898fe14c..5476e851216 100644
--- a/brain/knowledge/flows-execution/flow-runs.md
+++ b/brain/knowledge/flows-execution/flow-runs.md
@@ -54,7 +54,7 @@ A Flow Run records one execution of a specific flow version, from trigger to ter
- **Failed-trigger payload** survives past BullMQ job completion only because `buildFailedTriggerContext` writes it into the trigger step's `output` slot.
- **The trigger step's status IS the raw-vs-extracted discriminator for retry** — there is no separate field (a `payload` field was tried and removed as redundant). `FAILED` means "`output` holds a raw event, re-run `run()` on it" (`executeTrigger: true`); `SUCCEEDED` means "`output` is already the trigger's result, replay as-is" (`executeTrigger: false`). So any code that *fabricates* a trigger step without the engine having run — the `QUOTA_EXCEEDED` admission gate is the first — must pick the status from where its payload came: raw for sync webhooks, extracted for anything sourced from the worker RPC `submitPayloads` (which passes post-`TriggerHookType.RUN` output). Get it wrong on a polling trigger and retry re-polls against an already-advanced `lastPoll` cursor, so the run gets `undefined` or an unrelated newer item *and* silently consumes those fresh items' own runs.
- **Big step outputs**: over 32 KB inline → stored as a `LogSliceRef` pointer to a `FLOW_RUN_LOG_SLICE` file (`outputType === SLICE`); missing backing file throws `ENTITY_NOT_FOUND` (loud retry failure). Step *inputs* over 2 KB (`AP_FLOW_RUN_LOG_INPUT_TRUNCATE_THRESHOLD_KB`) become a display-only truncation placeholder.
-- **An INTERNAL_ERROR run does not record why it failed** — read the BullMQ job's `failedReason`, not the run. `reportFlowStatus` (`execute-flow.ts`) only forwards `logsFileId` when `data.logsFileId` is set, which is nil on every `BEGIN` run, so `engineRunCallbackService.uploadRunLog` skips persisting `internalError` into the log file. The run page and the log file both show nothing; the engine's actual error (plus its stderr) survives only in the job that `jobBroker.completeJob` moved to failed. On a dedicated worker group that job lives in `platform-<platformId>-jobs`, **not** `workerJobs` — pass `--queue` to `debug-failed-job.js` or it reports "job not found".
+- **An INTERNAL_ERROR's cause lives in three places — check them in this order.** (1) **The run log.** `addToQueue` always assigns a `logsFileId`, `reportFlowStatus` forwards it with the `internalError`, and `engineRunCallbackService.uploadRunLog` writes the error into the log file, so a platform admin sees it through the runs table's **View error** button (`getPopulated`). On Cloud only `ENGINE`-source errors are kept (`internalErrorEnabled`); `WORKER`-source ones are dropped there. (2) **The worker's `job.execute` event** (since 2026-09): `reportFlowStatus` sets `flowRun.status`, `flowRun.willRetry`, `flowRun.environment`, `flowRun.internalErrorSource`, the failed step's `step.name` when known, and attaches the error, so `service:activepieces-worker @flowRun.id:<id>` shows the cause next to `flow.id`/`project.id`. `outcome` still means the worker's handling verdict and stays `success` for an engine-reported failure — filter on `flowRun.status`, not `outcome`. (3) **The failed BullMQ job's `failedReason`** (engine message plus stderr, set by `jobBroker.completeJob`). On a dedicated worker group that job lives in `platform-<platformId>-jobs`, **not** `workerJobs` — pass `--queue` to `debug-failed-job.js` or it reports "job not found". An earlier version of this note said `logsFileId` is nil on every BEGIN run and the run log shows nothing; that was wrong.
- Retries only allowed on terminal states within `EXECUTION_DATA_RETENTION_DAYS`.
- **Credit metering (Autumn)**: on terminal runs (paid editions), `onFinish` does two tryCatch-wrapped billing steps that never break run completion. (1) A PRODUCTION run not in `QUOTA_EXCEEDED` charges +1 apCredit via `billingProvider.trackCredits` with idempotency key `{runId}:run`. (2) `flowRunAiUsageTracker` pre-scans the flow version for `@activepieces/piece-ai` steps, extracts per-provider/model usage from step outputs (`flow-run-ai-usage-extractor` — recurses into loops, fetches `FLOW_RUN_LOG_SLICE` files, falls back to flow-version settings on `**REDACTED**` models), meters `Σ(messages × model credit weight) + toolCalls` to Autumn (`{runId}:ai`, plus `{runId}:appSumoAi` for the managed-ACTIVEPIECES AppSumo cap), then emits the `AI_USAGE_PER_RUN` PostHog event — the license key is only the PostHog distinctId, no longer a gate on metering.
- **Credit gate is fail-open at admission**: the worker RPC `submitPayloads` checks `shouldBlockOnCredits` (blocks only when the platform is `billingEnforced` AND the cached balance is exhausted; CE default and Autumn-outage behavior is false). A blocked run is still admitted — as a `QUOTA_EXCEEDED` run with the trigger payload persisted in its log — so it stays retryable once credits return instead of being dropped. **`AP_EDITION=ee` skips the gate entirely** (`shouldBlockRunOnCredits` returns `false` before any provider call) so self-hosters pay no Redis/Autumn latency on admission — a temporary measure, see decision 000020.
diff --git a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
index 54acb8b64c7..176e7c74b96 100644
--- a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
+++ b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
@@ -1,6 +1,6 @@
import { inspect } from 'node:util'
import { ActivepiecesError, ErrorCode, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'
-import { onCallService } from '@activepieces/server-utils'
+import { onCallService, wideEvent } from '@activepieces/server-utils'
import { BeginExecuteFlowOperation, EngineOperationType, EngineResponseStatus, ExecuteFlowJobData, ExecutionType, FailedStep, FlowActionType, FlowRunStatus, flowStructureUtil, FlowVersion, ResumeExecuteFlowOperation, RunInternalError, RunInternalErrorSource, WorkerJobType } from '@activepieces/shared'
import { system, WorkerSystemProp } from '../../config/configs'
import { workerSettings } from '../../config/worker-settings'
@@ -189,6 +189,7 @@ function toInternalError(source: RunInternalErrorSource, error: unknown): RunInt
async function reportFlowStatus({ ctx, data, status, internalError, failedStep }: ReportFlowStatusParams): Promise<void> {
const willRetry = status === FlowRunStatus.INTERNAL_ERROR && !ctx.lastAttempt
+ recordRunOutcomeOnWideEvent({ status, internalError, willRetry, environment: data.environment, failedStepName: failedStep?.name })
// A status report has no log file of its own; carry logsFileId only for an internalError the server may
// persist into one (see uploadRunLog). Sending it on a plain status report would dangle flow_run.logsFileId.
await ctx.apiClient.uploadRunLog({
@@ -214,6 +215,20 @@ async function reportFlowStatus({ ctx, data, status, internalError, failedStep }
}
}
+function recordRunOutcomeOnWideEvent({ status, internalError, willRetry, environment, failedStepName }: RecordRunOutcomeParams): void {
+ wideEvent.set({ flowRun: { status, willRetry, environment } })
+ if (!isNil(failedStepName)) {
+ wideEvent.set({ step: { name: failedStepName } })
+ }
+ if (isNil(internalError)) {
+ return
+ }
+ wideEvent.set({ flowRun: { internalErrorSource: internalError.source, ...spreadIfDefined('internalErrorCode', internalError.code) } })
+ const error = new Error(internalError.message)
+ error.stack = internalError.message
+ wideEvent.error(error)
+}
+
function isDedicatedWorker(): boolean {
return !isNil(system.get(WorkerSystemProp.WORKER_GROUP_ID))
}
@@ -226,6 +241,14 @@ type ReportFlowStatusParams = {
failedStep?: FailedStep
}
+type RecordRunOutcomeParams = {
+ status: FlowRunStatus
+ internalError?: RunInternalError
+ willRetry: boolean
+ environment: ExecuteFlowJobData['environment']
+ failedStepName?: string
+}
+
type FindStepOwningPieceParams = {
flowVersion: FlowVersion
pieceName: string
diff --git a/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts b/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts
index cfc778434a6..8e275308061 100644
--- a/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts
+++ b/packages/server/worker/test/lib/execute/jobs/execute-flow.test.ts
@@ -9,6 +9,7 @@ vi.mock('../../../../src/lib/config/worker-settings', () => ({
},
}))
+import { createLogger, wideEvent } from '@activepieces/server-utils'
import { executeFlowJob } from '../../../../src/lib/execute/jobs/execute-flow'
import { JobResultKind } from '../../../../src/lib/execute/types'
@@ -338,4 +339,59 @@ describe('executeFlowJob', () => {
expect(reported).not.toHaveProperty('httpRequestId')
})
})
+
+ describe('job.execute wide event', () => {
+ const ENGINE_ERROR = 'EngineFileUploadError: Failed to upload engine file f-1: 403 Forbidden'
+
+ async function runInsideJobEvent(ctx: ReturnType<typeof makeMockContext>, data: ExecuteFlowJobData) {
+ const logger = createLogger({ event: 'job.execute', flowRun: { id: data.runId }, flow: { id: data.flowId } })
+ await wideEvent.run({ logger, fn: () => executeFlowJob.execute(ctx, data) })
+ return logger.getContext()
+ }
+
+ it('emits the run status and engine error next to the run ids', async () => {
+ const ctx = makeMockContext()
+ ctx.runtime.execute.mockResolvedValue({ status: EngineResponseStatus.INTERNAL_ERROR, error: ENGINE_ERROR })
+
+ const event = await runInsideJobEvent(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN }))
+
+ expect(event.flowRun).toEqual({ id: 'run-1', status: FlowRunStatus.INTERNAL_ERROR, willRetry: true, environment: RunEnvironment.PRODUCTION, internalErrorSource: 'ENGINE' })
+ expect(event.flow).toEqual({ id: 'flow-1' })
+ expect(event.error).toMatchObject({ message: ENGINE_ERROR })
+ })
+
+ it('keeps the original stack instead of pointing at the reporting helper', async () => {
+ const ctx = makeMockContext()
+ ctx.resolver.resolve.mockRejectedValue(new Error('resolver exploded'))
+ const logger = createLogger({ event: 'job.execute', flowRun: { id: 'run-1' } })
+
+ await expect(wideEvent.run({
+ logger,
+ fn: () => executeFlowJob.execute(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN })),
+ })).rejects.toThrow('resolver exploded')
+
+ const { error } = logger.getContext()
+ expect(error).toMatchObject({ stack: expect.stringContaining('execute-flow.test.ts') })
+ expect(error).not.toMatchObject({ stack: expect.stringContaining('src/lib/execute/jobs/execute-flow.ts') })
+ })
+
+ it('emits the status without an error for a user-caused FAILED run', async () => {
+ const ctx = makeMockContext({ resolveResult: { kind: 'flow-not-found' } })
+
+ const event = await runInsideJobEvent(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN }))
+
+ expect(event.flowRun).toEqual({ id: 'run-1', status: FlowRunStatus.FAILED, willRetry: false, environment: RunEnvironment.PRODUCTION })
+ expect(event.error).toBeUndefined()
+ })
+
+ it('emits the failed step name when the worker fails the run on a specific step', async () => {
+ const failedStep = { name: 'step_1', displayName: 'HTTP', message: 'The piece @activepieces/[email protected] is not installed' }
+ const ctx = makeMockContext({ resolveResult: { kind: 'disabled', failedStep } })
+
+ const event = await runInsideJobEvent(ctx, makeResumeJobData({ executionType: ExecutionType.BEGIN }))
+
+ expect(event.flowRun).toMatchObject({ status: FlowRunStatus.FAILED })
+ expect(event.step).toEqual({ name: 'step_1' })
+ })
+ })
})