AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(subflows): keep the calling flow waiting when a subflow crashes and is retried (#15642)
Hazem Adel7cccf090a105
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 01f06bd4c0a..3ceca4b720b 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -152,7 +152,7 @@ jobs:set -euo pipefailpids=()- npx turbo run test --filter=@activepieces/engine --filter=@activepieces/shared --filter=@activepieces/sandbox --filter=@activepieces/ai-providers --filter=@activepieces/core-execution --filter=@activepieces/core-piece-types --filter=@activepieces/server-utils --filter=@activepieces/pieces-framework --filter=web --filter=worker &+ npx turbo run test --filter=@activepieces/engine --filter=@activepieces/shared --filter=@activepieces/sandbox --filter=@activepieces/ai-providers --filter=@activepieces/core-execution --filter=@activepieces/core-piece-types --filter=@activepieces/server-utils --filter=@activepieces/pieces-framework --filter=@activepieces/piece-subflows --filter=web --filter=worker &pids+=($!)npx turbo run test-ce test-ee test-cloud check-migrations --filter=api &diff --git a/brain/knowledge/flows-execution/subflows.md b/brain/knowledge/flows-execution/subflows.mdindex f7c25a25031..9cb4243ab71 100644--- a/brain/knowledge/flows-execution/subflows.md+++ b/brain/knowledge/flows-execution/subflows.md@@ -31,6 +31,9 @@ No server entity of its own — subflows are ordinary flows plus two conventions- **Fan-out is not Call Flow.** Wait-for-response is incompatible with many batches inside one step; only Call Flow can wait.- **"Retry on failure" is hidden on the fan-out action, on purpose.** The framework default (`action.ts`) is `defaultValue: false` but *visible*, and step retry re-runs `run()` from the top — for a fire-and-forget fan-out that means re-reading the file from row 0 and re-dispatching every batch that already landed. A failure at batch 900 would duplicate 900 batches. So the action declares `retryOnFailure: { defaultValue: false, hide: true }`; `continueOnFailure` stays visible. `Call Flow` leaves both visible because one invocation is safe to retry — but with wait-for-response, a retry after the subflow has answered `status: 'error'` only rethrows the stored resume payload, so it costs ~28s of backoff and changes nothing. It must never re-call the subflow: that is the runaway loop of GIT-1712, see the step-retry gotchas in [flow-runs](flow-runs.md).- **`batchSize` is capped at 10,000, because "streaming bounds memory" only holds for sane batch sizes.** Peak memory is roughly `(MAX_IN_FLIGHT + 1) × batchSize × rowSize`, so an unbounded `batchSize` defeats the whole design. The cap that bites first in practice is not memory but the webhook `bodyLimit` (`max(AP_MAX_FILE_SIZE_MB + 4, AP_MAX_FLOW_RUN_LOG_SIZE_MB + 4, 25)` MB, 54MB by default) — a wide-row file will 413 well before 10,000 rows.+- **A child's `INTERNAL_ERROR` is not the end of that run.** `EXECUTE_FLOW` jobs carry `attempts: 2`, and the retry reuses the same job id — so the same run id flips `INTERNAL_ERROR → RUNNING → SUCCEEDED` and attempt 1 leaves no row behind. Anything that treats a child status as terminal must first ask whether the queue still owes an attempt: the worker stamps `willRetry` on its run-log upload (from `ConsumeJobRequest.lastAttempt`, computed in `job-broker.ts` from `attemptsMade` vs `opts.attempts`), and `childRunFailsParent` in `flow-runs-queue.ts` feeds it to `isFlowRunStateTerminal` as `ignoreInternalError` — that existing flag already meant "INTERNAL_ERROR is not final yet", so `willRetry` is the fact it was approximating. Because the switch consults it only in the `INTERNAL_ERROR` branch, a stale flag cannot suppress a genuine `FAILED`. The worker must keep sending `willRetry: false` explicitly: the metadata payload is a merged Redis hash, so omitting it would let a `true` from attempt 1 survive into the final attempt and strand the parent forever. Before that flag existed (GIT-1906) a crashed child failed its waiting parent in ~100ms, the retry then succeeded and its `Respond` callback hit an already-FAILED parent, which answers `200` and drops the payload silently.+- **A run whose parent is blocked on it retries in 30s, not the queue default of 8 minutes.** `addToQueue` passes `retryBackoffMs` for any run with `failParentOnFailure` and a `parentRunId`, so the override rides on that job's own opts. Do NOT reach for `defaultJobOptions` in `job-queue.ts` to tune this — it is shared by webhooks, triggers, agents and polling, so a change there is never subflow-scoped.+- **A discarded `Respond` is loud; a redelivered one is not.** `Respond` fails its step when the resume route answers `discarded: true`. That flag is **not** the route's `stale` flag: `stale` only says the waitpoint is gone, which is equally what a *successful* delivery looks like on retry — `Respond` posts with `retries: 10` and `fetch-http-client` retries on any thrown error, so a response lost on the way back is re-sent and finds its own waitpoint consumed. Nor can a run status settle it: `INTERNAL_ERROR`, `SUCCEEDED` and `FAILED` are all reachable *after* a successful delivery, and the first retry backs off only 1s. So the route keeps a positive record instead — `waitpoint_consumed:<id>` (1h TTL, comfortably past the ~5min retry window), and `discarded` is `stale && !evidence`. The write sits **inside** the consuming transaction, as the **first** step of `onReady`, and throws rather than logging — so a marker that cannot be written aborts before `enqueueResume` commits a BullMQ job the rollback cannot take back. Redis commits independently of Postgres, so ordering alone cannot make the two consistent; the read side closes the gap instead. **The waitpoint row is the authority**: still `PENDING` means nothing ever accepted the response, whatever the marker says, so an orphaned marker left by a rollback is ignored. Only once the row is gone does the marker speak, distinguishing a consumed waitpoint from one `onFinish` cleaned up. The `RUNNING`/`QUEUED` branch writes no marker — `complete()` leaves the row as `COMPLETED`, which is durable evidence in its own right, so `waitpointWasConsumed` falls back to it. The one place that must also write a marker is the GIT-1594 race block, which *deletes* that `COMPLETED` row: record before destroying the only evidence, or a later redelivery false-fails. Both reads fail safe: a Redis error reports *delivered*, because falsely failing a working flow is worse than a missed warning. HTTP stays `200`, and the legacy `/requests/:requestId` fallback has no waitpoint id so it carries no flag.- The dropdown lists published flows carrying a Callable Flow trigger and labels disabled ones `(inactive)`; streaming to a disabled flow throws before the first request.### Editionsdiff --git a/bun.lock b/bun.lockindex dc0039725a5..0de9eb9fdae 100644--- a/bun.lock+++ b/bun.lock@@ -892,7 +892,7 @@},"packages/pieces/community/aws-bedrock": {"name": "@activepieces/piece-aws-bedrock",- "version": "0.3.0",+ "version": "0.3.1","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",@@ -6969,7 +6969,7 @@},"packages/pieces/community/pinterest": {"name": "@activepieces/piece-pinterest",- "version": "0.1.8",+ "version": "0.1.9","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",@@ -8381,7 +8381,7 @@},"packages/pieces/community/slack": {"name": "@activepieces/piece-slack",- "version": "0.19.0",+ "version": "0.20.0","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",@@ -10065,7 +10065,7 @@},"packages/pieces/community/zagomail": {"name": "@activepieces/piece-zagomail",- "version": "0.1.6",+ "version": "0.1.7","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",@@ -10609,7 +10609,7 @@},"packages/pieces/core/subflows": {"name": "@activepieces/piece-subflows",- "version": "0.6.4",+ "version": "0.7.0","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",diff --git a/packages/core/execution/src/lib/engine/requests.ts b/packages/core/execution/src/lib/engine/requests.tsindex 3e1faf8ef2f..7fb8f5a8187 100644--- a/packages/core/execution/src/lib/engine/requests.ts+++ b/packages/core/execution/src/lib/engine/requests.ts@@ -29,6 +29,7 @@ export const UploadRunLogsRequest = z.object({runMs: z.number().optional(),workerHandlerId: z.string().optional(),httpRequestId: z.string().optional(),+ willRetry: z.boolean().optional(),})export type UploadRunLogsRequest = z.infer<typeof UploadRunLogsRequest>diff --git a/packages/core/execution/src/lib/workers/index.ts b/packages/core/execution/src/lib/workers/index.tsindex e21bf04309d..923a096dd3d 100644--- a/packages/core/execution/src/lib/workers/index.ts+++ b/packages/core/execution/src/lib/workers/index.ts@@ -82,6 +82,7 @@ export const ConsumeJobRequest = z.object({jobId: z.string(),jobData: JobData,attempsStarted: z.number(),+ lastAttempt: z.boolean().optional(),engineToken: z.string(),token: z.string(),queueName: z.string(),diff --git a/packages/pieces/core/subflows/package.json b/packages/pieces/core/subflows/package.jsonindex 2e02225bc77..dfe6c9c6630 100644--- a/packages/pieces/core/subflows/package.json+++ b/packages/pieces/core/subflows/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/piece-subflows",- "version": "0.6.4",+ "version": "0.7.0","main": "./dist/src/index.js","types": "./dist/src/index.d.ts","scripts": {diff --git a/packages/pieces/core/subflows/src/lib/actions/respond.ts b/packages/pieces/core/subflows/src/lib/actions/respond.tsindex a373192114e..95fd52362cf 100644--- a/packages/pieces/core/subflows/src/lib/actions/respond.ts+++ b/packages/pieces/core/subflows/src/lib/actions/respond.ts@@ -1,6 +1,5 @@import { DynamicPropsValue, PieceAuth, Property, StoreScope, createAction } from '@activepieces/pieces-framework';-import { callableFlowKey, CallableFlowResponse, MOCK_CALLBACK_IN_TEST_FLOW_URL } from '../common';-import { httpClient, HttpMethod } from '@activepieces/pieces-common';+import { callableFlowKey, deliverSubflowResponse, MOCK_CALLBACK_IN_TEST_FLOW_URL } from '../common';import { isNil } from '@activepieces/pieces-framework';export const response = createAction({@@ -62,15 +61,7 @@ export const response = createAction({const callbackUrl = await context.store.get<string>(callableFlowKey(context.run.id), StoreScope.FLOW);const isNotTestFlow = callbackUrl !== MOCK_CALLBACK_IN_TEST_FLOW_URL;if (isNotTestFlow && !isNil(callbackUrl)) {- await httpClient.sendRequest<CallableFlowResponse>({- method: HttpMethod.POST,- url: callbackUrl,- body: {- status: 'success',- data: response- },- retries: 10,- });+ await deliverSubflowResponse({ callbackUrl, response });}return response;},diff --git a/packages/pieces/core/subflows/src/lib/common.ts b/packages/pieces/core/subflows/src/lib/common.tsindex 109c13d4f5d..ad10faf3aa4 100644--- a/packages/pieces/core/subflows/src/lib/common.ts+++ b/packages/pieces/core/subflows/src/lib/common.ts@@ -134,6 +134,26 @@ export async function dispatchToSubflow({return response.body;}+export async function deliverSubflowResponse({+ callbackUrl,+ response,+}: DeliverSubflowResponseParams): Promise<void> {+ const acknowledgement = await httpClient.sendRequest<SubflowCallbackAck>({+ method: HttpMethod.POST,+ url: callbackUrl,+ body: {+ status: 'success',+ data: response,+ },+ retries: 10,+ });+ if (acknowledgement.body?.discarded === true) {+ throw new Error(+ 'The calling flow was no longer waiting for a response, so this response was discarded. Open the calling run to see why it stopped waiting.'+ );+ }+}+type ListParams = {flowsContext: FlowsContext,params?: ListFlowsContextParams@@ -147,4 +167,14 @@ type DispatchToSubflowParams = {data: unknown;callbackUrl?: string;retries?: number;-}\ No newline at end of file+}++type DeliverSubflowResponseParams = {+ callbackUrl: string;+ response: unknown;+}++type SubflowCallbackAck = {+ message?: string;+ discarded?: boolean;+}diff --git a/packages/pieces/core/subflows/test/deliver-subflow-response.test.ts b/packages/pieces/core/subflows/test/deliver-subflow-response.test.tsnew file mode 100644index 00000000000..48f55aa86ff--- /dev/null+++ b/packages/pieces/core/subflows/test/deliver-subflow-response.test.ts@@ -0,0 +1,52 @@+/// <reference types="vitest/globals" />++import { httpClient } from '@activepieces/pieces-common';+import { deliverSubflowResponse } from '../src/lib/common';++vi.mock('@activepieces/pieces-common', async (importOriginal) => {+ const actual = await importOriginal<typeof import('@activepieces/pieces-common')>();+ return { ...actual, httpClient: { sendRequest: vi.fn() } };+});++const sendRequest = vi.mocked(httpClient.sendRequest);++describe('deliverSubflowResponse', () => {+ beforeEach(() => {+ sendRequest.mockReset();+ });++ test('posts the response to the callback url', async () => {+ sendRequest.mockResolvedValue({ status: 200, body: { message: 'recorded', discarded: false } });++ await deliverSubflowResponse({ callbackUrl: 'https://host/resume', response: { ok: true } });++ expect(sendRequest.mock.calls[0][0]).toMatchObject({+ url: 'https://host/resume',+ body: { status: 'success', data: { ok: true } },+ });+ });++ test('throws when the caller had already stopped waiting, so the drop is not reported as success', async () => {+ sendRequest.mockResolvedValue({ status: 200, body: { message: 'expired', discarded: true } });++ await expect(+ deliverSubflowResponse({ callbackUrl: 'https://host/resume', response: { ok: true } })+ ).rejects.toThrow(/discarded/);+ });++ test('stays silent on a redelivery the server already accepted', async () => {+ sendRequest.mockResolvedValue({ status: 200, body: { message: 'expired', discarded: false } });++ await expect(+ deliverSubflowResponse({ callbackUrl: 'https://host/resume', response: { ok: true } })+ ).resolves.toBeUndefined();+ });++ test('stays silent when the acknowledgement carries no marker (older server)', async () => {+ sendRequest.mockResolvedValue({ status: 200, body: {} });++ await expect(+ deliverSubflowResponse({ callbackUrl: 'https://host/resume', response: { ok: true } })+ ).resolves.toBeUndefined();+ });+});diff --git a/packages/server/api/src/app/flows/flow-run/engine-run-callback-service.ts b/packages/server/api/src/app/flows/flow-run/engine-run-callback-service.tsindex 2f1287c05e1..dba64305baf 100644--- a/packages/server/api/src/app/flows/flow-run/engine-run-callback-service.ts+++ b/packages/server/api/src/app/flows/flow-run/engine-run-callback-service.ts@@ -53,6 +53,7 @@ export const engineRunCallbackService = (log: FastifyBaseLogger) => ({provisionMs: request.provisionMs,bootMs: request.bootMs,runMs: request.runMs,+ willRetry: request.willRetry,}await runsMetadataQueue(log).add(logData)diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts b/packages/server/api/src/app/flows/flow-run/flow-run-service.tsindex a4349fe5b7d..1016bc5af6d 100644--- a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts+++ b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts@@ -1,5 +1,5 @@import { ActivepiecesError, apId, Cursor, ErrorCode, FlowId, FlowRunId, FlowVersionId, isNil, PlatformId, ProjectId, SeekPage } from '@activepieces/core-utils'-import { apDayjs, wideEvent } from '@activepieces/server-utils'+import { apDayjs, apDayjsDuration, wideEvent } from '@activepieces/server-utils'import { ExecuteFlowJobData, ExecutionType, ExecutioOutputFile, FileCompression, FileType, FlowRetryStrategy, FlowRun, FlowRunCountByStatus, FlowRunStatus, FlowRunWithRetryError, FlowVersion, GenericStepOutput, isFlowRunStateTerminal, JobPayload, LATEST_JOB_DATA_SCHEMA_VERSION, logSerializer, LogSliceRef, ResumeReason, RunEnvironment, RunInternalError, SampleDataFileType, StepOutput, StepOutputStatus, StepOutputType, StreamStepProgress, WorkerJobType } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import pLimit from 'p-limit'@@ -26,6 +26,7 @@ import { flowRunSideEffects } from './flow-run-side-effects'import { runsMetadataQueue } from './flow-runs-queue'const CANCELLABLE_STATUSES: FlowRunStatus[] = [FlowRunStatus.PAUSED, FlowRunStatus.QUEUED]+const PARENT_BOUND_RETRY_BACKOFF_MS = apDayjsDuration(30, 'second').asMilliseconds()export const WEBHOOK_TIMEOUT_MS = system.getNumberOrThrow(AppSystemProp.WEBHOOK_TIMEOUT_SECONDS) * 1000@@ -648,10 +649,12 @@ export async function addToQueue(params: AddToQueueParams, log: FastifyBaseLoggeexecutionType: ExecutionType.BEGIN,executeTrigger: params.executeTrigger,}+ const aParentIsBlockedOnThisRun = params.flowRun.failParentOnFailure && !isNil(params.flowRun.parentRunId)await jobQueue(log).add({id: params.jobId ?? params.flowRun.id,type: JobType.ONE_TIME,data,+ retryBackoffMs: aParentIsBlockedOnThisRun ? PARENT_BOUND_RETRY_BACKOFF_MS : undefined,})return params.flowRun}diff --git a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.tsindex 90ab3cd62f9..9c8d111f8be 100644--- a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts+++ b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts@@ -100,11 +100,17 @@ export const runsMetadataQueue = (log: FastifyBaseLogger) => ({}const parentRunId = savedFlowRun.parentRunId- const shouldMarkParentAsFailed = savedFlowRun.failParentOnFailure && !isNil(parentRunId) && ![FlowRunStatus.SUCCEEDED, FlowRunStatus.RUNNING, FlowRunStatus.PAUSED, FlowRunStatus.QUEUED].includes(savedFlowRun.status)+ const shouldMarkParentAsFailed = !isNil(parentRunId) && childRunFailsParent({+ status: savedFlowRun.status,+ failParentOnFailure: savedFlowRun.failParentOnFailure,+ willRetry: runMetadata.willRetry,+ })if (shouldMarkParentAsFailed) {await markParentRunAsFailed({parentRunId,childRunId: savedFlowRun.id,+ childStatus: savedFlowRun.status,+ childFailureMessage: savedFlowRun.failedStep?.message,projectId: savedFlowRun.projectId,log,})@@ -188,9 +194,17 @@ function buildTimeline({ existingFlowRun, runMetadata }: BuildTimelineParams): R})}+export function childRunFailsParent({ status, failParentOnFailure, willRetry }: ChildRunFailsParentParams): boolean {+ return failParentOnFailure+ && status !== FlowRunStatus.SUCCEEDED+ && isFlowRunStateTerminal({ status, ignoreInternalError: willRetry === true })+}+export async function markParentRunAsFailed({parentRunId,childRunId,+ childStatus,+ childFailureMessage,projectId,log,}: MarkParentRunAsFailedParams): Promise<void> {@@ -208,7 +222,8 @@ export async function markParentRunAsFailed({body: {status: 'error',data: {- message: 'Subflow execution failed',+ message: childFailureMessage ?? 'Subflow execution failed',+ status: childStatus,link: childRunUrl,},},@@ -233,6 +248,12 @@ export async function markParentRunAsFailed({}}+type ChildRunFailsParentParams = {+ status: FlowRunStatus+ failParentOnFailure: boolean+ willRetry?: boolean+}+type BuildTimelineParams = {existingFlowRun: FlowRunrunMetadata: RunsMetadataUpsertData@@ -241,6 +262,8 @@ type BuildTimelineParams = {type MarkParentRunAsFailedParams = {parentRunId: stringchildRunId: string+ childStatus: FlowRunStatus+ childFailureMessage?: stringprojectId: stringlog: FastifyBaseLogger}diff --git a/packages/server/api/src/app/waitpoints/resume-controller.ts b/packages/server/api/src/app/waitpoints/resume-controller.tsindex 6afdd22b3c8..e1391948a21 100644--- a/packages/server/api/src/app/waitpoints/resume-controller.ts+++ b/packages/server/api/src/app/waitpoints/resume-controller.ts@@ -111,7 +111,7 @@ async function handleConfirmResume({ flowRunId, waitpointId, action, body, headeresumePayload: { body, headers, queryParams },})if (!acceptsHtml(headers)) {- await reply.send({ message: stale ? EXPIRED_MESSAGE : RECORDED_MESSAGE })+ await reply.send({ message: stale ? EXPIRED_MESSAGE : RECORDED_MESSAGE, discarded: await responseWasDiscarded({ flowRunId, waitpointId, stale, log }) })return}const theme = await resolveResumePageTheme({ projectId: flowRun.projectId, log })@@ -127,7 +127,7 @@ async function handleAsyncResume({ flowRunId, waitpointId, body, headers, queryPwaitpointId,resumePayload: { body, headers, queryParams },})- await reply.send({ message: stale ? EXPIRED_MESSAGE : RECORDED_MESSAGE })+ await reply.send({ message: stale ? EXPIRED_MESSAGE : RECORDED_MESSAGE, discarded: await responseWasDiscarded({ flowRunId, waitpointId, stale, log }) })}async function handleSyncResume({ flowRunId, waitpointId, body, headers, queryParams, log, reply, correlationId }: AsyncResumeHandlerParams & { correlationId: string }): Promise<void> {@@ -157,6 +157,18 @@ async function handleLegacySyncResume({ flowRunId, body, headers, queryParams, lawait reply.status(response.status).headers(response.headers).send(response.body)}+/**+ * A stale resume means the waitpoint is gone — which is equally what a SUCCEEDED delivery+ * looks like once its HTTP response is lost and the caller retries. Only the absence of a+ * consumption marker proves nobody ever accepted this response.+ */+async function responseWasDiscarded({ flowRunId, waitpointId, stale, log }: ResponseWasDiscardedParams): Promise<boolean> {+ if (!stale) {+ return false+ }+ return !(await resumeService(log).waitpointWasConsumed({ waitpointId, flowRunId }))+}+async function resolveResumePageTheme({ projectId, log }: { projectId: string, log: FastifyBaseLogger }): Promise<ResumePageTheme> {const platformId = await projectService(log).getPlatformId(projectId)return resumePageHooks.get(log).getTheme({ platformId })@@ -310,6 +322,13 @@ const STATUS_HTML_TEMPLATE = `<!DOCTYPE html></body></html>`+type ResponseWasDiscardedParams = {+ flowRunId: string+ waitpointId: string+ stale: boolean+ log: FastifyBaseLogger+}+type ConfirmationPageParams = {flowRunId: stringwaitpointId: stringdiff --git a/packages/server/api/src/app/waitpoints/resume-service.ts b/packages/server/api/src/app/waitpoints/resume-service.tsindex f0584ffa2a8..15413acf038 100644--- a/packages/server/api/src/app/waitpoints/resume-service.ts+++ b/packages/server/api/src/app/waitpoints/resume-service.ts@@ -1,8 +1,8 @@-import { apId, FlowRunId, isNil } from '@activepieces/core-utils'+import { apId, FlowRunId, isNil, tryCatch } from '@activepieces/core-utils'import { EngineHttpResponse, ExecutionType, FlowRun, FlowRunStatus, isFlowRunStateTerminal, ResumeReason, RunEnvironment, StreamStepProgress } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { StatusCodes } from 'http-status-codes'-import { distributedLock } from '../database/redis-connections'+import { distributedLock, distributedStore } from '../database/redis-connections'import { addToQueue, findFlowRunOrThrow, flowRunService, WEBHOOK_TIMEOUT_MS } from '../flows/flow-run/flow-run-service'import { flowRunSideEffects } from '../flows/flow-run/flow-run-side-effects'import { projectService } from '../project/project-service'@@ -29,6 +29,7 @@ export const resumeService = (log: FastifyBaseLogger) => ({resumePayload: resumePayload ?? null,workerHandlerId,onReady: async (waitpoint) => {+ await recordWaitpointConsumed({ waitpointId, log })await enqueueResume({flowRun,waitpoint,@@ -47,7 +48,6 @@ export const resumeService = (log: FastifyBaseLogger) => ({log.info({ flowRun: { id: flowRunId } }, '[resumeService#resumeFromWaitpointWithoutLock] Race detected: metadata worker wrote PAUSED after callback completed waitpoint; consuming waitpoint and enqueuing resume')// Consume the stale COMPLETED waitpoint under the lock so it cannot// poison the next createForPause call on a subsequent loop iteration- await waitpointService(log).delete({ id: latestWaitpoint.id })await enqueueResume({flowRun: currentFlowRun,waitpoint: latestWaitpoint,@@ -55,6 +55,8 @@ export const resumeService = (log: FastifyBaseLogger) => ({workerHandlerId,httpRequestId,}, log)+ await recordWaitpointConsumed({ waitpointId: latestWaitpoint.id, log })+ await waitpointService(log).delete({ id: latestWaitpoint.id })}}}@@ -62,6 +64,19 @@ export const resumeService = (log: FastifyBaseLogger) => ({return { flowRun, stale: !processed }},+ async waitpointWasConsumed({ waitpointId, flowRunId }: WaitpointWasConsumedParams): Promise<boolean> {+ const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId })+ if (!isNil(waitpoint)) {+ return waitpoint.status === WaitpointStatus.COMPLETED+ }+ const { data, error } = await tryCatch(() => distributedStore.get<boolean>(waitpointConsumedKey(waitpointId)))+ if (error) {+ log.error({ waitpoint: { id: waitpointId }, error: String(error) }, '[resumeService#waitpointWasConsumed] Could not read the consumption marker, assuming the response was delivered')+ return true+ }+ return data === true+ },+async legacyResume({ flowRunId, resumePayload, workerHandlerId }: LegacyResumeParams): Promise<ResumeFromWaitpointResult> {const flowRun = await findFlowRunOrThrow(flowRunId)if (flowRun.status !== FlowRunStatus.PAUSED) {@@ -128,6 +143,15 @@ export const resumeService = (log: FastifyBaseLogger) => ({},})+async function recordWaitpointConsumed({ waitpointId, log }: RecordWaitpointConsumedParams): Promise<void> {+ const { error } = await tryCatch(() => distributedStore.put(waitpointConsumedKey(waitpointId), true, WAITPOINT_CONSUMED_TTL_SECONDS))+ if (isNil(error)) {+ return+ }+ log.error({ waitpoint: { id: waitpointId }, error: String(error) }, '[resumeService#recordWaitpointConsumed] Could not record the consumption marker, rolling the resume back so the caller can retry')+ throw error+}+async function enqueueResume(params: EnqueueResumeParams, log: FastifyBaseLogger): Promise<void> {const { flowRun, waitpoint, resumePayload, workerHandlerId, httpRequestId } = paramsconst platformId = await projectService(log).getPlatformId(flowRun.projectId)@@ -150,6 +174,19 @@ async function enqueueResume(params: EnqueueResumeParams, log: FastifyBaseLoggerawait flowRunSideEffects(log).onResume({ flowRun, platformId })}+const WAITPOINT_CONSUMED_TTL_SECONDS = 3600+const waitpointConsumedKey = (waitpointId: string): string => `waitpoint_consumed:${waitpointId}`++type WaitpointWasConsumedParams = {+ waitpointId: string+ flowRunId: string+}++type RecordWaitpointConsumedParams = {+ waitpointId: string+ log: FastifyBaseLogger+}+type SyncResumePayload = {body?: unknownheaders?: Record<string, string>diff --git a/packages/server/api/src/app/workers/job-queue/job-broker.ts b/packages/server/api/src/app/workers/job-queue/job-broker.tsindex 9352e9086bf..923d67eecda 100644--- a/packages/server/api/src/app/workers/job-queue/job-broker.ts+++ b/packages/server/api/src/app/workers/job-queue/job-broker.ts@@ -150,6 +150,7 @@ async function tryDequeue(worker: BullMQWorker, queueName: string, log: FastifyBjobId,jobData: migratedData,attempsStarted: job.attemptsMade,+ lastAttempt: job.attemptsMade + 1 >= (job.opts.attempts ?? 1),engineToken,token,queueName,diff --git a/packages/server/api/src/app/workers/job-queue/job-queue.ts b/packages/server/api/src/app/workers/job-queue/job-queue.tsindex ad442a20828..dabc5a93f98 100644--- a/packages/server/api/src/app/workers/job-queue/job-queue.ts+++ b/packages/server/api/src/app/workers/job-queue/job-queue.ts@@ -52,6 +52,7 @@ export const jobQueue = (log: FastifyBaseLogger) => ({priority: JOB_PRIORITY[getDefaultJobPriority(data)],delay: params.delay,jobId: params.id,+ ...(isNil(params.retryBackoffMs) ? {} : { backoff: { type: 'exponential', delay: params.retryBackoffMs } }),...(data.jobType === WorkerJobType.EVENT_DESTINATION ? { removeOnFail: true } : {}),...([WorkerJobType.EXECUTE_AGENT_RUN, WorkerJobType.EXECUTE_AI].includes(data.jobType) ? { attempts: 1 } : {}),...isUserInteractionJob(data.jobType) ? {@@ -275,6 +276,8 @@ type BaseAddParams<JD extends Omit<JobData, 'engineToken'>, JT extends JobType>type RepeatingJobAddParams = BaseAddParams<PollingJobData | RenewWebhookJobData, JobType.REPEATING> & {scheduleOptions: ScheduleOptions}-type OneTimeJobAddParams = BaseAddParams<ExecuteFlowJobData | WebhookJobData | UserInteractionJobData | EventDestinationJobData | ExecuteAgentRunJobData | ExecuteAiJobData | ExecutePersonalizationResearchJobData, JobType.ONE_TIME>+type OneTimeJobAddParams = BaseAddParams<ExecuteFlowJobData | WebhookJobData | UserInteractionJobData | EventDestinationJobData | ExecuteAgentRunJobData | ExecuteAiJobData | ExecutePersonalizationResearchJobData, JobType.ONE_TIME> & {+ retryBackoffMs?: number+}export type AddJobParams<type extends JobType> = type extends JobType.REPEATING ? RepeatingJobAddParams : OneTimeJobAddParamsdiff --git a/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts b/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.tsindex 1d2134c4a78..b865d6e9b99 100644--- a/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts+++ b/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts@@ -71,7 +71,7 @@ const RUNS_METADATA_UPSERT_KEYS: (keyof RunsMetadataUpsertData)[] = ['triggeredBy', 'startTime', 'finishTime', 'status', 'tags','failedStep', 'stepNameToTest', 'parentRunId', 'failParentOnFailure','logsFileId', 'updated', 'stepsCount', 'requestId',- 'provisionMs', 'bootMs', 'runMs',+ 'provisionMs', 'bootMs', 'runMs', 'willRetry',]function stripToRunsMetadataUpsertData(params: RunsMetadataUpsertData): RunsMetadataUpsertData {@@ -123,4 +123,5 @@ export type RunsMetadataUpsertData = {provisionMs?: numberbootMs?: numberrunMs?: number+ willRetry?: boolean}diff --git a/packages/server/api/test/helpers/mocks/index.ts b/packages/server/api/test/helpers/mocks/index.tsindex 20611a2b901..4505ae825dc 100644--- a/packages/server/api/test/helpers/mocks/index.ts+++ b/packages/server/api/test/helpers/mocks/index.ts@@ -130,6 +130,7 @@ export const createMockProject = (project?: Partial<Project>): Project => {platformId: project?.platformId ?? apId(),externalId: project?.externalId ?? apId(),releasesEnabled: project?.releasesEnabled ?? false,+ sensitive: project?.sensitive ?? false,notifyFlowOwnerOnFailure: project?.notifyFlowOwnerOnFailure ?? false,metadata: project?.metadata ?? null,type: project?.type ?? ProjectType.TEAM,@@ -707,6 +708,10 @@ export const createMockAIProvider = async (aiProvider?: MockAIProviderParams): PmodelIds: aiProvider?.modelIds ?? [],projectScope: aiProvider?.projectScope ?? 'all',projectIds: aiProvider?.projectIds ?? [],+ status: 'active',+ statusReason: null,+ statusUpdated: null,+ statusVersion: 0,}}@@ -735,6 +740,19 @@ export const mockPieceMetadata = async (mockLog: FastifyBaseLogger): Promise<Piereturn mockPieceMetadata}+export const createMockWaitpoint = (waitpoint?: Partial<MockWaitpoint>): MockWaitpoint => {+ return {+ id: waitpoint?.id ?? apId(),+ flowRunId: waitpoint?.flowRunId ?? apId(),+ projectId: waitpoint?.projectId ?? apId(),+ stepName: waitpoint?.stepName ?? 'approval',+ type: waitpoint?.type ?? 'WEBHOOK',+ status: waitpoint?.status ?? 'PENDING',+ httpRequestId: waitpoint?.httpRequestId ?? null,+ workerHandlerId: waitpoint?.workerHandlerId ?? null,+ }+}+export const createMockFolder = (folder?: Partial<Folder>): Folder => {return {id: folder?.id ?? apId(),@@ -803,3 +821,14 @@ type MockBasicSetupParams = {platform?: Partial<Platform>project?: Partial<Project>}++export type MockWaitpoint = {+ id: string+ flowRunId: string+ projectId: string+ stepName: string+ type: string+ status: string+ httpRequestId: string | null+ workerHandlerId: string | null+}diff --git a/packages/server/api/test/integration/ce/flows/flow-run/child-internal-error-retry.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/child-internal-error-retry.test.tsnew file mode 100644index 00000000000..1dfddbb2e99--- /dev/null+++ b/packages/server/api/test/integration/ce/flows/flow-run/child-internal-error-retry.test.ts@@ -0,0 +1,92 @@+import { apId } from '@activepieces/core-utils'+import { FlowRunStatus, FlowVersionState, RunEnvironment } from '@activepieces/shared'+import { FastifyInstance } from 'fastify'+import { engineRunCallbackService } from '../../../../../src/app/flows/flow-run/engine-run-callback-service'+import { db } from '../../../../helpers/db'+import { createMockFlow, createMockFlowRun, createMockFlowVersion, createMockWaitpoint, mockAndSaveBasicSetup } from '../../../../helpers/mocks'+import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'++let app: FastifyInstance++beforeAll(async () => {+ app = await setupTestEnvironment()+})++afterAll(async () => {+ await teardownTestEnvironment()+})++async function createParentWithPausedWaitpointAndChild(projectId: string) {+ const flow = createMockFlow({ projectId })+ await db.save('flow', flow)++ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })+ await db.save('flow_version', flowVersion)++ const parentRun = createMockFlowRun({+ projectId,+ flowId: flow.id,+ flowVersionId: flowVersion.id,+ status: FlowRunStatus.PAUSED,+ environment: RunEnvironment.PRODUCTION,+ })+ await db.save('flow_run', parentRun)++ const waitpoint = createMockWaitpoint({ flowRunId: parentRun.id, projectId, stepName: 'step_1' })+ await db.save('waitpoint', waitpoint)++ const childRun = createMockFlowRun({+ projectId,+ flowId: flow.id,+ flowVersionId: flowVersion.id,+ status: FlowRunStatus.RUNNING,+ environment: RunEnvironment.PRODUCTION,+ parentRunId: parentRun.id,+ failParentOnFailure: true,+ finishTime: null,+ })+ await db.save('flow_run', childRun)++ return { parentRun, childRun, waitpointId: waitpoint.id }+}++async function reportChildInternalError({ projectId, childRunId, willRetry }: { projectId: string, childRunId: string, willRetry: boolean }) {+ await engineRunCallbackService(app.log).uploadRunLog({+ projectId,+ request: {+ runId: childRunId,+ projectId,+ status: FlowRunStatus.INTERNAL_ERROR,+ finishTime: new Date().toISOString(),+ willRetry,+ },+ })+}++describe('child run INTERNAL_ERROR with a pending queue retry', () => {+ it('leaves the parent paused while the child job still has an attempt left', async () => {+ const { mockProject } = await mockAndSaveBasicSetup()+ const { parentRun, childRun, waitpointId } = await createParentWithPausedWaitpointAndChild(mockProject.id)++ await reportChildInternalError({ projectId: mockProject.id, childRunId: childRun.id, willRetry: true })+ await new Promise((resolve) => setTimeout(resolve, 3000))++ const waitpoint = await db.findOneBy<{ status: string }>('waitpoint', { id: waitpointId })+ expect(waitpoint?.status).toBe('PENDING')++ const parent = await db.findOneBy<{ status: string }>('flow_run', { id: parentRun.id })+ expect(parent?.status).toBe(FlowRunStatus.PAUSED)+ }, 60_000)++ it('fails the parent once the child job has no attempt left', async () => {+ const { mockProject } = await mockAndSaveBasicSetup()+ const { childRun, waitpointId } = await createParentWithPausedWaitpointAndChild(mockProject.id)++ await reportChildInternalError({ projectId: mockProject.id, childRunId: childRun.id, willRetry: false })++ await vi.waitUntil(+ async () => await db.findOneBy('waitpoint', { id: waitpointId }) === null,+ { timeout: 10_000, interval: 250 },+ )+ }, 60_000)+})diff --git a/packages/server/api/test/integration/ce/flows/flow-run/mark-parent-run-cross-tenant.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/mark-parent-run-cross-tenant.test.tsindex 4f2ed2cb3f9..71700a37bd6 100644--- a/packages/server/api/test/integration/ce/flows/flow-run/mark-parent-run-cross-tenant.test.ts+++ b/packages/server/api/test/integration/ce/flows/flow-run/mark-parent-run-cross-tenant.test.ts@@ -3,7 +3,7 @@ import { FlowRunStatus, FlowVersionState, RunEnvironment } from '@activepieces/simport { FastifyInstance } from 'fastify'import { markParentRunAsFailed } from '../../../../../src/app/flows/flow-run/flow-runs-queue'import { db } from '../../../../helpers/db'-import { createMockFlow, createMockFlowVersion, createMockFlowRun, mockAndSaveBasicSetup } from '../../../../helpers/mocks'+import { createMockFlow, createMockFlowVersion, createMockFlowRun, createMockWaitpoint, mockAndSaveBasicSetup } from '../../../../helpers/mocks'import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'let app: FastifyInstance@@ -35,19 +35,10 @@ async function createPausedParentWithWaitpoint(projectId: string) {})await db.save('flow_run', flowRun)- const waitpointId = apId()- await db.save('waitpoint', {- id: waitpointId,- flowRunId: flowRun.id,- projectId,- stepName: 'approval',- type: 'WEBHOOK',- status: 'PENDING',- httpRequestId: null,- workerHandlerId: null,- })+ const waitpoint = createMockWaitpoint({ flowRunId: flowRun.id, projectId })+ await db.save('waitpoint', waitpoint)- return { flowRun, waitpointId }+ return { flowRun, waitpointId: waitpoint.id }}describe('markParentRunAsFailed tenant isolation', () => {@@ -60,6 +51,7 @@ describe('markParentRunAsFailed tenant isolation', () => {await markParentRunAsFailed({parentRunId: victimRun.id,childRunId: apId(),+ childStatus: FlowRunStatus.INTERNAL_ERROR,projectId: projectA.id,log: app.log,})@@ -79,6 +71,7 @@ describe('markParentRunAsFailed tenant isolation', () => {await markParentRunAsFailed({parentRunId: parentRun.id,childRunId: apId(),+ childStatus: FlowRunStatus.INTERNAL_ERROR,projectId: mockProject.id,log: app.log,})diff --git a/packages/server/api/test/integration/ce/flows/flow-run/resume-flow-run.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/resume-flow-run.test.tsindex e57c8cfa7a4..6f2fe0a1326 100644--- a/packages/server/api/test/integration/ce/flows/flow-run/resume-flow-run.test.ts+++ b/packages/server/api/test/integration/ce/flows/flow-run/resume-flow-run.test.ts@@ -9,6 +9,7 @@ import { pubsub } from '../../../../../src/app/helper/pubsub'import { engineResponseWatcher } from '../../../../../src/app/workers/engine-response-watcher'import { redisMetadataKey, RunsMetadataUpsertData } from '../../../../../src/app/workers/job'import { createHandlers } from '../../../../../src/app/workers/rpc/worker-rpc-service'+import { distributedStore } from '../../../../../src/app/database/redis-connections'import { db } from '../../../../helpers/db'import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'import { createTestContext, TestContext } from '../../../../helpers/test-context'@@ -517,6 +518,7 @@ describe('Resume flow run', () => {expect(firstResponse.statusCode).toBe(200)expect(firstResponse.json()).toEqual({message: 'Your response has been recorded. You can close this page now.',+ discarded: false,})const secondResponse = await app.inject({@@ -527,12 +529,113 @@ describe('Resume flow run', () => {expect(secondResponse.statusCode).toBe(200)expect(secondResponse.json()).toEqual({message: 'This link has expired. The action may have already been processed.',+ discarded: false,})const waitpointAfter = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })expect(waitpointAfter).toBeNull()})+ it.each([+ FlowRunStatus.INTERNAL_ERROR,+ FlowRunStatus.SUCCEEDED,+ FlowRunStatus.FAILED,+ ])('does not report a redelivered response as discarded when the caller has since reached %s', async (status) => {+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })++ const deliver = async () => app.inject({+ method: 'POST',+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,+ body: { status: 'success', data: { greeting: 'Hello' } },+ })++ await deliver()+ await db.update('flow_run', flowRun.id, { status })+ const redelivery = await deliver()++ expect(redelivery.json().discarded).toBe(false)+ })++ it('falls back to the completed waitpoint row when no consumption marker exists', async () => {+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })+ // RUNNING takes the complete() branch: the waitpoint is marked COMPLETED and kept,+ // and no Redis marker is written at all, so only the row can prove acceptance.+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.RUNNING })++ const accepted = await app.inject({+ method: 'POST',+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,+ body: { status: 'success', data: { greeting: 'Hello' } },+ })+ expect(accepted.json().discarded).toBe(false)++ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.FAILED })+ const redelivery = await app.inject({+ method: 'POST',+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,+ body: { status: 'success', data: { greeting: 'Hello' } },+ })++ expect(redelivery.json().discarded).toBe(false)+ })++ it('keeps the evidence when the race path deletes the completed waitpoint row', async () => {+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })+ // RUNNING takes the complete() branch, then the PAUSED race block below it deletes the+ // COMPLETED row — so the marker is the only evidence that can survive.+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.RUNNING })++ await app.inject({+ method: 'POST',+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,+ body: { status: 'success', data: { greeting: 'Hello' } },+ })++ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.FAILED })+ const redelivery = await app.inject({+ method: 'POST',+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,+ body: { status: 'success', data: { greeting: 'Hello' } },+ })++ expect(redelivery.json().discarded).toBe(false)+ })++ it('ignores a consumption marker while the waitpoint row is still pending', async () => {+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })+ // A rollback after the marker was written leaves the marker behind with the row intact.+ // The row is the authority: still PENDING means nothing ever accepted the response.+ await distributedStore.put(`waitpoint_consumed:${waitpoint!.id}`, true, 3600)+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.FAILED })++ const response = await app.inject({+ method: 'POST',+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,+ body: { status: 'success', data: { greeting: 'Hello' } },+ })++ expect(response.json().discarded).toBe(true)+ })++ it('reports a response as discarded when the waitpoint was never consumed', async () => {+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.FAILED })++ const response = await app.inject({+ method: 'POST',+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,+ body: { status: 'success', data: { greeting: 'Hello' } },+ })++ expect(response.statusCode).toBe(200)+ expect(response.json().discarded).toBe(true)+ })+it('should clean up waitpoints when flow is deleted via batchDeleteByFlowId', async () => {const { flowRun, flow } = await createPausedFlowRunWithWaitpoint({projectId: ctx.project.id,@@ -567,6 +670,7 @@ describe('Resume flow run', () => {expect(response.statusCode).toBe(200)expect(response.json()).toEqual({message: 'Your response has been recorded. You can close this page now.',+ discarded: false,})const waitpointAfter = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })@@ -733,6 +837,7 @@ describe('Resume flow run', () => {expect(response.statusCode).toBe(200)expect(response.json()).toEqual({message: 'Your response has been recorded. You can close this page now.',+ discarded: false,})const waitpointAfter = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })@@ -775,6 +880,7 @@ describe('Resume flow run', () => {expect(response.statusCode).toBe(200)expect(response.json()).toEqual({message: 'Your response has been recorded. You can close this page now.',+ discarded: false,})const waitpointAfter = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })diff --git a/packages/server/api/test/unit/app/flows/flow-run/child-run-fails-parent.test.ts b/packages/server/api/test/unit/app/flows/flow-run/child-run-fails-parent.test.tsnew file mode 100644index 00000000000..008897565a9--- /dev/null+++ b/packages/server/api/test/unit/app/flows/flow-run/child-run-fails-parent.test.ts@@ -0,0 +1,51 @@+import { FlowRunStatus } from '@activepieces/shared'+import { childRunFailsParent } from '../../../../../src/app/flows/flow-run/flow-runs-queue'++describe('childRunFailsParent', () => {+ it('does not fail the parent while the child job still owes an attempt', () => {+ expect(childRunFailsParent({+ status: FlowRunStatus.INTERNAL_ERROR,+ failParentOnFailure: true,+ willRetry: true,+ })).toBe(false)+ })++ it('fails the parent once the child job has exhausted its attempts', () => {+ expect(childRunFailsParent({+ status: FlowRunStatus.INTERNAL_ERROR,+ failParentOnFailure: true,+ willRetry: false,+ })).toBe(true)+ })++ it('fails the parent when the report carries no retry information', () => {+ expect(childRunFailsParent({+ status: FlowRunStatus.INTERNAL_ERROR,+ failParentOnFailure: true,+ })).toBe(true)+ })++ it.each([+ FlowRunStatus.FAILED,+ FlowRunStatus.CANCELED,+ ])('fails the parent on terminal %s even if a stale retry flag is set', (status) => {+ expect(childRunFailsParent({ status, failParentOnFailure: true, willRetry: true })).toBe(true)+ })++ it.each([+ FlowRunStatus.SUCCEEDED,+ FlowRunStatus.RUNNING,+ FlowRunStatus.PAUSED,+ FlowRunStatus.QUEUED,+ ])('leaves the parent alone while the child is %s', (status) => {+ expect(childRunFailsParent({ status, failParentOnFailure: true, willRetry: false })).toBe(false)+ })++ it('leaves the parent alone when the child is not linked to it', () => {+ expect(childRunFailsParent({+ status: FlowRunStatus.INTERNAL_ERROR,+ failParentOnFailure: false,+ willRetry: false,+ })).toBe(false)+ })+})diff --git a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts b/packages/server/worker/src/lib/execute/jobs/execute-flow.tsindex cb9091edeb3..54acb8b64c7 100644--- a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts+++ b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts@@ -188,6 +188,7 @@ function toInternalError(source: RunInternalErrorSource, error: unknown): RunInt}async function reportFlowStatus({ ctx, data, status, internalError, failedStep }: ReportFlowStatusParams): Promise<void> {+ const willRetry = status === FlowRunStatus.INTERNAL_ERROR && !ctx.lastAttempt// A status report has no log file of its own; carry logsFileId only for an internalError the server may// persist into one (see uploadRunLog). Sending it on a plain status report would dangle flow_run.logsFileId.await ctx.apiClient.uploadRunLog({@@ -199,6 +200,7 @@ async function reportFlowStatus({ ctx, data, status, internalError, failedStep }...(isNil(internalError) ? {} : { logsFileId: data.logsFileId }),internalError,failedStep,+ willRetry,...spreadIfDefined('workerHandlerId', data.workerHandlerId ?? undefined),...spreadIfDefined('httpRequestId', data.httpRequestId),})@@ -207,7 +209,7 @@ async function reportFlowStatus({ ctx, data, status, internalError, failedStep }onCallService(ctx.log, workerSettings.getSettings().PAGE_ONCALL_WEBHOOK).page({code: ErrorCode.ENGINE_OPERATION_FAILURE,message: `Flow run ${data.runId} ended with INTERNAL_ERROR`,- params: { runId: data.runId, flowId: data.flowId, projectId: data.projectId },+ params: { runId: data.runId, flowId: data.flowId, projectId: data.projectId, willRetry },}).catch((e) => ctx.log.error({ flowRun: { id: data.runId }, error: inspect(e) }, 'Failed to send on-call page for INTERNAL_ERROR'))}}diff --git a/packages/server/worker/src/lib/execute/types.ts b/packages/server/worker/src/lib/execute/types.tsindex 71139b526f0..a8d622dcdb2 100644--- a/packages/server/worker/src/lib/execute/types.ts+++ b/packages/server/worker/src/lib/execute/types.ts@@ -13,6 +13,7 @@ export type JobContext = {resolver: ResolverworkerIndex: numberjobId: string+ lastAttempt: booleanengineToken: stringinternalApiUrl: stringpublicApiUrl: stringdiff --git a/packages/server/worker/src/lib/worker.ts b/packages/server/worker/src/lib/worker.tsindex 1b8f06385e8..001030a3d51 100644--- a/packages/server/worker/src/lib/worker.ts+++ b/packages/server/worker/src/lib/worker.ts@@ -371,6 +371,7 @@ async function executeJob(apiClient: WorkerToApiContract, job: ConsumeJobRequest}),workerIndex,jobId: job.jobId,+ lastAttempt: job.lastAttempt ?? true,engineToken: job.engineToken,internalApiUrl,publicApiUrl,