Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(subflows): keep the calling flow waiting when a subflow crashes and is retried (#15642)

Hazem Adel7cccf090a105
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 01f06bd4c0a..3ceca4b720b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -152,7 +152,7 @@ jobs:
set -euo pipefail
pids=()
- npx turbo run test --filter=@activepieces/engine --filter=@activepieces/shared --filter=@activepieces/sandbox --filter=@activepieces/ai-providers --filter=@activepieces/core-execution --filter=@activepieces/core-piece-types --filter=@activepieces/server-utils --filter=@activepieces/pieces-framework --filter=web --filter=worker &
+ npx turbo run test --filter=@activepieces/engine --filter=@activepieces/shared --filter=@activepieces/sandbox --filter=@activepieces/ai-providers --filter=@activepieces/core-execution --filter=@activepieces/core-piece-types --filter=@activepieces/server-utils --filter=@activepieces/pieces-framework --filter=@activepieces/piece-subflows --filter=web --filter=worker &
pids+=($!)
npx turbo run test-ce test-ee test-cloud check-migrations --filter=api &
diff --git a/brain/knowledge/flows-execution/subflows.md b/brain/knowledge/flows-execution/subflows.md
index f7c25a25031..9cb4243ab71 100644
--- a/brain/knowledge/flows-execution/subflows.md
+++ b/brain/knowledge/flows-execution/subflows.md
@@ -31,6 +31,9 @@ No server entity of its own — subflows are ordinary flows plus two conventions
- **Fan-out is not Call Flow.** Wait-for-response is incompatible with many batches inside one step; only Call Flow can wait.
- **"Retry on failure" is hidden on the fan-out action, on purpose.** The framework default (`action.ts`) is `defaultValue: false` but *visible*, and step retry re-runs `run()` from the top — for a fire-and-forget fan-out that means re-reading the file from row 0 and re-dispatching every batch that already landed. A failure at batch 900 would duplicate 900 batches. So the action declares `retryOnFailure: { defaultValue: false, hide: true }`; `continueOnFailure` stays visible. `Call Flow` leaves both visible because one invocation is safe to retry — but with wait-for-response, a retry after the subflow has answered `status: 'error'` only rethrows the stored resume payload, so it costs ~28s of backoff and changes nothing. It must never re-call the subflow: that is the runaway loop of GIT-1712, see the step-retry gotchas in [flow-runs](flow-runs.md).
- **`batchSize` is capped at 10,000, because "streaming bounds memory" only holds for sane batch sizes.** Peak memory is roughly `(MAX_IN_FLIGHT + 1) × batchSize × rowSize`, so an unbounded `batchSize` defeats the whole design. The cap that bites first in practice is not memory but the webhook `bodyLimit` (`max(AP_MAX_FILE_SIZE_MB + 4, AP_MAX_FLOW_RUN_LOG_SIZE_MB + 4, 25)` MB, 54MB by default) — a wide-row file will 413 well before 10,000 rows.
+- **A child's `INTERNAL_ERROR` is not the end of that run.** `EXECUTE_FLOW` jobs carry `attempts: 2`, and the retry reuses the same job id — so the same run id flips `INTERNAL_ERROR → RUNNING → SUCCEEDED` and attempt 1 leaves no row behind. Anything that treats a child status as terminal must first ask whether the queue still owes an attempt: the worker stamps `willRetry` on its run-log upload (from `ConsumeJobRequest.lastAttempt`, computed in `job-broker.ts` from `attemptsMade` vs `opts.attempts`), and `childRunFailsParent` in `flow-runs-queue.ts` feeds it to `isFlowRunStateTerminal` as `ignoreInternalError` — that existing flag already meant "INTERNAL_ERROR is not final yet", so `willRetry` is the fact it was approximating. Because the switch consults it only in the `INTERNAL_ERROR` branch, a stale flag cannot suppress a genuine `FAILED`. The worker must keep sending `willRetry: false` explicitly: the metadata payload is a merged Redis hash, so omitting it would let a `true` from attempt 1 survive into the final attempt and strand the parent forever. Before that flag existed (GIT-1906) a crashed child failed its waiting parent in ~100ms, the retry then succeeded and its `Respond` callback hit an already-FAILED parent, which answers `200` and drops the payload silently.
+- **A run whose parent is blocked on it retries in 30s, not the queue default of 8 minutes.** `addToQueue` passes `retryBackoffMs` for any run with `failParentOnFailure` and a `parentRunId`, so the override rides on that job's own opts. Do NOT reach for `defaultJobOptions` in `job-queue.ts` to tune this — it is shared by webhooks, triggers, agents and polling, so a change there is never subflow-scoped.
+- **A discarded `Respond` is loud; a redelivered one is not.** `Respond` fails its step when the resume route answers `discarded: true`. That flag is **not** the route's `stale` flag: `stale` only says the waitpoint is gone, which is equally what a *successful* delivery looks like on retry — `Respond` posts with `retries: 10` and `fetch-http-client` retries on any thrown error, so a response lost on the way back is re-sent and finds its own waitpoint consumed. Nor can a run status settle it: `INTERNAL_ERROR`, `SUCCEEDED` and `FAILED` are all reachable *after* a successful delivery, and the first retry backs off only 1s. So the route keeps a positive record instead — `waitpoint_consumed:<id>` (1h TTL, comfortably past the ~5min retry window), and `discarded` is `stale && !evidence`. The write sits **inside** the consuming transaction, as the **first** step of `onReady`, and throws rather than logging — so a marker that cannot be written aborts before `enqueueResume` commits a BullMQ job the rollback cannot take back. Redis commits independently of Postgres, so ordering alone cannot make the two consistent; the read side closes the gap instead. **The waitpoint row is the authority**: still `PENDING` means nothing ever accepted the response, whatever the marker says, so an orphaned marker left by a rollback is ignored. Only once the row is gone does the marker speak, distinguishing a consumed waitpoint from one `onFinish` cleaned up. The `RUNNING`/`QUEUED` branch writes no marker — `complete()` leaves the row as `COMPLETED`, which is durable evidence in its own right, so `waitpointWasConsumed` falls back to it. The one place that must also write a marker is the GIT-1594 race block, which *deletes* that `COMPLETED` row: record before destroying the only evidence, or a later redelivery false-fails. Both reads fail safe: a Redis error reports *delivered*, because falsely failing a working flow is worse than a missed warning. HTTP stays `200`, and the legacy `/requests/:requestId` fallback has no waitpoint id so it carries no flag.
- The dropdown lists published flows carrying a Callable Flow trigger and labels disabled ones `(inactive)`; streaming to a disabled flow throws before the first request.
### Editions
diff --git a/bun.lock b/bun.lock
index dc0039725a5..0de9eb9fdae 100644
--- a/bun.lock
+++ b/bun.lock
@@ -892,7 +892,7 @@
},
"packages/pieces/community/aws-bedrock": {
"name": "@activepieces/piece-aws-bedrock",
- "version": "0.3.0",
+ "version": "0.3.1",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -6969,7 +6969,7 @@
},
"packages/pieces/community/pinterest": {
"name": "@activepieces/piece-pinterest",
- "version": "0.1.8",
+ "version": "0.1.9",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -8381,7 +8381,7 @@
},
"packages/pieces/community/slack": {
"name": "@activepieces/piece-slack",
- "version": "0.19.0",
+ "version": "0.20.0",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -10065,7 +10065,7 @@
},
"packages/pieces/community/zagomail": {
"name": "@activepieces/piece-zagomail",
- "version": "0.1.6",
+ "version": "0.1.7",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -10609,7 +10609,7 @@
},
"packages/pieces/core/subflows": {
"name": "@activepieces/piece-subflows",
- "version": "0.6.4",
+ "version": "0.7.0",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
diff --git a/packages/core/execution/src/lib/engine/requests.ts b/packages/core/execution/src/lib/engine/requests.ts
index 3e1faf8ef2f..7fb8f5a8187 100644
--- a/packages/core/execution/src/lib/engine/requests.ts
+++ b/packages/core/execution/src/lib/engine/requests.ts
@@ -29,6 +29,7 @@ export const UploadRunLogsRequest = z.object({
runMs: z.number().optional(),
workerHandlerId: z.string().optional(),
httpRequestId: z.string().optional(),
+ willRetry: z.boolean().optional(),
})
export type UploadRunLogsRequest = z.infer<typeof UploadRunLogsRequest>
diff --git a/packages/core/execution/src/lib/workers/index.ts b/packages/core/execution/src/lib/workers/index.ts
index e21bf04309d..923a096dd3d 100644
--- a/packages/core/execution/src/lib/workers/index.ts
+++ b/packages/core/execution/src/lib/workers/index.ts
@@ -82,6 +82,7 @@ export const ConsumeJobRequest = z.object({
jobId: z.string(),
jobData: JobData,
attempsStarted: z.number(),
+ lastAttempt: z.boolean().optional(),
engineToken: z.string(),
token: z.string(),
queueName: z.string(),
diff --git a/packages/pieces/core/subflows/package.json b/packages/pieces/core/subflows/package.json
index 2e02225bc77..dfe6c9c6630 100644
--- a/packages/pieces/core/subflows/package.json
+++ b/packages/pieces/core/subflows/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/piece-subflows",
- "version": "0.6.4",
+ "version": "0.7.0",
"main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts",
"scripts": {
diff --git a/packages/pieces/core/subflows/src/lib/actions/respond.ts b/packages/pieces/core/subflows/src/lib/actions/respond.ts
index a373192114e..95fd52362cf 100644
--- a/packages/pieces/core/subflows/src/lib/actions/respond.ts
+++ b/packages/pieces/core/subflows/src/lib/actions/respond.ts
@@ -1,6 +1,5 @@
import { DynamicPropsValue, PieceAuth, Property, StoreScope, createAction } from '@activepieces/pieces-framework';
-import { callableFlowKey, CallableFlowResponse, MOCK_CALLBACK_IN_TEST_FLOW_URL } from '../common';
-import { httpClient, HttpMethod } from '@activepieces/pieces-common';
+import { callableFlowKey, deliverSubflowResponse, MOCK_CALLBACK_IN_TEST_FLOW_URL } from '../common';
import { isNil } from '@activepieces/pieces-framework';
export const response = createAction({
@@ -62,15 +61,7 @@ export const response = createAction({
const callbackUrl = await context.store.get<string>(callableFlowKey(context.run.id), StoreScope.FLOW);
const isNotTestFlow = callbackUrl !== MOCK_CALLBACK_IN_TEST_FLOW_URL;
if (isNotTestFlow && !isNil(callbackUrl)) {
- await httpClient.sendRequest<CallableFlowResponse>({
- method: HttpMethod.POST,
- url: callbackUrl,
- body: {
- status: 'success',
- data: response
- },
- retries: 10,
- });
+ await deliverSubflowResponse({ callbackUrl, response });
}
return response;
},
diff --git a/packages/pieces/core/subflows/src/lib/common.ts b/packages/pieces/core/subflows/src/lib/common.ts
index 109c13d4f5d..ad10faf3aa4 100644
--- a/packages/pieces/core/subflows/src/lib/common.ts
+++ b/packages/pieces/core/subflows/src/lib/common.ts
@@ -134,6 +134,26 @@ export async function dispatchToSubflow({
return response.body;
}
+export async function deliverSubflowResponse({
+ callbackUrl,
+ response,
+}: DeliverSubflowResponseParams): Promise<void> {
+ const acknowledgement = await httpClient.sendRequest<SubflowCallbackAck>({
+ method: HttpMethod.POST,
+ url: callbackUrl,
+ body: {
+ status: 'success',
+ data: response,
+ },
+ retries: 10,
+ });
+ if (acknowledgement.body?.discarded === true) {
+ throw new Error(
+ 'The calling flow was no longer waiting for a response, so this response was discarded. Open the calling run to see why it stopped waiting.'
+ );
+ }
+}
+
type ListParams = {
flowsContext: FlowsContext,
params?: ListFlowsContextParams
@@ -147,4 +167,14 @@ type DispatchToSubflowParams = {
data: unknown;
callbackUrl?: string;
retries?: number;
-}
\ No newline at end of file
+}
+
+type DeliverSubflowResponseParams = {
+ callbackUrl: string;
+ response: unknown;
+}
+
+type SubflowCallbackAck = {
+ message?: string;
+ discarded?: boolean;
+}
diff --git a/packages/pieces/core/subflows/test/deliver-subflow-response.test.ts b/packages/pieces/core/subflows/test/deliver-subflow-response.test.ts
new file mode 100644
index 00000000000..48f55aa86ff
--- /dev/null
+++ b/packages/pieces/core/subflows/test/deliver-subflow-response.test.ts
@@ -0,0 +1,52 @@
+/// <reference types="vitest/globals" />
+
+import { httpClient } from '@activepieces/pieces-common';
+import { deliverSubflowResponse } from '../src/lib/common';
+
+vi.mock('@activepieces/pieces-common', async (importOriginal) => {
+ const actual = await importOriginal<typeof import('@activepieces/pieces-common')>();
+ return { ...actual, httpClient: { sendRequest: vi.fn() } };
+});
+
+const sendRequest = vi.mocked(httpClient.sendRequest);
+
+describe('deliverSubflowResponse', () => {
+ beforeEach(() => {
+ sendRequest.mockReset();
+ });
+
+ test('posts the response to the callback url', async () => {
+ sendRequest.mockResolvedValue({ status: 200, body: { message: 'recorded', discarded: false } });
+
+ await deliverSubflowResponse({ callbackUrl: 'https://host/resume', response: { ok: true } });
+
+ expect(sendRequest.mock.calls[0][0]).toMatchObject({
+ url: 'https://host/resume',
+ body: { status: 'success', data: { ok: true } },
+ });
+ });
+
+ test('throws when the caller had already stopped waiting, so the drop is not reported as success', async () => {
+ sendRequest.mockResolvedValue({ status: 200, body: { message: 'expired', discarded: true } });
+
+ await expect(
+ deliverSubflowResponse({ callbackUrl: 'https://host/resume', response: { ok: true } })
+ ).rejects.toThrow(/discarded/);
+ });
+
+ test('stays silent on a redelivery the server already accepted', async () => {
+ sendRequest.mockResolvedValue({ status: 200, body: { message: 'expired', discarded: false } });
+
+ await expect(
+ deliverSubflowResponse({ callbackUrl: 'https://host/resume', response: { ok: true } })
+ ).resolves.toBeUndefined();
+ });
+
+ test('stays silent when the acknowledgement carries no marker (older server)', async () => {
+ sendRequest.mockResolvedValue({ status: 200, body: {} });
+
+ await expect(
+ deliverSubflowResponse({ callbackUrl: 'https://host/resume', response: { ok: true } })
+ ).resolves.toBeUndefined();
+ });
+});
diff --git a/packages/server/api/src/app/flows/flow-run/engine-run-callback-service.ts b/packages/server/api/src/app/flows/flow-run/engine-run-callback-service.ts
index 2f1287c05e1..dba64305baf 100644
--- a/packages/server/api/src/app/flows/flow-run/engine-run-callback-service.ts
+++ b/packages/server/api/src/app/flows/flow-run/engine-run-callback-service.ts
@@ -53,6 +53,7 @@ export const engineRunCallbackService = (log: FastifyBaseLogger) => ({
provisionMs: request.provisionMs,
bootMs: request.bootMs,
runMs: request.runMs,
+ willRetry: request.willRetry,
}
await runsMetadataQueue(log).add(logData)
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
index a4349fe5b7d..1016bc5af6d 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
@@ -1,5 +1,5 @@
import { ActivepiecesError, apId, Cursor, ErrorCode, FlowId, FlowRunId, FlowVersionId, isNil, PlatformId, ProjectId, SeekPage } from '@activepieces/core-utils'
-import { apDayjs, wideEvent } from '@activepieces/server-utils'
+import { apDayjs, apDayjsDuration, wideEvent } from '@activepieces/server-utils'
import { ExecuteFlowJobData, ExecutionType, ExecutioOutputFile, FileCompression, FileType, FlowRetryStrategy, FlowRun, FlowRunCountByStatus, FlowRunStatus, FlowRunWithRetryError, FlowVersion, GenericStepOutput, isFlowRunStateTerminal, JobPayload, LATEST_JOB_DATA_SCHEMA_VERSION, logSerializer, LogSliceRef, ResumeReason, RunEnvironment, RunInternalError, SampleDataFileType, StepOutput, StepOutputStatus, StepOutputType, StreamStepProgress, WorkerJobType } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import pLimit from 'p-limit'
@@ -26,6 +26,7 @@ import { flowRunSideEffects } from './flow-run-side-effects'
import { runsMetadataQueue } from './flow-runs-queue'
const CANCELLABLE_STATUSES: FlowRunStatus[] = [FlowRunStatus.PAUSED, FlowRunStatus.QUEUED]
+const PARENT_BOUND_RETRY_BACKOFF_MS = apDayjsDuration(30, 'second').asMilliseconds()
export const WEBHOOK_TIMEOUT_MS = system.getNumberOrThrow(AppSystemProp.WEBHOOK_TIMEOUT_SECONDS) * 1000
@@ -648,10 +649,12 @@ export async function addToQueue(params: AddToQueueParams, log: FastifyBaseLogge
executionType: ExecutionType.BEGIN,
executeTrigger: params.executeTrigger,
}
+ const aParentIsBlockedOnThisRun = params.flowRun.failParentOnFailure && !isNil(params.flowRun.parentRunId)
await jobQueue(log).add({
id: params.jobId ?? params.flowRun.id,
type: JobType.ONE_TIME,
data,
+ retryBackoffMs: aParentIsBlockedOnThisRun ? PARENT_BOUND_RETRY_BACKOFF_MS : undefined,
})
return params.flowRun
}
diff --git a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
index 90ab3cd62f9..9c8d111f8be 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-runs-queue.ts
@@ -100,11 +100,17 @@ export const runsMetadataQueue = (log: FastifyBaseLogger) => ({
}
const parentRunId = savedFlowRun.parentRunId
- const shouldMarkParentAsFailed = savedFlowRun.failParentOnFailure && !isNil(parentRunId) && ![FlowRunStatus.SUCCEEDED, FlowRunStatus.RUNNING, FlowRunStatus.PAUSED, FlowRunStatus.QUEUED].includes(savedFlowRun.status)
+ const shouldMarkParentAsFailed = !isNil(parentRunId) && childRunFailsParent({
+ status: savedFlowRun.status,
+ failParentOnFailure: savedFlowRun.failParentOnFailure,
+ willRetry: runMetadata.willRetry,
+ })
if (shouldMarkParentAsFailed) {
await markParentRunAsFailed({
parentRunId,
childRunId: savedFlowRun.id,
+ childStatus: savedFlowRun.status,
+ childFailureMessage: savedFlowRun.failedStep?.message,
projectId: savedFlowRun.projectId,
log,
})
@@ -188,9 +194,17 @@ function buildTimeline({ existingFlowRun, runMetadata }: BuildTimelineParams): R
})
}
+export function childRunFailsParent({ status, failParentOnFailure, willRetry }: ChildRunFailsParentParams): boolean {
+ return failParentOnFailure
+ && status !== FlowRunStatus.SUCCEEDED
+ && isFlowRunStateTerminal({ status, ignoreInternalError: willRetry === true })
+}
+
export async function markParentRunAsFailed({
parentRunId,
childRunId,
+ childStatus,
+ childFailureMessage,
projectId,
log,
}: MarkParentRunAsFailedParams): Promise<void> {
@@ -208,7 +222,8 @@ export async function markParentRunAsFailed({
body: {
status: 'error',
data: {
- message: 'Subflow execution failed',
+ message: childFailureMessage ?? 'Subflow execution failed',
+ status: childStatus,
link: childRunUrl,
},
},
@@ -233,6 +248,12 @@ export async function markParentRunAsFailed({
}
}
+type ChildRunFailsParentParams = {
+ status: FlowRunStatus
+ failParentOnFailure: boolean
+ willRetry?: boolean
+}
+
type BuildTimelineParams = {
existingFlowRun: FlowRun
runMetadata: RunsMetadataUpsertData
@@ -241,6 +262,8 @@ type BuildTimelineParams = {
type MarkParentRunAsFailedParams = {
parentRunId: string
childRunId: string
+ childStatus: FlowRunStatus
+ childFailureMessage?: string
projectId: string
log: FastifyBaseLogger
}
diff --git a/packages/server/api/src/app/waitpoints/resume-controller.ts b/packages/server/api/src/app/waitpoints/resume-controller.ts
index 6afdd22b3c8..e1391948a21 100644
--- a/packages/server/api/src/app/waitpoints/resume-controller.ts
+++ b/packages/server/api/src/app/waitpoints/resume-controller.ts
@@ -111,7 +111,7 @@ async function handleConfirmResume({ flowRunId, waitpointId, action, body, heade
resumePayload: { body, headers, queryParams },
})
if (!acceptsHtml(headers)) {
- await reply.send({ message: stale ? EXPIRED_MESSAGE : RECORDED_MESSAGE })
+ await reply.send({ message: stale ? EXPIRED_MESSAGE : RECORDED_MESSAGE, discarded: await responseWasDiscarded({ flowRunId, waitpointId, stale, log }) })
return
}
const theme = await resolveResumePageTheme({ projectId: flowRun.projectId, log })
@@ -127,7 +127,7 @@ async function handleAsyncResume({ flowRunId, waitpointId, body, headers, queryP
waitpointId,
resumePayload: { body, headers, queryParams },
})
- await reply.send({ message: stale ? EXPIRED_MESSAGE : RECORDED_MESSAGE })
+ await reply.send({ message: stale ? EXPIRED_MESSAGE : RECORDED_MESSAGE, discarded: await responseWasDiscarded({ flowRunId, waitpointId, stale, log }) })
}
async function handleSyncResume({ flowRunId, waitpointId, body, headers, queryParams, log, reply, correlationId }: AsyncResumeHandlerParams & { correlationId: string }): Promise<void> {
@@ -157,6 +157,18 @@ async function handleLegacySyncResume({ flowRunId, body, headers, queryParams, l
await reply.status(response.status).headers(response.headers).send(response.body)
}
+/**
+ * A stale resume means the waitpoint is gone — which is equally what a SUCCEEDED delivery
+ * looks like once its HTTP response is lost and the caller retries. Only the absence of a
+ * consumption marker proves nobody ever accepted this response.
+ */
+async function responseWasDiscarded({ flowRunId, waitpointId, stale, log }: ResponseWasDiscardedParams): Promise<boolean> {
+ if (!stale) {
+ return false
+ }
+ return !(await resumeService(log).waitpointWasConsumed({ waitpointId, flowRunId }))
+}
+
async function resolveResumePageTheme({ projectId, log }: { projectId: string, log: FastifyBaseLogger }): Promise<ResumePageTheme> {
const platformId = await projectService(log).getPlatformId(projectId)
return resumePageHooks.get(log).getTheme({ platformId })
@@ -310,6 +322,13 @@ const STATUS_HTML_TEMPLATE = `<!DOCTYPE html>
</body>
</html>`
+type ResponseWasDiscardedParams = {
+ flowRunId: string
+ waitpointId: string
+ stale: boolean
+ log: FastifyBaseLogger
+}
+
type ConfirmationPageParams = {
flowRunId: string
waitpointId: string
diff --git a/packages/server/api/src/app/waitpoints/resume-service.ts b/packages/server/api/src/app/waitpoints/resume-service.ts
index f0584ffa2a8..15413acf038 100644
--- a/packages/server/api/src/app/waitpoints/resume-service.ts
+++ b/packages/server/api/src/app/waitpoints/resume-service.ts
@@ -1,8 +1,8 @@
-import { apId, FlowRunId, isNil } from '@activepieces/core-utils'
+import { apId, FlowRunId, isNil, tryCatch } from '@activepieces/core-utils'
import { EngineHttpResponse, ExecutionType, FlowRun, FlowRunStatus, isFlowRunStateTerminal, ResumeReason, RunEnvironment, StreamStepProgress } from '@activepieces/shared'
import { FastifyBaseLogger } from 'fastify'
import { StatusCodes } from 'http-status-codes'
-import { distributedLock } from '../database/redis-connections'
+import { distributedLock, distributedStore } from '../database/redis-connections'
import { addToQueue, findFlowRunOrThrow, flowRunService, WEBHOOK_TIMEOUT_MS } from '../flows/flow-run/flow-run-service'
import { flowRunSideEffects } from '../flows/flow-run/flow-run-side-effects'
import { projectService } from '../project/project-service'
@@ -29,6 +29,7 @@ export const resumeService = (log: FastifyBaseLogger) => ({
resumePayload: resumePayload ?? null,
workerHandlerId,
onReady: async (waitpoint) => {
+ await recordWaitpointConsumed({ waitpointId, log })
await enqueueResume({
flowRun,
waitpoint,
@@ -47,7 +48,6 @@ export const resumeService = (log: FastifyBaseLogger) => ({
log.info({ flowRun: { id: flowRunId } }, '[resumeService#resumeFromWaitpointWithoutLock] Race detected: metadata worker wrote PAUSED after callback completed waitpoint; consuming waitpoint and enqueuing resume')
// Consume the stale COMPLETED waitpoint under the lock so it cannot
// poison the next createForPause call on a subsequent loop iteration
- await waitpointService(log).delete({ id: latestWaitpoint.id })
await enqueueResume({
flowRun: currentFlowRun,
waitpoint: latestWaitpoint,
@@ -55,6 +55,8 @@ export const resumeService = (log: FastifyBaseLogger) => ({
workerHandlerId,
httpRequestId,
}, log)
+ await recordWaitpointConsumed({ waitpointId: latestWaitpoint.id, log })
+ await waitpointService(log).delete({ id: latestWaitpoint.id })
}
}
}
@@ -62,6 +64,19 @@ export const resumeService = (log: FastifyBaseLogger) => ({
return { flowRun, stale: !processed }
},
+ async waitpointWasConsumed({ waitpointId, flowRunId }: WaitpointWasConsumedParams): Promise<boolean> {
+ const waitpoint = await waitpointService(log).findByIdAndFlowRunId({ waitpointId, flowRunId })
+ if (!isNil(waitpoint)) {
+ return waitpoint.status === WaitpointStatus.COMPLETED
+ }
+ const { data, error } = await tryCatch(() => distributedStore.get<boolean>(waitpointConsumedKey(waitpointId)))
+ if (error) {
+ log.error({ waitpoint: { id: waitpointId }, error: String(error) }, '[resumeService#waitpointWasConsumed] Could not read the consumption marker, assuming the response was delivered')
+ return true
+ }
+ return data === true
+ },
+
async legacyResume({ flowRunId, resumePayload, workerHandlerId }: LegacyResumeParams): Promise<ResumeFromWaitpointResult> {
const flowRun = await findFlowRunOrThrow(flowRunId)
if (flowRun.status !== FlowRunStatus.PAUSED) {
@@ -128,6 +143,15 @@ export const resumeService = (log: FastifyBaseLogger) => ({
},
})
+async function recordWaitpointConsumed({ waitpointId, log }: RecordWaitpointConsumedParams): Promise<void> {
+ const { error } = await tryCatch(() => distributedStore.put(waitpointConsumedKey(waitpointId), true, WAITPOINT_CONSUMED_TTL_SECONDS))
+ if (isNil(error)) {
+ return
+ }
+ log.error({ waitpoint: { id: waitpointId }, error: String(error) }, '[resumeService#recordWaitpointConsumed] Could not record the consumption marker, rolling the resume back so the caller can retry')
+ throw error
+}
+
async function enqueueResume(params: EnqueueResumeParams, log: FastifyBaseLogger): Promise<void> {
const { flowRun, waitpoint, resumePayload, workerHandlerId, httpRequestId } = params
const platformId = await projectService(log).getPlatformId(flowRun.projectId)
@@ -150,6 +174,19 @@ async function enqueueResume(params: EnqueueResumeParams, log: FastifyBaseLogger
await flowRunSideEffects(log).onResume({ flowRun, platformId })
}
+const WAITPOINT_CONSUMED_TTL_SECONDS = 3600
+const waitpointConsumedKey = (waitpointId: string): string => `waitpoint_consumed:${waitpointId}`
+
+type WaitpointWasConsumedParams = {
+ waitpointId: string
+ flowRunId: string
+}
+
+type RecordWaitpointConsumedParams = {
+ waitpointId: string
+ log: FastifyBaseLogger
+}
+
type SyncResumePayload = {
body?: unknown
headers?: Record<string, string>
diff --git a/packages/server/api/src/app/workers/job-queue/job-broker.ts b/packages/server/api/src/app/workers/job-queue/job-broker.ts
index 9352e9086bf..923d67eecda 100644
--- a/packages/server/api/src/app/workers/job-queue/job-broker.ts
+++ b/packages/server/api/src/app/workers/job-queue/job-broker.ts
@@ -150,6 +150,7 @@ async function tryDequeue(worker: BullMQWorker, queueName: string, log: FastifyB
jobId,
jobData: migratedData,
attempsStarted: job.attemptsMade,
+ lastAttempt: job.attemptsMade + 1 >= (job.opts.attempts ?? 1),
engineToken,
token,
queueName,
diff --git a/packages/server/api/src/app/workers/job-queue/job-queue.ts b/packages/server/api/src/app/workers/job-queue/job-queue.ts
index ad442a20828..dabc5a93f98 100644
--- a/packages/server/api/src/app/workers/job-queue/job-queue.ts
+++ b/packages/server/api/src/app/workers/job-queue/job-queue.ts
@@ -52,6 +52,7 @@ export const jobQueue = (log: FastifyBaseLogger) => ({
priority: JOB_PRIORITY[getDefaultJobPriority(data)],
delay: params.delay,
jobId: params.id,
+ ...(isNil(params.retryBackoffMs) ? {} : { backoff: { type: 'exponential', delay: params.retryBackoffMs } }),
...(data.jobType === WorkerJobType.EVENT_DESTINATION ? { removeOnFail: true } : {}),
...([WorkerJobType.EXECUTE_AGENT_RUN, WorkerJobType.EXECUTE_AI].includes(data.jobType) ? { attempts: 1 } : {}),
...isUserInteractionJob(data.jobType) ? {
@@ -275,6 +276,8 @@ type BaseAddParams<JD extends Omit<JobData, 'engineToken'>, JT extends JobType>
type RepeatingJobAddParams = BaseAddParams<PollingJobData | RenewWebhookJobData, JobType.REPEATING> & {
scheduleOptions: ScheduleOptions
}
-type OneTimeJobAddParams = BaseAddParams<ExecuteFlowJobData | WebhookJobData | UserInteractionJobData | EventDestinationJobData | ExecuteAgentRunJobData | ExecuteAiJobData | ExecutePersonalizationResearchJobData, JobType.ONE_TIME>
+type OneTimeJobAddParams = BaseAddParams<ExecuteFlowJobData | WebhookJobData | UserInteractionJobData | EventDestinationJobData | ExecuteAgentRunJobData | ExecuteAiJobData | ExecutePersonalizationResearchJobData, JobType.ONE_TIME> & {
+ retryBackoffMs?: number
+}
export type AddJobParams<type extends JobType> = type extends JobType.REPEATING ? RepeatingJobAddParams : OneTimeJobAddParams
diff --git a/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts b/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts
index 1d2134c4a78..b865d6e9b99 100644
--- a/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts
+++ b/packages/server/api/src/app/workers/job/runs-metadata-queue-factory.ts
@@ -71,7 +71,7 @@ const RUNS_METADATA_UPSERT_KEYS: (keyof RunsMetadataUpsertData)[] = [
'triggeredBy', 'startTime', 'finishTime', 'status', 'tags',
'failedStep', 'stepNameToTest', 'parentRunId', 'failParentOnFailure',
'logsFileId', 'updated', 'stepsCount', 'requestId',
- 'provisionMs', 'bootMs', 'runMs',
+ 'provisionMs', 'bootMs', 'runMs', 'willRetry',
]
function stripToRunsMetadataUpsertData(params: RunsMetadataUpsertData): RunsMetadataUpsertData {
@@ -123,4 +123,5 @@ export type RunsMetadataUpsertData = {
provisionMs?: number
bootMs?: number
runMs?: number
+ willRetry?: boolean
}
diff --git a/packages/server/api/test/helpers/mocks/index.ts b/packages/server/api/test/helpers/mocks/index.ts
index 20611a2b901..4505ae825dc 100644
--- a/packages/server/api/test/helpers/mocks/index.ts
+++ b/packages/server/api/test/helpers/mocks/index.ts
@@ -130,6 +130,7 @@ export const createMockProject = (project?: Partial<Project>): Project => {
platformId: project?.platformId ?? apId(),
externalId: project?.externalId ?? apId(),
releasesEnabled: project?.releasesEnabled ?? false,
+ sensitive: project?.sensitive ?? false,
notifyFlowOwnerOnFailure: project?.notifyFlowOwnerOnFailure ?? false,
metadata: project?.metadata ?? null,
type: project?.type ?? ProjectType.TEAM,
@@ -707,6 +708,10 @@ export const createMockAIProvider = async (aiProvider?: MockAIProviderParams): P
modelIds: aiProvider?.modelIds ?? [],
projectScope: aiProvider?.projectScope ?? 'all',
projectIds: aiProvider?.projectIds ?? [],
+ status: 'active',
+ statusReason: null,
+ statusUpdated: null,
+ statusVersion: 0,
}
}
@@ -735,6 +740,19 @@ export const mockPieceMetadata = async (mockLog: FastifyBaseLogger): Promise<Pie
return mockPieceMetadata
}
+export const createMockWaitpoint = (waitpoint?: Partial<MockWaitpoint>): MockWaitpoint => {
+ return {
+ id: waitpoint?.id ?? apId(),
+ flowRunId: waitpoint?.flowRunId ?? apId(),
+ projectId: waitpoint?.projectId ?? apId(),
+ stepName: waitpoint?.stepName ?? 'approval',
+ type: waitpoint?.type ?? 'WEBHOOK',
+ status: waitpoint?.status ?? 'PENDING',
+ httpRequestId: waitpoint?.httpRequestId ?? null,
+ workerHandlerId: waitpoint?.workerHandlerId ?? null,
+ }
+}
+
export const createMockFolder = (folder?: Partial<Folder>): Folder => {
return {
id: folder?.id ?? apId(),
@@ -803,3 +821,14 @@ type MockBasicSetupParams = {
platform?: Partial<Platform>
project?: Partial<Project>
}
+
+export type MockWaitpoint = {
+ id: string
+ flowRunId: string
+ projectId: string
+ stepName: string
+ type: string
+ status: string
+ httpRequestId: string | null
+ workerHandlerId: string | null
+}
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/child-internal-error-retry.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/child-internal-error-retry.test.ts
new file mode 100644
index 00000000000..1dfddbb2e99
--- /dev/null
+++ b/packages/server/api/test/integration/ce/flows/flow-run/child-internal-error-retry.test.ts
@@ -0,0 +1,92 @@
+import { apId } from '@activepieces/core-utils'
+import { FlowRunStatus, FlowVersionState, RunEnvironment } from '@activepieces/shared'
+import { FastifyInstance } from 'fastify'
+import { engineRunCallbackService } from '../../../../../src/app/flows/flow-run/engine-run-callback-service'
+import { db } from '../../../../helpers/db'
+import { createMockFlow, createMockFlowRun, createMockFlowVersion, createMockWaitpoint, mockAndSaveBasicSetup } from '../../../../helpers/mocks'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'
+
+let app: FastifyInstance
+
+beforeAll(async () => {
+ app = await setupTestEnvironment()
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+async function createParentWithPausedWaitpointAndChild(projectId: string) {
+ const flow = createMockFlow({ projectId })
+ await db.save('flow', flow)
+
+ const flowVersion = createMockFlowVersion({ flowId: flow.id, state: FlowVersionState.LOCKED })
+ await db.save('flow_version', flowVersion)
+
+ const parentRun = createMockFlowRun({
+ projectId,
+ flowId: flow.id,
+ flowVersionId: flowVersion.id,
+ status: FlowRunStatus.PAUSED,
+ environment: RunEnvironment.PRODUCTION,
+ })
+ await db.save('flow_run', parentRun)
+
+ const waitpoint = createMockWaitpoint({ flowRunId: parentRun.id, projectId, stepName: 'step_1' })
+ await db.save('waitpoint', waitpoint)
+
+ const childRun = createMockFlowRun({
+ projectId,
+ flowId: flow.id,
+ flowVersionId: flowVersion.id,
+ status: FlowRunStatus.RUNNING,
+ environment: RunEnvironment.PRODUCTION,
+ parentRunId: parentRun.id,
+ failParentOnFailure: true,
+ finishTime: null,
+ })
+ await db.save('flow_run', childRun)
+
+ return { parentRun, childRun, waitpointId: waitpoint.id }
+}
+
+async function reportChildInternalError({ projectId, childRunId, willRetry }: { projectId: string, childRunId: string, willRetry: boolean }) {
+ await engineRunCallbackService(app.log).uploadRunLog({
+ projectId,
+ request: {
+ runId: childRunId,
+ projectId,
+ status: FlowRunStatus.INTERNAL_ERROR,
+ finishTime: new Date().toISOString(),
+ willRetry,
+ },
+ })
+}
+
+describe('child run INTERNAL_ERROR with a pending queue retry', () => {
+ it('leaves the parent paused while the child job still has an attempt left', async () => {
+ const { mockProject } = await mockAndSaveBasicSetup()
+ const { parentRun, childRun, waitpointId } = await createParentWithPausedWaitpointAndChild(mockProject.id)
+
+ await reportChildInternalError({ projectId: mockProject.id, childRunId: childRun.id, willRetry: true })
+ await new Promise((resolve) => setTimeout(resolve, 3000))
+
+ const waitpoint = await db.findOneBy<{ status: string }>('waitpoint', { id: waitpointId })
+ expect(waitpoint?.status).toBe('PENDING')
+
+ const parent = await db.findOneBy<{ status: string }>('flow_run', { id: parentRun.id })
+ expect(parent?.status).toBe(FlowRunStatus.PAUSED)
+ }, 60_000)
+
+ it('fails the parent once the child job has no attempt left', async () => {
+ const { mockProject } = await mockAndSaveBasicSetup()
+ const { childRun, waitpointId } = await createParentWithPausedWaitpointAndChild(mockProject.id)
+
+ await reportChildInternalError({ projectId: mockProject.id, childRunId: childRun.id, willRetry: false })
+
+ await vi.waitUntil(
+ async () => await db.findOneBy('waitpoint', { id: waitpointId }) === null,
+ { timeout: 10_000, interval: 250 },
+ )
+ }, 60_000)
+})
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/mark-parent-run-cross-tenant.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/mark-parent-run-cross-tenant.test.ts
index 4f2ed2cb3f9..71700a37bd6 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/mark-parent-run-cross-tenant.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/mark-parent-run-cross-tenant.test.ts
@@ -3,7 +3,7 @@ import { FlowRunStatus, FlowVersionState, RunEnvironment } from '@activepieces/s
import { FastifyInstance } from 'fastify'
import { markParentRunAsFailed } from '../../../../../src/app/flows/flow-run/flow-runs-queue'
import { db } from '../../../../helpers/db'
-import { createMockFlow, createMockFlowVersion, createMockFlowRun, mockAndSaveBasicSetup } from '../../../../helpers/mocks'
+import { createMockFlow, createMockFlowVersion, createMockFlowRun, createMockWaitpoint, mockAndSaveBasicSetup } from '../../../../helpers/mocks'
import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'
let app: FastifyInstance
@@ -35,19 +35,10 @@ async function createPausedParentWithWaitpoint(projectId: string) {
})
await db.save('flow_run', flowRun)
- const waitpointId = apId()
- await db.save('waitpoint', {
- id: waitpointId,
- flowRunId: flowRun.id,
- projectId,
- stepName: 'approval',
- type: 'WEBHOOK',
- status: 'PENDING',
- httpRequestId: null,
- workerHandlerId: null,
- })
+ const waitpoint = createMockWaitpoint({ flowRunId: flowRun.id, projectId })
+ await db.save('waitpoint', waitpoint)
- return { flowRun, waitpointId }
+ return { flowRun, waitpointId: waitpoint.id }
}
describe('markParentRunAsFailed tenant isolation', () => {
@@ -60,6 +51,7 @@ describe('markParentRunAsFailed tenant isolation', () => {
await markParentRunAsFailed({
parentRunId: victimRun.id,
childRunId: apId(),
+ childStatus: FlowRunStatus.INTERNAL_ERROR,
projectId: projectA.id,
log: app.log,
})
@@ -79,6 +71,7 @@ describe('markParentRunAsFailed tenant isolation', () => {
await markParentRunAsFailed({
parentRunId: parentRun.id,
childRunId: apId(),
+ childStatus: FlowRunStatus.INTERNAL_ERROR,
projectId: mockProject.id,
log: app.log,
})
diff --git a/packages/server/api/test/integration/ce/flows/flow-run/resume-flow-run.test.ts b/packages/server/api/test/integration/ce/flows/flow-run/resume-flow-run.test.ts
index e57c8cfa7a4..6f2fe0a1326 100644
--- a/packages/server/api/test/integration/ce/flows/flow-run/resume-flow-run.test.ts
+++ b/packages/server/api/test/integration/ce/flows/flow-run/resume-flow-run.test.ts
@@ -9,6 +9,7 @@ import { pubsub } from '../../../../../src/app/helper/pubsub'
import { engineResponseWatcher } from '../../../../../src/app/workers/engine-response-watcher'
import { redisMetadataKey, RunsMetadataUpsertData } from '../../../../../src/app/workers/job'
import { createHandlers } from '../../../../../src/app/workers/rpc/worker-rpc-service'
+import { distributedStore } from '../../../../../src/app/database/redis-connections'
import { db } from '../../../../helpers/db'
import { createMockFlow, createMockFlowRun, createMockFlowVersion } from '../../../../helpers/mocks'
import { createTestContext, TestContext } from '../../../../helpers/test-context'
@@ -517,6 +518,7 @@ describe('Resume flow run', () => {
expect(firstResponse.statusCode).toBe(200)
expect(firstResponse.json()).toEqual({
message: 'Your response has been recorded. You can close this page now.',
+ discarded: false,
})
const secondResponse = await app.inject({
@@ -527,12 +529,113 @@ describe('Resume flow run', () => {
expect(secondResponse.statusCode).toBe(200)
expect(secondResponse.json()).toEqual({
message: 'This link has expired. The action may have already been processed.',
+ discarded: false,
})
const waitpointAfter = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })
expect(waitpointAfter).toBeNull()
})
+ it.each([
+ FlowRunStatus.INTERNAL_ERROR,
+ FlowRunStatus.SUCCEEDED,
+ FlowRunStatus.FAILED,
+ ])('does not report a redelivered response as discarded when the caller has since reached %s', async (status) => {
+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })
+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })
+
+ const deliver = async () => app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,
+ body: { status: 'success', data: { greeting: 'Hello' } },
+ })
+
+ await deliver()
+ await db.update('flow_run', flowRun.id, { status })
+ const redelivery = await deliver()
+
+ expect(redelivery.json().discarded).toBe(false)
+ })
+
+ it('falls back to the completed waitpoint row when no consumption marker exists', async () => {
+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })
+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })
+ // RUNNING takes the complete() branch: the waitpoint is marked COMPLETED and kept,
+ // and no Redis marker is written at all, so only the row can prove acceptance.
+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.RUNNING })
+
+ const accepted = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,
+ body: { status: 'success', data: { greeting: 'Hello' } },
+ })
+ expect(accepted.json().discarded).toBe(false)
+
+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.FAILED })
+ const redelivery = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,
+ body: { status: 'success', data: { greeting: 'Hello' } },
+ })
+
+ expect(redelivery.json().discarded).toBe(false)
+ })
+
+ it('keeps the evidence when the race path deletes the completed waitpoint row', async () => {
+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })
+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })
+ // RUNNING takes the complete() branch, then the PAUSED race block below it deletes the
+ // COMPLETED row — so the marker is the only evidence that can survive.
+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.RUNNING })
+
+ await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,
+ body: { status: 'success', data: { greeting: 'Hello' } },
+ })
+
+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.FAILED })
+ const redelivery = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,
+ body: { status: 'success', data: { greeting: 'Hello' } },
+ })
+
+ expect(redelivery.json().discarded).toBe(false)
+ })
+
+ it('ignores a consumption marker while the waitpoint row is still pending', async () => {
+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })
+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })
+ // A rollback after the marker was written leaves the marker behind with the row intact.
+ // The row is the authority: still PENDING means nothing ever accepted the response.
+ await distributedStore.put(`waitpoint_consumed:${waitpoint!.id}`, true, 3600)
+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.FAILED })
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,
+ body: { status: 'success', data: { greeting: 'Hello' } },
+ })
+
+ expect(response.json().discarded).toBe(true)
+ })
+
+ it('reports a response as discarded when the waitpoint was never consumed', async () => {
+ const { flowRun } = await createPausedFlowRunWithWaitpoint({ projectId: ctx.project.id })
+ const waitpoint = await db.findOneBy<{ id: string }>('waitpoint', { flowRunId: flowRun.id })
+ await db.update('flow_run', flowRun.id, { status: FlowRunStatus.FAILED })
+
+ const response = await app.inject({
+ method: 'POST',
+ url: `/api/v1/flow-runs/${flowRun.id}/waitpoints/${waitpoint!.id}`,
+ body: { status: 'success', data: { greeting: 'Hello' } },
+ })
+
+ expect(response.statusCode).toBe(200)
+ expect(response.json().discarded).toBe(true)
+ })
+
it('should clean up waitpoints when flow is deleted via batchDeleteByFlowId', async () => {
const { flowRun, flow } = await createPausedFlowRunWithWaitpoint({
projectId: ctx.project.id,
@@ -567,6 +670,7 @@ describe('Resume flow run', () => {
expect(response.statusCode).toBe(200)
expect(response.json()).toEqual({
message: 'Your response has been recorded. You can close this page now.',
+ discarded: false,
})
const waitpointAfter = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })
@@ -733,6 +837,7 @@ describe('Resume flow run', () => {
expect(response.statusCode).toBe(200)
expect(response.json()).toEqual({
message: 'Your response has been recorded. You can close this page now.',
+ discarded: false,
})
const waitpointAfter = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })
@@ -775,6 +880,7 @@ describe('Resume flow run', () => {
expect(response.statusCode).toBe(200)
expect(response.json()).toEqual({
message: 'Your response has been recorded. You can close this page now.',
+ discarded: false,
})
const waitpointAfter = await db.findOneBy('waitpoint', { flowRunId: flowRun.id })
diff --git a/packages/server/api/test/unit/app/flows/flow-run/child-run-fails-parent.test.ts b/packages/server/api/test/unit/app/flows/flow-run/child-run-fails-parent.test.ts
new file mode 100644
index 00000000000..008897565a9
--- /dev/null
+++ b/packages/server/api/test/unit/app/flows/flow-run/child-run-fails-parent.test.ts
@@ -0,0 +1,51 @@
+import { FlowRunStatus } from '@activepieces/shared'
+import { childRunFailsParent } from '../../../../../src/app/flows/flow-run/flow-runs-queue'
+
+describe('childRunFailsParent', () => {
+ it('does not fail the parent while the child job still owes an attempt', () => {
+ expect(childRunFailsParent({
+ status: FlowRunStatus.INTERNAL_ERROR,
+ failParentOnFailure: true,
+ willRetry: true,
+ })).toBe(false)
+ })
+
+ it('fails the parent once the child job has exhausted its attempts', () => {
+ expect(childRunFailsParent({
+ status: FlowRunStatus.INTERNAL_ERROR,
+ failParentOnFailure: true,
+ willRetry: false,
+ })).toBe(true)
+ })
+
+ it('fails the parent when the report carries no retry information', () => {
+ expect(childRunFailsParent({
+ status: FlowRunStatus.INTERNAL_ERROR,
+ failParentOnFailure: true,
+ })).toBe(true)
+ })
+
+ it.each([
+ FlowRunStatus.FAILED,
+ FlowRunStatus.CANCELED,
+ ])('fails the parent on terminal %s even if a stale retry flag is set', (status) => {
+ expect(childRunFailsParent({ status, failParentOnFailure: true, willRetry: true })).toBe(true)
+ })
+
+ it.each([
+ FlowRunStatus.SUCCEEDED,
+ FlowRunStatus.RUNNING,
+ FlowRunStatus.PAUSED,
+ FlowRunStatus.QUEUED,
+ ])('leaves the parent alone while the child is %s', (status) => {
+ expect(childRunFailsParent({ status, failParentOnFailure: true, willRetry: false })).toBe(false)
+ })
+
+ it('leaves the parent alone when the child is not linked to it', () => {
+ expect(childRunFailsParent({
+ status: FlowRunStatus.INTERNAL_ERROR,
+ failParentOnFailure: false,
+ willRetry: false,
+ })).toBe(false)
+ })
+})
diff --git a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
index cb9091edeb3..54acb8b64c7 100644
--- a/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
+++ b/packages/server/worker/src/lib/execute/jobs/execute-flow.ts
@@ -188,6 +188,7 @@ function toInternalError(source: RunInternalErrorSource, error: unknown): RunInt
}
async function reportFlowStatus({ ctx, data, status, internalError, failedStep }: ReportFlowStatusParams): Promise<void> {
+ const willRetry = status === FlowRunStatus.INTERNAL_ERROR && !ctx.lastAttempt
// A status report has no log file of its own; carry logsFileId only for an internalError the server may
// persist into one (see uploadRunLog). Sending it on a plain status report would dangle flow_run.logsFileId.
await ctx.apiClient.uploadRunLog({
@@ -199,6 +200,7 @@ async function reportFlowStatus({ ctx, data, status, internalError, failedStep }
...(isNil(internalError) ? {} : { logsFileId: data.logsFileId }),
internalError,
failedStep,
+ willRetry,
...spreadIfDefined('workerHandlerId', data.workerHandlerId ?? undefined),
...spreadIfDefined('httpRequestId', data.httpRequestId),
})
@@ -207,7 +209,7 @@ async function reportFlowStatus({ ctx, data, status, internalError, failedStep }
onCallService(ctx.log, workerSettings.getSettings().PAGE_ONCALL_WEBHOOK).page({
code: ErrorCode.ENGINE_OPERATION_FAILURE,
message: `Flow run ${data.runId} ended with INTERNAL_ERROR`,
- params: { runId: data.runId, flowId: data.flowId, projectId: data.projectId },
+ params: { runId: data.runId, flowId: data.flowId, projectId: data.projectId, willRetry },
}).catch((e) => ctx.log.error({ flowRun: { id: data.runId }, error: inspect(e) }, 'Failed to send on-call page for INTERNAL_ERROR'))
}
}
diff --git a/packages/server/worker/src/lib/execute/types.ts b/packages/server/worker/src/lib/execute/types.ts
index 71139b526f0..a8d622dcdb2 100644
--- a/packages/server/worker/src/lib/execute/types.ts
+++ b/packages/server/worker/src/lib/execute/types.ts
@@ -13,6 +13,7 @@ export type JobContext = {
resolver: Resolver
workerIndex: number
jobId: string
+ lastAttempt: boolean
engineToken: string
internalApiUrl: string
publicApiUrl: string
diff --git a/packages/server/worker/src/lib/worker.ts b/packages/server/worker/src/lib/worker.ts
index 1b8f06385e8..001030a3d51 100644
--- a/packages/server/worker/src/lib/worker.ts
+++ b/packages/server/worker/src/lib/worker.ts
@@ -371,6 +371,7 @@ async function executeJob(apiClient: WorkerToApiContract, job: ConsumeJobRequest
}),
workerIndex,
jobId: job.jobId,
+ lastAttempt: job.lastAttempt ?? true,
engineToken: job.engineToken,
internalApiUrl,
publicApiUrl,